[==========] Running 8 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 6 tests from LeaderRebalancerTest
[ RUN      ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260709 13:06:53.173017 24113 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.140.126:37707
I20260709 13:06:53.174580 24113 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260709 13:06:53.175606 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:53.192273 24120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:53.192710 24122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:53.193173 24119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:53.194418 24113 server_base.cc:1061] running on GCE node
I20260709 13:06:54.409401 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:54.409667 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:54.409894 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602414409862 us; error 0 us; skew 500 ppm
I20260709 13:06:54.410727 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:54.416059 24113 webserver.cc:533] Webserver started at http://127.23.140.126:38541/ using document root <none> and password file <none>
I20260709 13:06:54.416930 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:54.417167 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:54.417668 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:54.423359 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/master-0-root/instance:
uuid: "2357efedd24947459e4bb0fd26f6a96d"
format_stamp: "Formatted at 2026-07-09 13:06:54 on dist-test-slave-k73n"
I20260709 13:06:54.431191 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.009s	sys 0.000s
I20260709 13:06:54.436372 24128 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:54.437603 24113 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 13:06:54.437958 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/master-0-root
uuid: "2357efedd24947459e4bb0fd26f6a96d"
format_stamp: "Formatted at 2026-07-09 13:06:54 on dist-test-slave-k73n"
I20260709 13:06:54.438313 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:54.493708 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:54.495221 24113 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260709 13:06:54.495697 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:54.567389 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.126:37707
I20260709 13:06:54.567508 24179 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.126:37707 every 8 connection(s)
I20260709 13:06:54.575340 24180 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:54.597409 24180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d: Bootstrap starting.
I20260709 13:06:54.603427 24180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:54.605417 24180 log.cc:826] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d: Log is configured to *not* fsync() on all Append() calls
I20260709 13:06:54.610051 24180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d: No bootstrap required, opened a new log
I20260709 13:06:54.630594 24180 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2357efedd24947459e4bb0fd26f6a96d" member_type: VOTER }
I20260709 13:06:54.631206 24180 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:54.631443 24180 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2357efedd24947459e4bb0fd26f6a96d, State: Initialized, Role: FOLLOWER
I20260709 13:06:54.632494 24180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2357efedd24947459e4bb0fd26f6a96d" member_type: VOTER }
I20260709 13:06:54.633023 24180 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:06:54.633378 24180 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:06:54.633731 24180 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:54.637874 24180 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2357efedd24947459e4bb0fd26f6a96d" member_type: VOTER }
I20260709 13:06:54.638600 24180 leader_election.cc:304] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2357efedd24947459e4bb0fd26f6a96d; no voters: 
I20260709 13:06:54.640336 24180 leader_election.cc:290] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:06:54.640705 24183 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:54.642963 24183 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [term 1 LEADER]: Becoming Leader. State: Replica: 2357efedd24947459e4bb0fd26f6a96d, State: Running, Role: LEADER
I20260709 13:06:54.643960 24183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2357efedd24947459e4bb0fd26f6a96d" member_type: VOTER }
I20260709 13:06:54.644977 24180 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:06:54.654248 24184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2357efedd24947459e4bb0fd26f6a96d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2357efedd24947459e4bb0fd26f6a96d" member_type: VOTER } }
I20260709 13:06:54.654803 24185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2357efedd24947459e4bb0fd26f6a96d. Latest consensus state: current_term: 1 leader_uuid: "2357efedd24947459e4bb0fd26f6a96d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2357efedd24947459e4bb0fd26f6a96d" member_type: VOTER } }
I20260709 13:06:54.655076 24184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [sys.catalog]: This master's current role is: LEADER
I20260709 13:06:54.655378 24185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [sys.catalog]: This master's current role is: LEADER
I20260709 13:06:54.658102 24192 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:06:54.669814 24192 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:06:54.673002 24113 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:06:54.688616 24192 catalog_manager.cc:1383] Generated new cluster ID: 39750c6436ac4d908a674d3b556bd19c
I20260709 13:06:54.688894 24192 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:06:54.712332 24192 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:06:54.713927 24192 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:06:54.728183 24192 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d: Generated new TSK 0
I20260709 13:06:54.729089 24192 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:06:54.739269 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:54.747702 24202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:54.748713 24203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:54.754729 24113 server_base.cc:1061] running on GCE node
W20260709 13:06:54.754691 24205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:54.755726 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:54.755913 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:54.756138 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602414756106 us; error 0 us; skew 500 ppm
I20260709 13:06:54.756795 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:54.761292 24113 webserver.cc:533] Webserver started at http://127.23.140.65:45891/ using document root <none> and password file <none>
I20260709 13:06:54.761852 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:54.762059 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:54.762358 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:54.763471 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/ts-0-root/instance:
uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
format_stamp: "Formatted at 2026-07-09 13:06:54 on dist-test-slave-k73n"
I20260709 13:06:54.769279 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.004s
I20260709 13:06:54.773780 24210 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:54.774623 24113 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20260709 13:06:54.774946 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/ts-0-root
uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
format_stamp: "Formatted at 2026-07-09 13:06:54 on dist-test-slave-k73n"
I20260709 13:06:54.775249 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:54.797421 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:54.798732 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:54.801508 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:54.806245 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:54.806461 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:54.806773 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:54.806918 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:54.914336 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.65:41615
I20260709 13:06:54.914546 24272 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.65:41615 every 8 connection(s)
I20260709 13:06:54.921437 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:54.931447 24277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:54.936947 24278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:54.939474 24280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:54.941705 24113 server_base.cc:1061] running on GCE node
I20260709 13:06:54.942814 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:54.943089 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:54.943305 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602414943286 us; error 0 us; skew 500 ppm
I20260709 13:06:54.944149 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:54.947909 24113 webserver.cc:533] Webserver started at http://127.23.140.66:37021/ using document root <none> and password file <none>
I20260709 13:06:54.948527 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:54.948782 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:54.948899 24273 heartbeater.cc:344] Connected to a master server at 127.23.140.126:37707
I20260709 13:06:54.949185 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:54.949407 24273 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:54.950366 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/ts-1-root/instance:
uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
format_stamp: "Formatted at 2026-07-09 13:06:54 on dist-test-slave-k73n"
I20260709 13:06:54.950577 24273 heartbeater.cc:507] Master 127.23.140.126:37707 requested a full tablet report, sending...
I20260709 13:06:54.954097 24145 ts_manager.cc:194] Registered new tserver with Master: b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615)
I20260709 13:06:54.956981 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.001s
I20260709 13:06:54.956964 24145 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60634
I20260709 13:06:54.962352 24285 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:54.963240 24113 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 13:06:54.963527 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/ts-1-root
uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
format_stamp: "Formatted at 2026-07-09 13:06:54 on dist-test-slave-k73n"
I20260709 13:06:54.963793 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:54.995232 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:54.996485 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:54.998673 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:55.001513 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:55.001739 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:55.001976 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:55.002148 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:55.097795 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.66:42417
I20260709 13:06:55.097906 24347 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.66:42417 every 8 connection(s)
I20260709 13:06:55.106973 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:55.114440 24351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:55.121258 24348 heartbeater.cc:344] Connected to a master server at 127.23.140.126:37707
I20260709 13:06:55.121814 24348 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:55.122951 24348 heartbeater.cc:507] Master 127.23.140.126:37707 requested a full tablet report, sending...
W20260709 13:06:55.124006 24352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:55.125761 24145 ts_manager.cc:194] Registered new tserver with Master: 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:06:55.128012 24145 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60640
I20260709 13:06:55.131228 24113 server_base.cc:1061] running on GCE node
W20260709 13:06:55.132347 24354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:55.133435 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:55.133667 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:55.133842 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602415133819 us; error 0 us; skew 500 ppm
I20260709 13:06:55.134362 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:55.136864 24113 webserver.cc:533] Webserver started at http://127.23.140.67:33967/ using document root <none> and password file <none>
I20260709 13:06:55.137372 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:55.137576 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:55.137836 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:55.138901 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/ts-2-root/instance:
uuid: "f450d3cd998e454da3e92d0381791c02"
format_stamp: "Formatted at 2026-07-09 13:06:55 on dist-test-slave-k73n"
I20260709 13:06:55.143568 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 13:06:55.147207 24359 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:55.148082 24113 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 13:06:55.148368 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/ts-2-root
uuid: "f450d3cd998e454da3e92d0381791c02"
format_stamp: "Formatted at 2026-07-09 13:06:55 on dist-test-slave-k73n"
I20260709 13:06:55.148671 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602412999163-24113-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:55.170938 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:55.172214 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:55.173722 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:55.175930 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:55.176152 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:55.176388 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:55.176582 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:55.273118 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.67:35329
I20260709 13:06:55.273197 24421 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.67:35329 every 8 connection(s)
I20260709 13:06:55.287859 24422 heartbeater.cc:344] Connected to a master server at 127.23.140.126:37707
I20260709 13:06:55.288228 24422 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:55.288997 24422 heartbeater.cc:507] Master 127.23.140.126:37707 requested a full tablet report, sending...
I20260709 13:06:55.290848 24145 ts_manager.cc:194] Registered new tserver with Master: f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:06:55.291239 24113 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.011014562s
I20260709 13:06:55.292498 24145 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60654
I20260709 13:06:55.341902 24145 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60666:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:06:55.345404 24145 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:06:55.447121 24387 tablet_service.cc:1511] Processing CreateTablet for tablet 83da7aa0dd1047908cfc3f6e16337706 (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:06:55.447122 24385 tablet_service.cc:1511] Processing CreateTablet for tablet df51c4376a72401495d95fe4c2df882a (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:06:55.447952 24386 tablet_service.cc:1511] Processing CreateTablet for tablet 69c6de3b5e014474b86031c03d56f84f (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:06:55.450948 24385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df51c4376a72401495d95fe4c2df882a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.451609 24386 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c6de3b5e014474b86031c03d56f84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.457477 24387 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83da7aa0dd1047908cfc3f6e16337706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.461263 24238 tablet_service.cc:1511] Processing CreateTablet for tablet 83da7aa0dd1047908cfc3f6e16337706 (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:06:55.462244 24235 tablet_service.cc:1511] Processing CreateTablet for tablet b3249533cb3842de91db1dc4cb6a4bcc (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:06:55.462788 24238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83da7aa0dd1047908cfc3f6e16337706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.462415 24237 tablet_service.cc:1511] Processing CreateTablet for tablet 69c6de3b5e014474b86031c03d56f84f (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:06:55.471467 24235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3249533cb3842de91db1dc4cb6a4bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.472540 24237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c6de3b5e014474b86031c03d56f84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.468992 24384 tablet_service.cc:1511] Processing CreateTablet for tablet b3249533cb3842de91db1dc4cb6a4bcc (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:06:55.475875 24384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3249533cb3842de91db1dc4cb6a4bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.464974 24236 tablet_service.cc:1511] Processing CreateTablet for tablet df51c4376a72401495d95fe4c2df882a (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:06:55.473274 24383 tablet_service.cc:1511] Processing CreateTablet for tablet 9587f033fa334d328ccd9149b5b22bcf (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:06:55.481418 24236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df51c4376a72401495d95fe4c2df882a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.484475 24234 tablet_service.cc:1511] Processing CreateTablet for tablet 9587f033fa334d328ccd9149b5b22bcf (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:06:55.483199 24233 tablet_service.cc:1511] Processing CreateTablet for tablet e395d0ca9a1f4f08bb3fed809205f52d (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:06:55.487404 24234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9587f033fa334d328ccd9149b5b22bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.494621 24233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e395d0ca9a1f4f08bb3fed809205f52d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.498363 24383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9587f033fa334d328ccd9149b5b22bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.497219 24382 tablet_service.cc:1511] Processing CreateTablet for tablet e395d0ca9a1f4f08bb3fed809205f52d (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:06:55.503957 24232 tablet_service.cc:1511] Processing CreateTablet for tablet 76d88f7a9bc84cbcb633d2433ddad860 (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:06:55.510667 24313 tablet_service.cc:1511] Processing CreateTablet for tablet 83da7aa0dd1047908cfc3f6e16337706 (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:06:55.533573 24382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e395d0ca9a1f4f08bb3fed809205f52d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.533649 24232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76d88f7a9bc84cbcb633d2433ddad860. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.520557 24308 tablet_service.cc:1511] Processing CreateTablet for tablet e395d0ca9a1f4f08bb3fed809205f52d (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:06:55.538350 24313 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83da7aa0dd1047908cfc3f6e16337706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.538861 24307 tablet_service.cc:1511] Processing CreateTablet for tablet 76d88f7a9bc84cbcb633d2433ddad860 (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:06:55.544968 24307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76d88f7a9bc84cbcb633d2433ddad860. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.544773 24381 tablet_service.cc:1511] Processing CreateTablet for tablet 76d88f7a9bc84cbcb633d2433ddad860 (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:06:55.548090 24311 tablet_service.cc:1511] Processing CreateTablet for tablet df51c4376a72401495d95fe4c2df882a (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:06:55.541468 24312 tablet_service.cc:1511] Processing CreateTablet for tablet 69c6de3b5e014474b86031c03d56f84f (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:06:55.544991 24231 tablet_service.cc:1511] Processing CreateTablet for tablet 51f37fd5f5c34b13885b93a9eee3bc8f (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:06:55.534649 24310 tablet_service.cc:1511] Processing CreateTablet for tablet b3249533cb3842de91db1dc4cb6a4bcc (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:06:55.554486 24306 tablet_service.cc:1511] Processing CreateTablet for tablet 51f37fd5f5c34b13885b93a9eee3bc8f (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:06:55.555727 24231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51f37fd5f5c34b13885b93a9eee3bc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.560379 24311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df51c4376a72401495d95fe4c2df882a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.560477 24381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76d88f7a9bc84cbcb633d2433ddad860. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.564427 24310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3249533cb3842de91db1dc4cb6a4bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.554504 24309 tablet_service.cc:1511] Processing CreateTablet for tablet 9587f033fa334d328ccd9149b5b22bcf (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:06:55.575223 24312 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c6de3b5e014474b86031c03d56f84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.594120 24440 tablet_bootstrap.cc:492] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02: Bootstrap starting.
I20260709 13:06:55.597195 24230 tablet_service.cc:1511] Processing CreateTablet for tablet dc38bf75745341a08fa2ca990c4d199d (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:06:55.596575 24385 tablet_service.cc:1511] Processing CreateTablet for tablet 51f37fd5f5c34b13885b93a9eee3bc8f (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:06:55.603034 24385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51f37fd5f5c34b13885b93a9eee3bc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.604074 24230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc38bf75745341a08fa2ca990c4d199d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.604522 24306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51f37fd5f5c34b13885b93a9eee3bc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.613619 24309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9587f033fa334d328ccd9149b5b22bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.621474 24308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e395d0ca9a1f4f08bb3fed809205f52d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.625427 24305 tablet_service.cc:1511] Processing CreateTablet for tablet dc38bf75745341a08fa2ca990c4d199d (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:06:55.626787 24305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc38bf75745341a08fa2ca990c4d199d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:06:55.663324 24193 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:55.672741 24441 tablet_bootstrap.cc:492] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311: Bootstrap starting.
I20260709 13:06:55.741839 24440 tablet_bootstrap.cc:654] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:55.742924 24387 tablet_service.cc:1511] Processing CreateTablet for tablet dc38bf75745341a08fa2ca990c4d199d (DEFAULT_TABLE table=test-workload [id=cb45e31576724713afe1e2a914a981c1]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:06:55.744680 24387 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc38bf75745341a08fa2ca990c4d199d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:55.770365 24441 tablet_bootstrap.cc:654] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:55.780254 24440 tablet_bootstrap.cc:492] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02: No bootstrap required, opened a new log
I20260709 13:06:55.780853 24440 ts_tablet_manager.cc:1403] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02: Time spent bootstrapping tablet: real 0.219s	user 0.023s	sys 0.011s
I20260709 13:06:55.782469 24442 tablet_bootstrap.cc:492] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304: Bootstrap starting.
I20260709 13:06:55.785710 24440 raft_consensus.cc:359] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.786623 24440 raft_consensus.cc:385] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:55.786949 24440 raft_consensus.cc:740] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Initialized, Role: FOLLOWER
I20260709 13:06:55.787775 24440 consensus_queue.cc:260] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.790518 24422 heartbeater.cc:499] Master 127.23.140.126:37707 was elected leader, sending a full tablet report...
I20260709 13:06:55.791574 24441 tablet_bootstrap.cc:492] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311: No bootstrap required, opened a new log
I20260709 13:06:55.791862 24442 tablet_bootstrap.cc:654] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:55.792011 24441 ts_tablet_manager.cc:1403] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311: Time spent bootstrapping tablet: real 0.130s	user 0.009s	sys 0.007s
I20260709 13:06:55.793795 24440 ts_tablet_manager.cc:1434] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.001s
I20260709 13:06:55.794773 24440 tablet_bootstrap.cc:492] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02: Bootstrap starting.
I20260709 13:06:55.795013 24441 raft_consensus.cc:359] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } }
I20260709 13:06:55.795704 24441 raft_consensus.cc:385] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:55.795998 24441 raft_consensus.cc:740] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Initialized, Role: FOLLOWER
I20260709 13:06:55.796721 24441 consensus_queue.cc:260] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } }
I20260709 13:06:55.810375 24273 heartbeater.cc:499] Master 127.23.140.126:37707 was elected leader, sending a full tablet report...
I20260709 13:06:55.811259 24441 ts_tablet_manager.cc:1434] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311: Time spent starting tablet: real 0.019s	user 0.004s	sys 0.012s
I20260709 13:06:55.812608 24441 tablet_bootstrap.cc:492] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311: Bootstrap starting.
I20260709 13:06:55.819824 24440 tablet_bootstrap.cc:654] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:55.820487 24441 tablet_bootstrap.cc:654] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:55.827351 24442 tablet_bootstrap.cc:492] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304: No bootstrap required, opened a new log
I20260709 13:06:55.827812 24442 ts_tablet_manager.cc:1403] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304: Time spent bootstrapping tablet: real 0.052s	user 0.011s	sys 0.018s
I20260709 13:06:55.828450 24441 tablet_bootstrap.cc:492] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311: No bootstrap required, opened a new log
I20260709 13:06:55.828919 24441 ts_tablet_manager.cc:1403] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311: Time spent bootstrapping tablet: real 0.017s	user 0.006s	sys 0.007s
I20260709 13:06:55.830669 24442 raft_consensus.cc:359] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.831388 24442 raft_consensus.cc:385] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:55.831704 24442 raft_consensus.cc:740] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Initialized, Role: FOLLOWER
I20260709 13:06:55.831655 24441 raft_consensus.cc:359] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.832453 24441 raft_consensus.cc:385] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:55.832763 24441 raft_consensus.cc:740] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Initialized, Role: FOLLOWER
I20260709 13:06:55.834834 24441 consensus_queue.cc:260] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.835750 24440 tablet_bootstrap.cc:492] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02: No bootstrap required, opened a new log
I20260709 13:06:55.836211 24440 ts_tablet_manager.cc:1403] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02: Time spent bootstrapping tablet: real 0.042s	user 0.015s	sys 0.007s
I20260709 13:06:55.837189 24441 ts_tablet_manager.cc:1434] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:06:55.838090 24441 tablet_bootstrap.cc:492] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311: Bootstrap starting.
I20260709 13:06:55.838936 24440 raft_consensus.cc:359] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.839622 24440 raft_consensus.cc:385] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:55.839943 24440 raft_consensus.cc:740] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Initialized, Role: FOLLOWER
I20260709 13:06:55.839784 24442 consensus_queue.cc:260] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.840682 24440 consensus_queue.cc:260] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.843048 24440 ts_tablet_manager.cc:1434] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:06:55.843911 24440 tablet_bootstrap.cc:492] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02: Bootstrap starting.
I20260709 13:06:55.847596 24441 tablet_bootstrap.cc:654] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:55.850353 24348 heartbeater.cc:499] Master 127.23.140.126:37707 was elected leader, sending a full tablet report...
I20260709 13:06:55.851060 24440 tablet_bootstrap.cc:654] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:55.855829 24442 ts_tablet_manager.cc:1434] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304: Time spent starting tablet: real 0.028s	user 0.005s	sys 0.004s
I20260709 13:06:55.856889 24442 tablet_bootstrap.cc:492] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304: Bootstrap starting.
W20260709 13:06:55.857059 24349 tablet.cc:2406] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 13:06:55.862920 24441 tablet_bootstrap.cc:492] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311: No bootstrap required, opened a new log
I20260709 13:06:55.863462 24441 ts_tablet_manager.cc:1403] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311: Time spent bootstrapping tablet: real 0.026s	user 0.014s	sys 0.000s
I20260709 13:06:55.864763 24442 tablet_bootstrap.cc:654] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:55.866752 24441 raft_consensus.cc:359] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.868002 24441 raft_consensus.cc:385] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:55.869755 24441 raft_consensus.cc:740] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Initialized, Role: FOLLOWER
I20260709 13:06:55.873245 24441 consensus_queue.cc:260] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.872310 24440 tablet_bootstrap.cc:492] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02: No bootstrap required, opened a new log
I20260709 13:06:55.875073 24440 ts_tablet_manager.cc:1403] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02: Time spent bootstrapping tablet: real 0.031s	user 0.018s	sys 0.009s
I20260709 13:06:55.879966 24441 ts_tablet_manager.cc:1434] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.000s
I20260709 13:06:55.885852 24440 raft_consensus.cc:359] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } }
I20260709 13:06:55.886585 24440 raft_consensus.cc:385] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:55.886862 24440 raft_consensus.cc:740] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Initialized, Role: FOLLOWER
I20260709 13:06:55.887567 24440 consensus_queue.cc:260] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } }
I20260709 13:06:55.889690 24441 tablet_bootstrap.cc:492] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311: Bootstrap starting.
I20260709 13:06:55.895650 24440 ts_tablet_manager.cc:1434] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.001s
I20260709 13:06:55.898730 24440 tablet_bootstrap.cc:492] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02: Bootstrap starting.
I20260709 13:06:55.898906 24442 tablet_bootstrap.cc:492] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304: No bootstrap required, opened a new log
I20260709 13:06:55.899986 24442 ts_tablet_manager.cc:1403] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304: Time spent bootstrapping tablet: real 0.043s	user 0.014s	sys 0.020s
I20260709 13:06:55.903033 24441 tablet_bootstrap.cc:654] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:55.904423 24442 raft_consensus.cc:359] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.905288 24442 raft_consensus.cc:385] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:55.905539 24440 tablet_bootstrap.cc:654] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:55.905664 24442 raft_consensus.cc:740] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Initialized, Role: FOLLOWER
I20260709 13:06:55.906901 24442 consensus_queue.cc:260] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.910211 24445 raft_consensus.cc:493] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:55.910749 24447 raft_consensus.cc:493] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:55.910857 24445 raft_consensus.cc:515] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.911470 24447 raft_consensus.cc:515] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.916385 24447 leader_election.cc:290] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:06:55.918118 24445 leader_election.cc:290] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:06:55.924885 24441 tablet_bootstrap.cc:492] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311: No bootstrap required, opened a new log
I20260709 13:06:55.925453 24441 ts_tablet_manager.cc:1403] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311: Time spent bootstrapping tablet: real 0.036s	user 0.011s	sys 0.008s
I20260709 13:06:55.935213 24442 ts_tablet_manager.cc:1434] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304: Time spent starting tablet: real 0.035s	user 0.005s	sys 0.001s
I20260709 13:06:55.940490 24442 tablet_bootstrap.cc:492] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304: Bootstrap starting.
I20260709 13:06:55.929072 24441 raft_consensus.cc:359] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.936604 24440 tablet_bootstrap.cc:492] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02: No bootstrap required, opened a new log
I20260709 13:06:55.946664 24441 raft_consensus.cc:385] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:55.946929 24440 ts_tablet_manager.cc:1403] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02: Time spent bootstrapping tablet: real 0.048s	user 0.021s	sys 0.014s
I20260709 13:06:55.947121 24441 raft_consensus.cc:740] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Initialized, Role: FOLLOWER
I20260709 13:06:55.950222 24440 raft_consensus.cc:359] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.951040 24440 raft_consensus.cc:385] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:55.951365 24440 raft_consensus.cc:740] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Initialized, Role: FOLLOWER
I20260709 13:06:55.952028 24441 consensus_queue.cc:260] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.952160 24440 consensus_queue.cc:260] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.954346 24441 ts_tablet_manager.cc:1434] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.001s
I20260709 13:06:55.954614 24440 ts_tablet_manager.cc:1434] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.006s
I20260709 13:06:55.955317 24441 tablet_bootstrap.cc:492] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311: Bootstrap starting.
I20260709 13:06:55.955618 24440 tablet_bootstrap.cc:492] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02: Bootstrap starting.
I20260709 13:06:55.960362 24442 tablet_bootstrap.cc:654] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:55.965013 24447 raft_consensus.cc:493] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:55.965513 24447 raft_consensus.cc:515] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } }
I20260709 13:06:55.967902 24447 leader_election.cc:290] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:06:55.971486 24447 raft_consensus.cc:493] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:55.973383 24447 raft_consensus.cc:515] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:55.975528 24447 leader_election.cc:290] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:06:55.981958 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
W20260709 13:06:55.983899 24360 leader_election.cc:343] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:55.989465 24440 tablet_bootstrap.cc:654] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:55.991791 24248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" is_pre_election: true
I20260709 13:06:55.992586 24248 raft_consensus.cc:2468] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 0.
I20260709 13:06:55.994259 24362 leader_election.cc:304] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 2ae85b9b08454b42a0d9b3059aaa1304
I20260709 13:06:55.995036 24441 tablet_bootstrap.cc:654] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:55.995165 24445 raft_consensus.cc:2804] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:55.995580 24445 raft_consensus.cc:493] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:55.995949 24445 raft_consensus.cc:3060] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:55.997645 24322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83da7aa0dd1047908cfc3f6e16337706" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:06:55.999166 24321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3249533cb3842de91db1dc4cb6a4bcc" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:06:56.000428 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
W20260709 13:06:56.002632 24211 leader_election.cc:343] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:06:56.003315 24211 leader_election.cc:343] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:56.003342 24445 raft_consensus.cc:515] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
W20260709 13:06:56.004057 24211 leader_election.cc:343] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:56.006062 24445 leader_election.cc:290] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:06:56.007417 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:06:56.007885 24248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:06:56.008520 24248 raft_consensus.cc:3060] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:06:56.008631 24360 leader_election.cc:343] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:56.011400 24397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02" is_pre_election: true
I20260709 13:06:56.011639 24396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83da7aa0dd1047908cfc3f6e16337706" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02" is_pre_election: true
I20260709 13:06:56.012264 24396 raft_consensus.cc:2468] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 0.
I20260709 13:06:56.012624 24397 raft_consensus.cc:2393] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in current term 1: Already voted for candidate f450d3cd998e454da3e92d0381791c02 in this term.
I20260709 13:06:56.013494 24211 leader_election.cc:304] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 2ae85b9b08454b42a0d9b3059aaa1304
I20260709 13:06:56.014460 24447 raft_consensus.cc:2804] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:56.014787 24447 raft_consensus.cc:493] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:56.014950 24211 leader_election.cc:304] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311; no voters: 2ae85b9b08454b42a0d9b3059aaa1304, f450d3cd998e454da3e92d0381791c02
I20260709 13:06:56.015156 24447 raft_consensus.cc:3060] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.013470 24395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3249533cb3842de91db1dc4cb6a4bcc" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02" is_pre_election: true
I20260709 13:06:56.016754 24248 raft_consensus.cc:2468] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 1.
I20260709 13:06:56.016923 24395 raft_consensus.cc:2468] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 0.
I20260709 13:06:56.018193 24362 leader_election.cc:304] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 2ae85b9b08454b42a0d9b3059aaa1304
I20260709 13:06:56.019001 24445 raft_consensus.cc:2804] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:56.019857 24211 leader_election.cc:304] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 2ae85b9b08454b42a0d9b3059aaa1304
I20260709 13:06:56.021361 24464 raft_consensus.cc:2749] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:56.022500 24447 raft_consensus.cc:515] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } }
I20260709 13:06:56.023082 24464 raft_consensus.cc:2804] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:56.024221 24464 raft_consensus.cc:493] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:56.025080 24464 raft_consensus.cc:3060] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.028821 24440 tablet_bootstrap.cc:492] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02: No bootstrap required, opened a new log
I20260709 13:06:56.029474 24440 ts_tablet_manager.cc:1403] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02: Time spent bootstrapping tablet: real 0.074s	user 0.018s	sys 0.000s
I20260709 13:06:56.025089 24447 leader_election.cc:290] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 election: Requested vote from peers f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:06:56.037407 24442 tablet_bootstrap.cc:492] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304: No bootstrap required, opened a new log
I20260709 13:06:56.038007 24442 ts_tablet_manager.cc:1403] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304: Time spent bootstrapping tablet: real 0.098s	user 0.010s	sys 0.003s
I20260709 13:06:56.041832 24442 raft_consensus.cc:359] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.042771 24442 raft_consensus.cc:385] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:56.043169 24442 raft_consensus.cc:740] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Initialized, Role: FOLLOWER
I20260709 13:06:56.044296 24447 raft_consensus.cc:493] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:56.044888 24447 raft_consensus.cc:515] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.043246 24445 raft_consensus.cc:697] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Becoming Leader. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Running, Role: LEADER
I20260709 13:06:56.047434 24447 leader_election.cc:290] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:06:56.048818 24445 consensus_queue.cc:237] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.051826 24442 consensus_queue.cc:260] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.054589 24441 tablet_bootstrap.cc:492] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311: No bootstrap required, opened a new log
I20260709 13:06:56.055034 24395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83da7aa0dd1047908cfc3f6e16337706" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:06:56.049836 24440 raft_consensus.cc:359] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.056685 24395 raft_consensus.cc:3060] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.059628 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83da7aa0dd1047908cfc3f6e16337706" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:06:56.060389 24397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9587f033fa334d328ccd9149b5b22bcf" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02" is_pre_election: true
I20260709 13:06:56.060964 24442 ts_tablet_manager.cc:1434] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304: Time spent starting tablet: real 0.023s	user 0.009s	sys 0.000s
I20260709 13:06:56.060889 24321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9587f033fa334d328ccd9149b5b22bcf" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:06:56.058143 24441 ts_tablet_manager.cc:1403] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311: Time spent bootstrapping tablet: real 0.103s	user 0.016s	sys 0.000s
I20260709 13:06:56.061569 24321 raft_consensus.cc:2468] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 0.
I20260709 13:06:56.062436 24442 tablet_bootstrap.cc:492] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304: Bootstrap starting.
W20260709 13:06:56.063544 24211 leader_election.cc:343] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:56.076761 24395 raft_consensus.cc:2468] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 1.
I20260709 13:06:56.062568 24464 raft_consensus.cc:515] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.081085 24441 raft_consensus.cc:359] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.084126 24441 raft_consensus.cc:385] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:56.084432 24441 raft_consensus.cc:740] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Initialized, Role: FOLLOWER
I20260709 13:06:56.084638 24464 leader_election.cc:290] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 election: Requested vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:06:56.057087 24440 raft_consensus.cc:385] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:56.087520 24211 leader_election.cc:304] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311; no voters: 
I20260709 13:06:56.087625 24440 raft_consensus.cc:740] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Initialized, Role: FOLLOWER
I20260709 13:06:56.094872 24440 consensus_queue.cc:260] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.095786 24397 raft_consensus.cc:2468] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 0.
I20260709 13:06:56.087716 24441 consensus_queue.cc:260] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.097942 24441 ts_tablet_manager.cc:1434] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311: Time spent starting tablet: real 0.035s	user 0.004s	sys 0.003s
I20260709 13:06:56.098887 24441 tablet_bootstrap.cc:492] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311: Bootstrap starting.
I20260709 13:06:56.092806 24464 raft_consensus.cc:2804] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:56.102155 24464 raft_consensus.cc:493] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:56.102605 24464 raft_consensus.cc:3060] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.104290 24440 ts_tablet_manager.cc:1434] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02: Time spent starting tablet: real 0.074s	user 0.009s	sys 0.000s
I20260709 13:06:56.105191 24440 tablet_bootstrap.cc:492] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02: Bootstrap starting.
I20260709 13:06:56.109018 24321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3249533cb3842de91db1dc4cb6a4bcc" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:06:56.112396 24397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3249533cb3842de91db1dc4cb6a4bcc" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:06:56.113263 24397 raft_consensus.cc:3060] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.114182 24442 tablet_bootstrap.cc:654] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:56.120100 24440 tablet_bootstrap.cc:654] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:56.124388 24211 leader_election.cc:304] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 2ae85b9b08454b42a0d9b3059aaa1304
I20260709 13:06:56.126097 24441 tablet_bootstrap.cc:654] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:56.127115 24466 raft_consensus.cc:2804] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:56.127771 24466 raft_consensus.cc:697] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Becoming Leader. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Running, Role: LEADER
I20260709 13:06:56.121913 24143 catalog_manager.cc:5697] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 reported cstate change: term changed from 0 to 1, leader changed from <none> to f450d3cd998e454da3e92d0381791c02 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f450d3cd998e454da3e92d0381791c02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:56.129572 24466 consensus_queue.cc:237] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } }
I20260709 13:06:56.132668 24442 tablet_bootstrap.cc:492] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304: No bootstrap required, opened a new log
I20260709 13:06:56.133101 24442 ts_tablet_manager.cc:1403] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304: Time spent bootstrapping tablet: real 0.071s	user 0.012s	sys 0.002s
I20260709 13:06:56.135768 24442 raft_consensus.cc:359] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.136418 24442 raft_consensus.cc:385] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:56.136700 24442 raft_consensus.cc:740] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Initialized, Role: FOLLOWER
I20260709 13:06:56.142709 24445 raft_consensus.cc:493] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:56.143261 24445 raft_consensus.cc:515] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.145684 24445 leader_election.cc:290] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:06:56.146354 24248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" is_pre_election: true
I20260709 13:06:56.146953 24248 raft_consensus.cc:2468] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 0.
I20260709 13:06:56.147148 24321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:06:56.147904 24321 raft_consensus.cc:2468] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 0.
I20260709 13:06:56.148353 24362 leader_election.cc:304] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:06:56.149436 24445 raft_consensus.cc:2804] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:56.149869 24445 raft_consensus.cc:493] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:56.150238 24445 raft_consensus.cc:3060] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:06:56.150915 24211 leader_election.cc:343] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:56.156633 24397 raft_consensus.cc:2468] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 1.
I20260709 13:06:56.159034 24442 consensus_queue.cc:260] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.160053 24211 leader_election.cc:304] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 2ae85b9b08454b42a0d9b3059aaa1304
I20260709 13:06:56.162142 24447 raft_consensus.cc:2804] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:56.140445 24464 raft_consensus.cc:515] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.166643 24321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9587f033fa334d328ccd9149b5b22bcf" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:06:56.167301 24441 tablet_bootstrap.cc:492] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311: No bootstrap required, opened a new log
I20260709 13:06:56.167259 24321 raft_consensus.cc:3060] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.167848 24441 ts_tablet_manager.cc:1403] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311: Time spent bootstrapping tablet: real 0.069s	user 0.019s	sys 0.009s
I20260709 13:06:56.161451 24445 raft_consensus.cc:515] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.172482 24440 tablet_bootstrap.cc:492] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02: No bootstrap required, opened a new log
I20260709 13:06:56.173662 24445 leader_election.cc:290] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:06:56.174147 24447 raft_consensus.cc:697] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Becoming Leader. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Running, Role: LEADER
I20260709 13:06:56.174335 24248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:06:56.174875 24440 ts_tablet_manager.cc:1403] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02: Time spent bootstrapping tablet: real 0.070s	user 0.010s	sys 0.006s
I20260709 13:06:56.174885 24248 raft_consensus.cc:3060] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.174822 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:06:56.175571 24323 raft_consensus.cc:3060] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.176322 24447 consensus_queue.cc:237] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.176100 24442 ts_tablet_manager.cc:1434] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304: Time spent starting tablet: real 0.043s	user 0.003s	sys 0.002s
I20260709 13:06:56.183351 24442 tablet_bootstrap.cc:492] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304: Bootstrap starting.
I20260709 13:06:56.182452 24464 leader_election.cc:290] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 election: Requested vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:06:56.185292 24323 raft_consensus.cc:2468] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 1.
I20260709 13:06:56.186833 24360 leader_election.cc:304] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:06:56.184978 24441 raft_consensus.cc:359] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.187433 24441 raft_consensus.cc:385] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:56.187748 24445 raft_consensus.cc:2804] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:56.187811 24441 raft_consensus.cc:740] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Initialized, Role: FOLLOWER
I20260709 13:06:56.188313 24445 raft_consensus.cc:697] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Becoming Leader. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Running, Role: LEADER
I20260709 13:06:56.182322 24321 raft_consensus.cc:2468] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 1.
I20260709 13:06:56.203577 24248 raft_consensus.cc:2468] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 1.
I20260709 13:06:56.205616 24397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9587f033fa334d328ccd9149b5b22bcf" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:06:56.206204 24397 raft_consensus.cc:3060] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.209723 24445 consensus_queue.cc:237] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.193641 24441 consensus_queue.cc:260] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.213442 24211 leader_election.cc:304] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311; no voters: 
I20260709 13:06:56.217465 24441 ts_tablet_manager.cc:1434] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311: Time spent starting tablet: real 0.049s	user 0.010s	sys 0.000s
I20260709 13:06:56.218469 24441 tablet_bootstrap.cc:492] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311: Bootstrap starting.
I20260709 13:06:56.218211 24440 raft_consensus.cc:359] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.218917 24440 raft_consensus.cc:385] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:56.219199 24440 raft_consensus.cc:740] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Initialized, Role: FOLLOWER
I20260709 13:06:56.220221 24442 tablet_bootstrap.cc:654] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:56.222035 24447 raft_consensus.cc:2804] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:56.222630 24447 raft_consensus.cc:697] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Becoming Leader. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Running, Role: LEADER
I20260709 13:06:56.224257 24397 raft_consensus.cc:2468] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 1.
I20260709 13:06:56.224138 24447 consensus_queue.cc:237] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.222959 24440 consensus_queue.cc:260] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.229959 24465 raft_consensus.cc:493] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:56.230409 24465 raft_consensus.cc:515] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.235656 24465 leader_election.cc:290] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:06:56.238498 24321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:06:56.240209 24441 tablet_bootstrap.cc:654] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:56.242136 24360 leader_election.cc:343] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:56.240250 24248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" is_pre_election: true
I20260709 13:06:56.249341 24464 raft_consensus.cc:493] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:56.250175 24464 raft_consensus.cc:515] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
W20260709 13:06:56.252214 24362 leader_election.cc:343] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:56.252828 24362 leader_election.cc:304] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f450d3cd998e454da3e92d0381791c02; no voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311
I20260709 13:06:56.253759 24465 raft_consensus.cc:2749] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:56.255854 24464 leader_election.cc:290] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:06:56.262352 24321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51f37fd5f5c34b13885b93a9eee3bc8f" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:06:56.264153 24145 catalog_manager.cc:5697] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 reported cstate change: term changed from 0 to 1, leader changed from <none> to b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } }
W20260709 13:06:56.274684 24211 leader_election.cc:343] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:56.282938 24450 raft_consensus.cc:493] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:56.284577 24450 raft_consensus.cc:515] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.287240 24440 ts_tablet_manager.cc:1434] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02: Time spent starting tablet: real 0.112s	user 0.010s	sys 0.001s
I20260709 13:06:56.290041 24440 tablet_bootstrap.cc:492] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02: Bootstrap starting.
I20260709 13:06:56.286095 24397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51f37fd5f5c34b13885b93a9eee3bc8f" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02" is_pre_election: true
W20260709 13:06:56.301383 24211 leader_election.cc:343] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:56.305912 24211 leader_election.cc:304] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311; no voters: 2ae85b9b08454b42a0d9b3059aaa1304, f450d3cd998e454da3e92d0381791c02
I20260709 13:06:56.305502 24450 leader_election.cc:290] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:06:56.308393 24447 raft_consensus.cc:2749] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:56.309902 24442 tablet_bootstrap.cc:492] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304: No bootstrap required, opened a new log
I20260709 13:06:56.313530 24441 tablet_bootstrap.cc:492] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311: No bootstrap required, opened a new log
I20260709 13:06:56.311626 24442 ts_tablet_manager.cc:1403] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304: Time spent bootstrapping tablet: real 0.128s	user 0.016s	sys 0.005s
I20260709 13:06:56.312777 24143 catalog_manager.cc:5697] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 reported cstate change: term changed from 0 to 1, leader changed from <none> to f450d3cd998e454da3e92d0381791c02 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f450d3cd998e454da3e92d0381791c02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:56.314985 24441 ts_tablet_manager.cc:1403] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311: Time spent bootstrapping tablet: real 0.097s	user 0.012s	sys 0.023s
I20260709 13:06:56.319595 24440 tablet_bootstrap.cc:654] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:56.320590 24441 raft_consensus.cc:359] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.322036 24441 raft_consensus.cc:385] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:56.322328 24442 raft_consensus.cc:359] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.323019 24442 raft_consensus.cc:385] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:56.323318 24442 raft_consensus.cc:740] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Initialized, Role: FOLLOWER
I20260709 13:06:56.322357 24441 raft_consensus.cc:740] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Initialized, Role: FOLLOWER
I20260709 13:06:56.324651 24441 consensus_queue.cc:260] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.327018 24441 ts_tablet_manager.cc:1434] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 13:06:56.327970 24441 tablet_bootstrap.cc:492] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311: Bootstrap starting.
I20260709 13:06:56.329437 24442 consensus_queue.cc:260] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.329715 24145 catalog_manager.cc:5697] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 reported cstate change: term changed from 0 to 1, leader changed from <none> to b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:56.331609 24145 catalog_manager.cc:5697] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 reported cstate change: term changed from 0 to 1, leader changed from <none> to b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:56.335541 24441 tablet_bootstrap.cc:654] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:56.338573 24442 ts_tablet_manager.cc:1434] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304: Time spent starting tablet: real 0.024s	user 0.003s	sys 0.003s
I20260709 13:06:56.339546 24442 tablet_bootstrap.cc:492] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304: Bootstrap starting.
I20260709 13:06:56.345944 24248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc38bf75745341a08fa2ca990c4d199d" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" is_pre_election: true
W20260709 13:06:56.352878 24288 leader_election.cc:343] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:56.367444 24397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc38bf75745341a08fa2ca990c4d199d" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02" is_pre_election: true
I20260709 13:06:56.367533 24442 tablet_bootstrap.cc:654] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:56.369165 24286 leader_election.cc:343] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:56.369529 24441 tablet_bootstrap.cc:492] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311: No bootstrap required, opened a new log
I20260709 13:06:56.369644 24286 leader_election.cc:304] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304; no voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02
I20260709 13:06:56.370250 24441 ts_tablet_manager.cc:1403] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311: Time spent bootstrapping tablet: real 0.042s	user 0.019s	sys 0.000s
I20260709 13:06:56.370983 24450 raft_consensus.cc:2749] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:56.372476 24440 tablet_bootstrap.cc:492] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02: No bootstrap required, opened a new log
I20260709 13:06:56.373539 24440 ts_tablet_manager.cc:1403] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02: Time spent bootstrapping tablet: real 0.084s	user 0.010s	sys 0.018s
I20260709 13:06:56.373245 24441 raft_consensus.cc:359] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.375033 24441 raft_consensus.cc:385] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:56.375332 24441 raft_consensus.cc:740] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Initialized, Role: FOLLOWER
I20260709 13:06:56.376170 24441 consensus_queue.cc:260] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.377291 24440 raft_consensus.cc:359] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.378103 24440 raft_consensus.cc:385] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:56.378608 24441 ts_tablet_manager.cc:1434] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:06:56.378469 24440 raft_consensus.cc:740] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Initialized, Role: FOLLOWER
I20260709 13:06:56.379734 24441 tablet_bootstrap.cc:492] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311: Bootstrap starting.
I20260709 13:06:56.379585 24440 consensus_queue.cc:260] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.383319 24440 ts_tablet_manager.cc:1434] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02: Time spent starting tablet: real 0.009s	user 0.001s	sys 0.005s
I20260709 13:06:56.387127 24442 tablet_bootstrap.cc:492] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304: No bootstrap required, opened a new log
I20260709 13:06:56.387806 24442 ts_tablet_manager.cc:1403] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304: Time spent bootstrapping tablet: real 0.048s	user 0.019s	sys 0.000s
I20260709 13:06:56.388741 24440 tablet_bootstrap.cc:492] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02: Bootstrap starting.
I20260709 13:06:56.391355 24442 raft_consensus.cc:359] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.392459 24442 raft_consensus.cc:385] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:56.392840 24442 raft_consensus.cc:740] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Initialized, Role: FOLLOWER
I20260709 13:06:56.393687 24442 consensus_queue.cc:260] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.395542 24440 tablet_bootstrap.cc:654] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:56.396587 24442 ts_tablet_manager.cc:1434] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:06:56.397755 24442 tablet_bootstrap.cc:492] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304: Bootstrap starting.
I20260709 13:06:56.402278 24441 tablet_bootstrap.cc:654] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:56.404892 24442 tablet_bootstrap.cc:654] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:56.414378 24441 tablet_bootstrap.cc:492] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311: No bootstrap required, opened a new log
I20260709 13:06:56.414911 24441 ts_tablet_manager.cc:1403] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311: Time spent bootstrapping tablet: real 0.035s	user 0.017s	sys 0.000s
I20260709 13:06:56.415880 24450 raft_consensus.cc:493] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:56.416460 24450 raft_consensus.cc:515] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.417768 24441 raft_consensus.cc:359] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.418509 24441 raft_consensus.cc:385] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:56.418778 24450 leader_election.cc:290] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:06:56.418880 24441 raft_consensus.cc:740] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Initialized, Role: FOLLOWER
I20260709 13:06:56.419888 24397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3249533cb3842de91db1dc4cb6a4bcc" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02" is_pre_election: true
I20260709 13:06:56.419777 24441 consensus_queue.cc:260] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.420643 24397 raft_consensus.cc:2393] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2ae85b9b08454b42a0d9b3059aaa1304 in current term 1: Already voted for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in this term.
I20260709 13:06:56.420849 24248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3249533cb3842de91db1dc4cb6a4bcc" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" is_pre_election: true
I20260709 13:06:56.423031 24288 leader_election.cc:304] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304; no voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02
I20260709 13:06:56.424053 24442 tablet_bootstrap.cc:492] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304: No bootstrap required, opened a new log
I20260709 13:06:56.424115 24450 raft_consensus.cc:3060] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.427800 24442 ts_tablet_manager.cc:1403] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304: Time spent bootstrapping tablet: real 0.030s	user 0.010s	sys 0.007s
I20260709 13:06:56.429677 24441 ts_tablet_manager.cc:1434] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 13:06:56.432255 24440 tablet_bootstrap.cc:492] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02: No bootstrap required, opened a new log
I20260709 13:06:56.431819 24442 raft_consensus.cc:359] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.432721 24440 ts_tablet_manager.cc:1403] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02: Time spent bootstrapping tablet: real 0.044s	user 0.010s	sys 0.009s
I20260709 13:06:56.432832 24442 raft_consensus.cc:385] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:56.433354 24442 raft_consensus.cc:740] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Initialized, Role: FOLLOWER
I20260709 13:06:56.434366 24442 consensus_queue.cc:260] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.436074 24450 raft_consensus.cc:2749] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:56.439051 24440 raft_consensus.cc:359] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.439741 24440 raft_consensus.cc:385] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:56.440011 24440 raft_consensus.cc:740] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Initialized, Role: FOLLOWER
I20260709 13:06:56.440701 24440 consensus_queue.cc:260] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.444128 24442 ts_tablet_manager.cc:1434] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304: Time spent starting tablet: real 0.016s	user 0.010s	sys 0.003s
I20260709 13:06:56.445286 24442 tablet_bootstrap.cc:492] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304: Bootstrap starting.
I20260709 13:06:56.449963 24440 ts_tablet_manager.cc:1434] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02: Time spent starting tablet: real 0.017s	user 0.004s	sys 0.004s
I20260709 13:06:56.451864 24442 tablet_bootstrap.cc:654] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:56.453747 24440 tablet_bootstrap.cc:492] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02: Bootstrap starting.
I20260709 13:06:56.459246 24442 tablet_bootstrap.cc:492] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304: No bootstrap required, opened a new log
I20260709 13:06:56.459775 24442 ts_tablet_manager.cc:1403] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304: Time spent bootstrapping tablet: real 0.015s	user 0.011s	sys 0.003s
I20260709 13:06:56.460402 24440 tablet_bootstrap.cc:654] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:56.462904 24442 raft_consensus.cc:359] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.463709 24442 raft_consensus.cc:385] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:56.464090 24442 raft_consensus.cc:740] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Initialized, Role: FOLLOWER
I20260709 13:06:56.464828 24442 consensus_queue.cc:260] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.466394 24440 tablet_bootstrap.cc:492] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02: No bootstrap required, opened a new log
I20260709 13:06:56.466811 24440 ts_tablet_manager.cc:1403] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02: Time spent bootstrapping tablet: real 0.013s	user 0.010s	sys 0.002s
I20260709 13:06:56.466816 24442 ts_tablet_manager.cc:1434] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:06:56.467808 24442 tablet_bootstrap.cc:492] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304: Bootstrap starting.
I20260709 13:06:56.469410 24440 raft_consensus.cc:359] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.470085 24440 raft_consensus.cc:385] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:56.470348 24440 raft_consensus.cc:740] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Initialized, Role: FOLLOWER
I20260709 13:06:56.471033 24440 consensus_queue.cc:260] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.472926 24440 ts_tablet_manager.cc:1434] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.002s
I20260709 13:06:56.473316 24442 tablet_bootstrap.cc:654] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:56.478693 24442 tablet_bootstrap.cc:492] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304: No bootstrap required, opened a new log
I20260709 13:06:56.479094 24442 ts_tablet_manager.cc:1403] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.004s
I20260709 13:06:56.481245 24442 raft_consensus.cc:359] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } }
I20260709 13:06:56.481715 24442 raft_consensus.cc:385] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:56.481930 24442 raft_consensus.cc:740] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Initialized, Role: FOLLOWER
I20260709 13:06:56.482523 24442 consensus_queue.cc:260] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } }
I20260709 13:06:56.484488 24442 ts_tablet_manager.cc:1434] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20260709 13:06:56.501860 24321 raft_consensus.cc:3060] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.510648 24481 consensus_queue.cc:1048] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:56.524412 24481 consensus_queue.cc:1048] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:56.526166 24481 raft_consensus.cc:493] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:56.526762 24481 raft_consensus.cc:515] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.531239 24247 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51f37fd5f5c34b13885b93a9eee3bc8f" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" is_pre_election: true
I20260709 13:06:56.531914 24247 raft_consensus.cc:2468] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 0.
I20260709 13:06:56.533520 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51f37fd5f5c34b13885b93a9eee3bc8f" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:06:56.533874 24362 leader_election.cc:304] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:06:56.534179 24323 raft_consensus.cc:2468] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 0.
I20260709 13:06:56.534976 24465 raft_consensus.cc:2804] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:56.535442 24465 raft_consensus.cc:493] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:56.535859 24465 raft_consensus.cc:3060] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.541225 24481 leader_election.cc:290] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:06:56.545547 24465 raft_consensus.cc:515] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.549405 24247 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51f37fd5f5c34b13885b93a9eee3bc8f" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:06:56.550302 24247 raft_consensus.cc:3060] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.552561 24481 raft_consensus.cc:493] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:56.553726 24481 raft_consensus.cc:515] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.556895 24465 leader_election.cc:290] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:06:56.558434 24481 leader_election.cc:290] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:06:56.558766 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51f37fd5f5c34b13885b93a9eee3bc8f" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:06:56.559355 24323 raft_consensus.cc:3060] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.560261 24322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc38bf75745341a08fa2ca990c4d199d" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:06:56.561074 24322 raft_consensus.cc:2468] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 0.
I20260709 13:06:56.562202 24247 raft_consensus.cc:2468] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 1.
I20260709 13:06:56.562916 24360 leader_election.cc:304] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:06:56.564198 24362 leader_election.cc:304] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:06:56.564814 24481 raft_consensus.cc:2804] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:56.565207 24481 raft_consensus.cc:493] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:56.565568 24481 raft_consensus.cc:3060] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.565753 24445 raft_consensus.cc:2804] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:56.566291 24445 raft_consensus.cc:697] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Becoming Leader. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Running, Role: LEADER
I20260709 13:06:56.567154 24445 consensus_queue.cc:237] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.578941 24323 raft_consensus.cc:2468] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 1.
I20260709 13:06:56.581918 24248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc38bf75745341a08fa2ca990c4d199d" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" is_pre_election: true
I20260709 13:06:56.582726 24248 raft_consensus.cc:2468] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 0.
I20260709 13:06:56.584672 24465 raft_consensus.cc:493] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:56.585222 24465 raft_consensus.cc:515] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.587404 24465 leader_election.cc:290] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:06:56.587112 24145 catalog_manager.cc:5697] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 reported cstate change: term changed from 0 to 1, leader changed from <none> to f450d3cd998e454da3e92d0381791c02 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f450d3cd998e454da3e92d0381791c02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:56.588645 24481 raft_consensus.cc:515] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.589216 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d88f7a9bc84cbcb633d2433ddad860" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:06:56.589872 24323 raft_consensus.cc:2468] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 0.
I20260709 13:06:56.591251 24360 leader_election.cc:304] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:06:56.591639 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc38bf75745341a08fa2ca990c4d199d" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:06:56.591768 24247 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc38bf75745341a08fa2ca990c4d199d" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:06:56.592196 24465 raft_consensus.cc:2804] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:56.592468 24247 raft_consensus.cc:3060] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.592700 24465 raft_consensus.cc:493] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:56.592289 24323 raft_consensus.cc:3060] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.593200 24465 raft_consensus.cc:3060] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.599413 24247 raft_consensus.cc:2468] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 1.
I20260709 13:06:56.599984 24481 leader_election.cc:290] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:06:56.600400 24465 raft_consensus.cc:515] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.600737 24323 raft_consensus.cc:2468] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 1.
I20260709 13:06:56.602726 24465 leader_election.cc:290] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:06:56.600695 24362 leader_election.cc:304] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:06:56.605984 24465 raft_consensus.cc:2804] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:56.607257 24465 raft_consensus.cc:697] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Becoming Leader. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Running, Role: LEADER
I20260709 13:06:56.607951 24246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d88f7a9bc84cbcb633d2433ddad860" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:06:56.608593 24246 raft_consensus.cc:3060] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.608320 24465 consensus_queue.cc:237] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.606645 24248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d88f7a9bc84cbcb633d2433ddad860" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" is_pre_election: true
I20260709 13:06:56.616016 24246 raft_consensus.cc:2468] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 1.
I20260709 13:06:56.617038 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d88f7a9bc84cbcb633d2433ddad860" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:06:56.617300 24362 leader_election.cc:304] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:06:56.617679 24323 raft_consensus.cc:3060] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.618274 24481 raft_consensus.cc:2804] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:56.618758 24481 raft_consensus.cc:697] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Becoming Leader. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Running, Role: LEADER
I20260709 13:06:56.619570 24481 consensus_queue.cc:237] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.624534 24323 raft_consensus.cc:2468] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 1.
I20260709 13:06:56.628119 24145 catalog_manager.cc:5697] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 reported cstate change: term changed from 0 to 1, leader changed from <none> to f450d3cd998e454da3e92d0381791c02 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f450d3cd998e454da3e92d0381791c02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:56.638093 24447 raft_consensus.cc:493] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:56.638489 24481 consensus_queue.cc:1048] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:56.638583 24447 raft_consensus.cc:515] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.640610 24447 leader_election.cc:290] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:06:56.642314 24143 catalog_manager.cc:5697] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 reported cstate change: term changed from 0 to 1, leader changed from <none> to f450d3cd998e454da3e92d0381791c02 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f450d3cd998e454da3e92d0381791c02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:56.646597 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:06:56.647207 24323 raft_consensus.cc:2468] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 0.
I20260709 13:06:56.648106 24211 leader_election.cc:304] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311; no voters: 
I20260709 13:06:56.648903 24447 raft_consensus.cc:2804] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:56.649879 24447 raft_consensus.cc:493] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:56.650236 24447 raft_consensus.cc:3060] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.650319 24397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02" is_pre_election: true
I20260709 13:06:56.650943 24397 raft_consensus.cc:2468] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 0.
I20260709 13:06:56.665633 24504 consensus_queue.cc:1048] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:56.673978 24447 raft_consensus.cc:515] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
W20260709 13:06:56.674573 24193 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:56.680136 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:06:56.680765 24323 raft_consensus.cc:3060] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.680951 24397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:06:56.681586 24397 raft_consensus.cc:3060] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.688128 24323 raft_consensus.cc:2468] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 1.
I20260709 13:06:56.689153 24211 leader_election.cc:304] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311; no voters: 
I20260709 13:06:56.689903 24464 raft_consensus.cc:2804] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:56.690294 24447 leader_election.cc:290] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 1 election: Requested vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:06:56.692989 24447 consensus_queue.cc:1048] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:56.699050 24397 raft_consensus.cc:2468] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 1.
I20260709 13:06:56.704865 24447 consensus_queue.cc:1048] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:56.708078 24464 raft_consensus.cc:697] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Becoming Leader. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Running, Role: LEADER
I20260709 13:06:56.709048 24464 consensus_queue.cc:237] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.718163 24323 raft_consensus.cc:3060] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:56.727682 24506 consensus_queue.cc:1048] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:56.738014 24506 consensus_queue.cc:1048] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:56.741400 24143 catalog_manager.cc:5697] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 reported cstate change: term changed from 0 to 1, leader changed from <none> to b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:56.757217 24464 consensus_queue.cc:1048] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:56.778455 24447 consensus_queue.cc:1048] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:56.922343 24395 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f"
dest_uuid: "f450d3cd998e454da3e92d0381791c02"
mode: GRACEFUL
new_leader_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
 from {username='slave'} at 127.0.0.1:36818
I20260709 13:06:56.923026 24395 raft_consensus.cc:606] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Received request to transfer leadership to TS b81d16b3ad324fefa4ad3c3b5f4b3311
I20260709 13:06:56.927961 24395 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a"
dest_uuid: "f450d3cd998e454da3e92d0381791c02"
mode: GRACEFUL
new_leader_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
 from {username='slave'} at 127.0.0.1:36818
I20260709 13:06:56.928555 24395 raft_consensus.cc:606] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Received request to transfer leadership to TS b81d16b3ad324fefa4ad3c3b5f4b3311
I20260709 13:06:56.942490 24465 raft_consensus.cc:993] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02: : Instructing follower b81d16b3ad324fefa4ad3c3b5f4b3311 to start an election
I20260709 13:06:56.943015 24465 raft_consensus.cc:1081] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Signalling peer b81d16b3ad324fefa4ad3c3b5f4b3311 to start an election
I20260709 13:06:56.944677 24246 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f"
dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
 from {username='slave'} at 127.0.0.1:39428
I20260709 13:06:56.945259 24246 raft_consensus.cc:493] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:56.945602 24246 raft_consensus.cc:3060] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:56.949469 24247 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9587f033fa334d328ccd9149b5b22bcf"
dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
mode: GRACEFUL
new_leader_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
 from {username='slave'} at 127.0.0.1:39442
I20260709 13:06:56.950037 24247 raft_consensus.cc:606] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Received request to transfer leadership to TS 2ae85b9b08454b42a0d9b3059aaa1304
I20260709 13:06:56.951346 24246 raft_consensus.cc:515] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.954274 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:06:56.954793 24323 raft_consensus.cc:3060] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:56.955436 24247 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d"
dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
mode: GRACEFUL
new_leader_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
 from {username='slave'} at 127.0.0.1:39442
I20260709 13:06:56.955507 24395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:06:56.956101 24247 raft_consensus.cc:606] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Received request to transfer leadership to TS 2ae85b9b08454b42a0d9b3059aaa1304
I20260709 13:06:56.956120 24395 raft_consensus.cc:3055] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:56.956612 24395 raft_consensus.cc:740] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Running, Role: LEADER
I20260709 13:06:56.957509 24395 consensus_queue.cc:260] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.958649 24395 raft_consensus.cc:3060] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:56.961297 24323 raft_consensus.cc:2468] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 2.
I20260709 13:06:56.962602 24211 leader_election.cc:304] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311; no voters: 
I20260709 13:06:56.963356 24447 raft_consensus.cc:2804] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:56.963801 24447 raft_consensus.cc:697] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 LEADER]: Becoming Leader. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Running, Role: LEADER
I20260709 13:06:56.964864 24395 raft_consensus.cc:2468] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 2.
I20260709 13:06:56.964579 24447 consensus_queue.cc:237] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:56.956498 24246 leader_election.cc:290] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 2 election: Requested vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:06:56.969962 24395 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "76d88f7a9bc84cbcb633d2433ddad860"
dest_uuid: "f450d3cd998e454da3e92d0381791c02"
mode: GRACEFUL
new_leader_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
 from {username='slave'} at 127.0.0.1:36818
I20260709 13:06:56.970528 24395 raft_consensus.cc:606] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Received request to transfer leadership to TS 2ae85b9b08454b42a0d9b3059aaa1304
I20260709 13:06:56.972345 24145 catalog_manager.cc:5697] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 reported cstate change: term changed from 1 to 2, leader changed from f450d3cd998e454da3e92d0381791c02 (127.23.140.67) to b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:56.975791 24395 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "51f37fd5f5c34b13885b93a9eee3bc8f"
dest_uuid: "f450d3cd998e454da3e92d0381791c02"
mode: GRACEFUL
new_leader_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
 from {username='slave'} at 127.0.0.1:36818
I20260709 13:06:56.976260 24395 raft_consensus.cc:606] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Received request to transfer leadership to TS 2ae85b9b08454b42a0d9b3059aaa1304
I20260709 13:06:56.998909 24504 consensus_queue.cc:1048] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:57.011404 24445 raft_consensus.cc:993] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02: : Instructing follower 2ae85b9b08454b42a0d9b3059aaa1304 to start an election
I20260709 13:06:57.011968 24481 raft_consensus.cc:1081] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Signalling peer 2ae85b9b08454b42a0d9b3059aaa1304 to start an election
I20260709 13:06:57.016965 24321 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "51f37fd5f5c34b13885b93a9eee3bc8f"
dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
 from {username='slave'} at 127.0.0.1:56324
I20260709 13:06:57.017539 24321 raft_consensus.cc:493] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:57.017910 24321 raft_consensus.cc:3060] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:57.019135 24445 consensus_queue.cc:1048] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:57.025151 24321 raft_consensus.cc:515] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:57.027726 24247 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51f37fd5f5c34b13885b93a9eee3bc8f" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:06:57.028003 24395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51f37fd5f5c34b13885b93a9eee3bc8f" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:06:57.028513 24395 raft_consensus.cc:3055] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:57.028836 24395 raft_consensus.cc:740] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Running, Role: LEADER
I20260709 13:06:57.029645 24395 consensus_queue.cc:260] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:57.030329 24395 raft_consensus.cc:3060] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:57.033272 24321 leader_election.cc:290] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 2 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:06:57.033816 24323 raft_consensus.cc:1240] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Rejecting Update request from peer f450d3cd998e454da3e92d0381791c02 for earlier term 1. Current term is 2. Ops: []
I20260709 13:06:57.040004 24395 raft_consensus.cc:2468] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae85b9b08454b42a0d9b3059aaa1304 in term 2.
I20260709 13:06:57.041119 24286 leader_election.cc:304] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, f450d3cd998e454da3e92d0381791c02; no voters: b81d16b3ad324fefa4ad3c3b5f4b3311
I20260709 13:06:57.041844 24491 raft_consensus.cc:2804] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:57.045616 24491 raft_consensus.cc:697] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Becoming Leader. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Running, Role: LEADER
I20260709 13:06:57.046440 24491 consensus_queue.cc:237] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:57.053198 24145 catalog_manager.cc:5697] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 reported cstate change: term changed from 1 to 2, leader changed from f450d3cd998e454da3e92d0381791c02 (127.23.140.67) to 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:57.156343 24447 consensus_queue.cc:1048] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:57.159157 24481 consensus_queue.cc:1048] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:57.174364 24506 consensus_queue.cc:1048] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:57.175522 24504 consensus_queue.cc:1048] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:57.196764 24504 raft_consensus.cc:993] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02: : Instructing follower 2ae85b9b08454b42a0d9b3059aaa1304 to start an election
I20260709 13:06:57.197558 24504 raft_consensus.cc:1081] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Signalling peer 2ae85b9b08454b42a0d9b3059aaa1304 to start an election
I20260709 13:06:57.198377 24445 consensus_queue.cc:1048] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:57.200366 24510 raft_consensus.cc:993] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311: : Instructing follower 2ae85b9b08454b42a0d9b3059aaa1304 to start an election
I20260709 13:06:57.200812 24506 raft_consensus.cc:1081] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Signalling peer 2ae85b9b08454b42a0d9b3059aaa1304 to start an election
I20260709 13:06:57.203075 24321 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d"
dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
 from {username='slave'} at 127.0.0.1:56314
I20260709 13:06:57.203601 24321 raft_consensus.cc:493] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:57.203995 24321 raft_consensus.cc:3060] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:57.205355 24323 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "76d88f7a9bc84cbcb633d2433ddad860"
dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
 from {username='slave'} at 127.0.0.1:56324
I20260709 13:06:57.205878 24323 raft_consensus.cc:493] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:57.206234 24323 raft_consensus.cc:3060] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:57.211190 24321 raft_consensus.cc:515] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:57.218119 24247 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:06:57.218721 24247 raft_consensus.cc:3055] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:57.219046 24247 raft_consensus.cc:740] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Running, Role: LEADER
I20260709 13:06:57.219827 24247 consensus_queue.cc:260] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:57.220479 24323 raft_consensus.cc:515] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:57.225764 24248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d88f7a9bc84cbcb633d2433ddad860" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:06:57.225803 24504 raft_consensus.cc:993] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02: : Instructing follower b81d16b3ad324fefa4ad3c3b5f4b3311 to start an election
I20260709 13:06:57.226655 24504 raft_consensus.cc:1081] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Signalling peer b81d16b3ad324fefa4ad3c3b5f4b3311 to start an election
I20260709 13:06:57.227577 24323 leader_election.cc:290] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 2 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:06:57.230443 24245 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a"
dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
 from {username='slave'} at 127.0.0.1:39428
I20260709 13:06:57.231223 24245 raft_consensus.cc:493] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:57.231665 24245 raft_consensus.cc:3060] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:57.226576 24248 raft_consensus.cc:3060] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:57.237391 24395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d88f7a9bc84cbcb633d2433ddad860" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:06:57.238022 24395 raft_consensus.cc:3055] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:57.238341 24395 raft_consensus.cc:740] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Running, Role: LEADER
I20260709 13:06:57.239275 24395 consensus_queue.cc:260] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:57.240645 24248 raft_consensus.cc:2468] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae85b9b08454b42a0d9b3059aaa1304 in term 2.
I20260709 13:06:57.237402 24247 raft_consensus.cc:3060] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:57.240835 24395 raft_consensus.cc:3060] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:57.242285 24504 consensus_queue.cc:1048] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:57.242884 24288 leader_election.cc:304] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311; no voters: 
I20260709 13:06:57.243733 24491 raft_consensus.cc:2804] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:57.244189 24491 raft_consensus.cc:697] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Becoming Leader. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Running, Role: LEADER
I20260709 13:06:57.245524 24396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:06:57.246110 24396 raft_consensus.cc:3060] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:57.241588 24321 leader_election.cc:290] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 2 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:06:57.244974 24491 consensus_queue.cc:237] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:57.255496 24396 raft_consensus.cc:2468] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae85b9b08454b42a0d9b3059aaa1304 in term 2.
I20260709 13:06:57.256644 24247 raft_consensus.cc:2468] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae85b9b08454b42a0d9b3059aaa1304 in term 2.
I20260709 13:06:57.259275 24286 leader_election.cc:304] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:06:57.257886 24245 raft_consensus.cc:515] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:57.249791 24395 raft_consensus.cc:2468] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae85b9b08454b42a0d9b3059aaa1304 in term 2.
I20260709 13:06:57.260142 24491 raft_consensus.cc:2804] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:57.260648 24491 raft_consensus.cc:697] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Becoming Leader. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Running, Role: LEADER
I20260709 13:06:57.261525 24491 consensus_queue.cc:237] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:57.282069 24245 leader_election.cc:290] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 2 election: Requested vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:06:57.284397 24321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:06:57.285547 24395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:06:57.286191 24395 raft_consensus.cc:3055] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:57.286545 24395 raft_consensus.cc:740] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Running, Role: LEADER
I20260709 13:06:57.287436 24395 consensus_queue.cc:260] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:57.288792 24395 raft_consensus.cc:3060] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:57.289891 24321 raft_consensus.cc:3060] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:57.295265 24395 raft_consensus.cc:2468] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 2.
I20260709 13:06:57.296622 24211 leader_election.cc:304] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:06:57.297528 24506 raft_consensus.cc:2804] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:57.298055 24506 raft_consensus.cc:697] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 LEADER]: Becoming Leader. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Running, Role: LEADER
I20260709 13:06:57.298842 24506 consensus_queue.cc:237] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:57.302551 24321 raft_consensus.cc:2468] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 2.
I20260709 13:06:57.302886 24145 catalog_manager.cc:5697] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 reported cstate change: term changed from 1 to 2, leader changed from f450d3cd998e454da3e92d0381791c02 (127.23.140.67) to 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:57.312885 24144 catalog_manager.cc:5697] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 reported cstate change: term changed from 1 to 2, leader changed from f450d3cd998e454da3e92d0381791c02 (127.23.140.67) to b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:57.320489 24143 catalog_manager.cc:5697] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 reported cstate change: term changed from 1 to 2, leader changed from b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65) to 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:57.325520 24447 raft_consensus.cc:993] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311: : Instructing follower 2ae85b9b08454b42a0d9b3059aaa1304 to start an election
I20260709 13:06:57.325985 24510 raft_consensus.cc:1081] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Signalling peer 2ae85b9b08454b42a0d9b3059aaa1304 to start an election
I20260709 13:06:57.327595 24322 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9587f033fa334d328ccd9149b5b22bcf"
dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
 from {username='slave'} at 127.0.0.1:56314
I20260709 13:06:57.328114 24322 raft_consensus.cc:493] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:57.328454 24322 raft_consensus.cc:3060] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:57.336268 24322 raft_consensus.cc:515] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:57.338529 24322 leader_election.cc:290] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 2 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:06:57.339564 24395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9587f033fa334d328ccd9149b5b22bcf" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:06:57.340133 24395 raft_consensus.cc:3060] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:57.340545 24247 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9587f033fa334d328ccd9149b5b22bcf" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:06:57.341089 24247 raft_consensus.cc:3055] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:57.341435 24247 raft_consensus.cc:740] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Running, Role: LEADER
I20260709 13:06:57.342218 24247 consensus_queue.cc:260] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:57.343410 24247 raft_consensus.cc:3060] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:57.346881 24395 raft_consensus.cc:2468] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae85b9b08454b42a0d9b3059aaa1304 in term 2.
I20260709 13:06:57.348232 24286 leader_election.cc:304] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:06:57.349113 24491 raft_consensus.cc:2804] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:57.349599 24491 raft_consensus.cc:697] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Becoming Leader. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Running, Role: LEADER
I20260709 13:06:57.349974 24247 raft_consensus.cc:2468] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae85b9b08454b42a0d9b3059aaa1304 in term 2.
I20260709 13:06:57.350445 24491 consensus_queue.cc:237] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:06:57.359122 24143 catalog_manager.cc:5697] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 reported cstate change: term changed from 1 to 2, leader changed from b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65) to 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:57.364085 24322 raft_consensus.cc:1275] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Refusing update from remote peer b81d16b3ad324fefa4ad3c3b5f4b3311: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:57.365444 24447 consensus_queue.cc:1048] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:57.377600 24395 raft_consensus.cc:1275] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Refusing update from remote peer b81d16b3ad324fefa4ad3c3b5f4b3311: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:57.379096 24510 consensus_queue.cc:1048] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:57.515244 24395 raft_consensus.cc:1275] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Refusing update from remote peer 2ae85b9b08454b42a0d9b3059aaa1304: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:57.516726 24491 consensus_queue.cc:1048] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:57.527408 24247 raft_consensus.cc:3060] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:57.537593 24247 raft_consensus.cc:1275] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Refusing update from remote peer 2ae85b9b08454b42a0d9b3059aaa1304: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:57.540496 24528 consensus_queue.cc:1048] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:57.646410 24395 raft_consensus.cc:1275] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Refusing update from remote peer 2ae85b9b08454b42a0d9b3059aaa1304: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:57.647696 24528 consensus_queue.cc:1048] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:57.649966 24247 raft_consensus.cc:1275] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Refusing update from remote peer 2ae85b9b08454b42a0d9b3059aaa1304: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:57.651350 24528 consensus_queue.cc:1048] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:57.655606 24245 raft_consensus.cc:1275] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Refusing update from remote peer 2ae85b9b08454b42a0d9b3059aaa1304: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:57.658260 24528 consensus_queue.cc:1048] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:57.665750 24395 raft_consensus.cc:1275] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Refusing update from remote peer 2ae85b9b08454b42a0d9b3059aaa1304: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:57.667289 24491 consensus_queue.cc:1048] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:57.753290 24245 raft_consensus.cc:1275] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Refusing update from remote peer 2ae85b9b08454b42a0d9b3059aaa1304: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:57.754535 24491 consensus_queue.cc:1048] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:57.761232 24395 raft_consensus.cc:1275] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Refusing update from remote peer 2ae85b9b08454b42a0d9b3059aaa1304: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:57.763190 24520 consensus_queue.cc:1048] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:57.819134 24322 raft_consensus.cc:1275] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Refusing update from remote peer b81d16b3ad324fefa4ad3c3b5f4b3311: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:57.820457 24510 consensus_queue.cc:1048] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:57.828837 24395 raft_consensus.cc:1275] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Refusing update from remote peer b81d16b3ad324fefa4ad3c3b5f4b3311: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:57.830650 24506 consensus_queue.cc:1048] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:00.008222 24113 auto_leader_rebalancer-test.cc:316] The leader distribution is 
I20260709 13:07:00.009037 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:00.030213 24245 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a"
dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
mode: GRACEFUL
new_leader_uuid: "f450d3cd998e454da3e92d0381791c02"
 from {username='slave'} at 127.0.0.1:39444
I20260709 13:07:00.030740 24245 raft_consensus.cc:606] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 LEADER]: Received request to transfer leadership to TS f450d3cd998e454da3e92d0381791c02
I20260709 13:07:00.041318 24322 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d"
dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
mode: GRACEFUL
new_leader_uuid: "f450d3cd998e454da3e92d0381791c02"
 from {username='slave'} at 127.0.0.1:56340
I20260709 13:07:00.041785 24322 raft_consensus.cc:606] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Received request to transfer leadership to TS f450d3cd998e454da3e92d0381791c02
I20260709 13:07:00.042856 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:07:00.043310 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:07:00.099560 24520 raft_consensus.cc:993] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304: : Instructing follower f450d3cd998e454da3e92d0381791c02 to start an election
I20260709 13:07:00.100019 24531 raft_consensus.cc:1081] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Signalling peer f450d3cd998e454da3e92d0381791c02 to start an election
I20260709 13:07:00.101490 24396 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d"
dest_uuid: "f450d3cd998e454da3e92d0381791c02"
 from {username='slave'} at 127.0.0.1:36802
I20260709 13:07:00.101910 24396 raft_consensus.cc:493] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:00.102151 24396 raft_consensus.cc:3060] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:00.106369 24396 raft_consensus.cc:515] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:00.107868 24396 leader_election.cc:290] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 3 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:07:00.108876 24322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:07:00.108812 24245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:07:00.109647 24245 raft_consensus.cc:3060] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:00.109607 24322 raft_consensus.cc:3055] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:00.110188 24322 raft_consensus.cc:740] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Running, Role: LEADER
I20260709 13:07:00.111104 24322 consensus_queue.cc:260] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:00.112397 24322 raft_consensus.cc:3060] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:00.116909 24245 raft_consensus.cc:2468] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 3.
I20260709 13:07:00.118256 24362 leader_election.cc:304] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:07:00.118958 24504 raft_consensus.cc:2804] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:00.119294 24322 raft_consensus.cc:2468] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 3.
I20260709 13:07:00.120366 24504 raft_consensus.cc:697] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 3 LEADER]: Becoming Leader. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Running, Role: LEADER
I20260709 13:07:00.121320 24504 consensus_queue.cc:237] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:00.128211 24144 catalog_manager.cc:5697] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 reported cstate change: term changed from 2 to 3, leader changed from 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66) to f450d3cd998e454da3e92d0381791c02 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "f450d3cd998e454da3e92d0381791c02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:00.257339 24506 raft_consensus.cc:993] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311: : Instructing follower f450d3cd998e454da3e92d0381791c02 to start an election
I20260709 13:07:00.257871 24506 raft_consensus.cc:1081] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 LEADER]: Signalling peer f450d3cd998e454da3e92d0381791c02 to start an election
I20260709 13:07:00.259364 24396 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a"
dest_uuid: "f450d3cd998e454da3e92d0381791c02"
 from {username='slave'} at 127.0.0.1:36786
I20260709 13:07:00.259840 24396 raft_consensus.cc:493] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:00.260142 24396 raft_consensus.cc:3060] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:00.265115 24396 raft_consensus.cc:515] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:00.266814 24396 leader_election.cc:290] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 3 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:07:00.267423 24245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:07:00.267774 24322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:07:00.267993 24245 raft_consensus.cc:3055] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:00.268301 24322 raft_consensus.cc:3060] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:00.268330 24245 raft_consensus.cc:740] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Running, Role: LEADER
I20260709 13:07:00.269215 24245 consensus_queue.cc:260] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:00.270097 24245 raft_consensus.cc:3060] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:00.272746 24322 raft_consensus.cc:2468] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 3.
I20260709 13:07:00.273967 24360 leader_election.cc:304] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:07:00.274670 24245 raft_consensus.cc:2468] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 3.
I20260709 13:07:00.275259 24504 raft_consensus.cc:2804] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:00.275761 24504 raft_consensus.cc:697] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 3 LEADER]: Becoming Leader. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Running, Role: LEADER
I20260709 13:07:00.276579 24504 consensus_queue.cc:237] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:00.286131 24144 catalog_manager.cc:5697] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 reported cstate change: term changed from 2 to 3, leader changed from b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65) to f450d3cd998e454da3e92d0381791c02 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "f450d3cd998e454da3e92d0381791c02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:00.662177 24245 raft_consensus.cc:1275] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Refusing update from remote peer f450d3cd998e454da3e92d0381791c02: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:00.663328 24504 consensus_queue.cc:1048] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:00.674306 24322 raft_consensus.cc:1275] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Refusing update from remote peer f450d3cd998e454da3e92d0381791c02: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:00.675751 24504 consensus_queue.cc:1048] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:00.716786 24322 raft_consensus.cc:1275] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Refusing update from remote peer f450d3cd998e454da3e92d0381791c02: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:00.718135 24504 consensus_queue.cc:1048] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:00.727823 24245 raft_consensus.cc:1275] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Refusing update from remote peer f450d3cd998e454da3e92d0381791c02: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:00.729949 24504 consensus_queue.cc:1048] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:01.044030 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:01.053582 24245 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "83da7aa0dd1047908cfc3f6e16337706"
dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
mode: GRACEFUL
new_leader_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
 from {username='slave'} at 127.0.0.1:39442
I20260709 13:07:01.054049 24245 raft_consensus.cc:606] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Received request to transfer leadership to TS 2ae85b9b08454b42a0d9b3059aaa1304
I20260709 13:07:01.058672 24245 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f"
dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
mode: GRACEFUL
new_leader_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
 from {username='slave'} at 127.0.0.1:39442
I20260709 13:07:01.059176 24245 raft_consensus.cc:606] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 LEADER]: Received request to transfer leadership to TS 2ae85b9b08454b42a0d9b3059aaa1304
I20260709 13:07:01.063879 24396 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a"
dest_uuid: "f450d3cd998e454da3e92d0381791c02"
mode: GRACEFUL
new_leader_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
 from {username='slave'} at 127.0.0.1:36818
I20260709 13:07:01.064333 24396 raft_consensus.cc:606] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 3 LEADER]: Received request to transfer leadership to TS 2ae85b9b08454b42a0d9b3059aaa1304
I20260709 13:07:01.068773 24245 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b3249533cb3842de91db1dc4cb6a4bcc"
dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
mode: GRACEFUL
new_leader_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
 from {username='slave'} at 127.0.0.1:39442
I20260709 13:07:01.069284 24245 raft_consensus.cc:606] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Received request to transfer leadership to TS 2ae85b9b08454b42a0d9b3059aaa1304
I20260709 13:07:01.076184 24396 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d"
dest_uuid: "f450d3cd998e454da3e92d0381791c02"
mode: GRACEFUL
new_leader_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
 from {username='slave'} at 127.0.0.1:36818
I20260709 13:07:01.076714 24396 raft_consensus.cc:606] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 3 LEADER]: Received request to transfer leadership to TS 2ae85b9b08454b42a0d9b3059aaa1304
I20260709 13:07:01.199530 24535 raft_consensus.cc:993] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02: : Instructing follower 2ae85b9b08454b42a0d9b3059aaa1304 to start an election
I20260709 13:07:01.199995 24549 raft_consensus.cc:1081] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 3 LEADER]: Signalling peer 2ae85b9b08454b42a0d9b3059aaa1304 to start an election
I20260709 13:07:01.201390 24322 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a"
dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
 from {username='slave'} at 127.0.0.1:56324
I20260709 13:07:01.201817 24322 raft_consensus.cc:493] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:01.202071 24322 raft_consensus.cc:3060] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:01.206221 24322 raft_consensus.cc:515] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:01.207847 24322 leader_election.cc:290] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 4 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:07:01.208612 24245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:07:01.208873 24396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:07:01.209120 24245 raft_consensus.cc:3060] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:01.209501 24396 raft_consensus.cc:3055] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:07:01.209790 24396 raft_consensus.cc:740] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Running, Role: LEADER
I20260709 13:07:01.210429 24396 consensus_queue.cc:260] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:01.211306 24396 raft_consensus.cc:3060] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:01.213901 24245 raft_consensus.cc:2468] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae85b9b08454b42a0d9b3059aaa1304 in term 4.
I20260709 13:07:01.214990 24288 leader_election.cc:304] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311; no voters: 
I20260709 13:07:01.215760 24520 raft_consensus.cc:2804] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:07:01.216122 24396 raft_consensus.cc:2468] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae85b9b08454b42a0d9b3059aaa1304 in term 4.
I20260709 13:07:01.216951 24520 raft_consensus.cc:697] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 4 LEADER]: Becoming Leader. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Running, Role: LEADER
I20260709 13:07:01.217844 24520 consensus_queue.cc:237] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:01.223707 24144 catalog_manager.cc:5697] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 reported cstate change: term changed from 3 to 4, leader changed from f450d3cd998e454da3e92d0381791c02 (127.23.140.67) to 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66). New cstate: current_term: 4 leader_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:01.273285 24549 raft_consensus.cc:993] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02: : Instructing follower 2ae85b9b08454b42a0d9b3059aaa1304 to start an election
I20260709 13:07:01.273761 24555 raft_consensus.cc:1081] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 3 LEADER]: Signalling peer 2ae85b9b08454b42a0d9b3059aaa1304 to start an election
I20260709 13:07:01.275379 24322 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d"
dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
 from {username='slave'} at 127.0.0.1:56324
I20260709 13:07:01.275915 24322 raft_consensus.cc:493] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:01.276212 24322 raft_consensus.cc:3060] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:01.282112 24322 raft_consensus.cc:515] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:01.283573 24322 leader_election.cc:290] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 4 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:07:01.284377 24245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:07:01.284569 24396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:07:01.284930 24245 raft_consensus.cc:3060] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:01.285081 24396 raft_consensus.cc:3055] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:07:01.285516 24396 raft_consensus.cc:740] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Running, Role: LEADER
I20260709 13:07:01.286235 24396 consensus_queue.cc:260] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:01.287107 24396 raft_consensus.cc:3060] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:01.289549 24245 raft_consensus.cc:2468] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae85b9b08454b42a0d9b3059aaa1304 in term 4.
I20260709 13:07:01.290506 24288 leader_election.cc:304] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311; no voters: 
I20260709 13:07:01.291145 24520 raft_consensus.cc:2804] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:07:01.291617 24520 raft_consensus.cc:697] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 4 LEADER]: Becoming Leader. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Running, Role: LEADER
I20260709 13:07:01.291934 24396 raft_consensus.cc:2468] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae85b9b08454b42a0d9b3059aaa1304 in term 4.
I20260709 13:07:01.292497 24520 consensus_queue.cc:237] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:01.298030 24144 catalog_manager.cc:5697] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 reported cstate change: term changed from 3 to 4, leader changed from f450d3cd998e454da3e92d0381791c02 (127.23.140.67) to 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66). New cstate: current_term: 4 leader_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:01.329099 24506 raft_consensus.cc:993] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311: : Instructing follower 2ae85b9b08454b42a0d9b3059aaa1304 to start an election
I20260709 13:07:01.329548 24556 raft_consensus.cc:1081] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Signalling peer 2ae85b9b08454b42a0d9b3059aaa1304 to start an election
I20260709 13:07:01.330960 24322 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b3249533cb3842de91db1dc4cb6a4bcc"
dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
 from {username='slave'} at 127.0.0.1:56314
I20260709 13:07:01.331434 24322 raft_consensus.cc:493] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:01.331737 24322 raft_consensus.cc:3060] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:01.336059 24322 raft_consensus.cc:515] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:01.337553 24322 leader_election.cc:290] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 2 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:07:01.338435 24245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3249533cb3842de91db1dc4cb6a4bcc" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:07:01.338619 24396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3249533cb3842de91db1dc4cb6a4bcc" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:07:01.338922 24245 raft_consensus.cc:3055] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:01.339138 24396 raft_consensus.cc:3060] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:01.339249 24245 raft_consensus.cc:740] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Running, Role: LEADER
I20260709 13:07:01.339989 24245 consensus_queue.cc:260] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:01.340844 24245 raft_consensus.cc:3060] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:01.343830 24396 raft_consensus.cc:2468] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae85b9b08454b42a0d9b3059aaa1304 in term 2.
I20260709 13:07:01.344875 24286 leader_election.cc:304] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:07:01.345326 24245 raft_consensus.cc:2468] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae85b9b08454b42a0d9b3059aaa1304 in term 2.
I20260709 13:07:01.345716 24520 raft_consensus.cc:2804] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:01.346182 24520 raft_consensus.cc:697] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Becoming Leader. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Running, Role: LEADER
I20260709 13:07:01.346941 24520 consensus_queue.cc:237] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:01.353646 24144 catalog_manager.cc:5697] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 reported cstate change: term changed from 1 to 2, leader changed from b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65) to 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:01.354462 24506 raft_consensus.cc:993] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311: : Instructing follower 2ae85b9b08454b42a0d9b3059aaa1304 to start an election
I20260709 13:07:01.354871 24556 raft_consensus.cc:1081] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Signalling peer 2ae85b9b08454b42a0d9b3059aaa1304 to start an election
I20260709 13:07:01.356187 24322 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "83da7aa0dd1047908cfc3f6e16337706"
dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
 from {username='slave'} at 127.0.0.1:56314
I20260709 13:07:01.356623 24322 raft_consensus.cc:493] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:01.356899 24322 raft_consensus.cc:3060] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:01.362176 24322 raft_consensus.cc:515] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } }
I20260709 13:07:01.363654 24322 leader_election.cc:290] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 2 election: Requested vote from peers f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329), b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615)
I20260709 13:07:01.364347 24396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83da7aa0dd1047908cfc3f6e16337706" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:07:01.364588 24245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83da7aa0dd1047908cfc3f6e16337706" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:07:01.364918 24396 raft_consensus.cc:3060] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:01.365145 24245 raft_consensus.cc:3055] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:01.365592 24245 raft_consensus.cc:740] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Running, Role: LEADER
I20260709 13:07:01.366219 24245 consensus_queue.cc:260] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } }
I20260709 13:07:01.367051 24245 raft_consensus.cc:3060] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:01.369632 24396 raft_consensus.cc:2468] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae85b9b08454b42a0d9b3059aaa1304 in term 2.
I20260709 13:07:01.370533 24286 leader_election.cc:304] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:07:01.371193 24520 raft_consensus.cc:2804] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:01.371567 24245 raft_consensus.cc:2468] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae85b9b08454b42a0d9b3059aaa1304 in term 2.
I20260709 13:07:01.371570 24520 raft_consensus.cc:697] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Becoming Leader. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Running, Role: LEADER
I20260709 13:07:01.372592 24520 consensus_queue.cc:237] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } }
I20260709 13:07:01.378693 24144 catalog_manager.cc:5697] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 reported cstate change: term changed from 1 to 2, leader changed from b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65) to 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:01.492050 24506 raft_consensus.cc:993] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311: : Instructing follower 2ae85b9b08454b42a0d9b3059aaa1304 to start an election
I20260709 13:07:01.492491 24556 raft_consensus.cc:1081] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 LEADER]: Signalling peer 2ae85b9b08454b42a0d9b3059aaa1304 to start an election
I20260709 13:07:01.493943 24322 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f"
dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
 from {username='slave'} at 127.0.0.1:56314
I20260709 13:07:01.494480 24322 raft_consensus.cc:493] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:01.494820 24322 raft_consensus.cc:3060] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:01.499580 24322 raft_consensus.cc:515] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:01.501453 24322 leader_election.cc:290] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 3 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:07:01.502206 24245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:07:01.502594 24396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:07:01.502739 24245 raft_consensus.cc:3055] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:01.503127 24245 raft_consensus.cc:740] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Running, Role: LEADER
I20260709 13:07:01.503159 24396 raft_consensus.cc:3060] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:01.503876 24245 consensus_queue.cc:260] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:01.504741 24245 raft_consensus.cc:3060] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:01.508222 24396 raft_consensus.cc:2468] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae85b9b08454b42a0d9b3059aaa1304 in term 3.
I20260709 13:07:01.509418 24286 leader_election.cc:304] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:07:01.509627 24245 raft_consensus.cc:2468] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae85b9b08454b42a0d9b3059aaa1304 in term 3.
I20260709 13:07:01.510262 24520 raft_consensus.cc:2804] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:01.510807 24520 raft_consensus.cc:697] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 LEADER]: Becoming Leader. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Running, Role: LEADER
I20260709 13:07:01.512109 24520 consensus_queue.cc:237] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:01.519219 24144 catalog_manager.cc:5697] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 reported cstate change: term changed from 2 to 3, leader changed from b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65) to 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66). New cstate: current_term: 3 leader_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:01.653980 24396 raft_consensus.cc:1275] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Refusing update from remote peer 2ae85b9b08454b42a0d9b3059aaa1304: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:07:01.655176 24520 consensus_queue.cc:1048] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:01.666254 24247 raft_consensus.cc:1275] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 FOLLOWER]: Refusing update from remote peer 2ae85b9b08454b42a0d9b3059aaa1304: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:07:01.667701 24561 consensus_queue.cc:1048] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:01.682415 24247 raft_consensus.cc:1275] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 FOLLOWER]: Refusing update from remote peer 2ae85b9b08454b42a0d9b3059aaa1304: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:07:01.683749 24561 consensus_queue.cc:1048] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:01.691898 24396 raft_consensus.cc:1275] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Refusing update from remote peer 2ae85b9b08454b42a0d9b3059aaa1304: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:07:01.693861 24520 consensus_queue.cc:1048] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:01.747078 24396 raft_consensus.cc:1275] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Refusing update from remote peer 2ae85b9b08454b42a0d9b3059aaa1304: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:01.747449 24247 raft_consensus.cc:1275] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Refusing update from remote peer 2ae85b9b08454b42a0d9b3059aaa1304: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:01.748472 24551 consensus_queue.cc:1048] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:01.749076 24520 consensus_queue.cc:1048] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:01.769520 24247 raft_consensus.cc:1275] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Refusing update from remote peer 2ae85b9b08454b42a0d9b3059aaa1304: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:01.770740 24561 consensus_queue.cc:1048] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:01.780897 24396 raft_consensus.cc:1275] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Refusing update from remote peer 2ae85b9b08454b42a0d9b3059aaa1304: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:01.782680 24561 consensus_queue.cc:1048] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:01.923421 24247 raft_consensus.cc:1275] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Refusing update from remote peer 2ae85b9b08454b42a0d9b3059aaa1304: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:01.925004 24520 consensus_queue.cc:1048] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:01.934875 24396 raft_consensus.cc:1275] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Refusing update from remote peer 2ae85b9b08454b42a0d9b3059aaa1304: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:01.936280 24520 consensus_queue.cc:1048] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:04.720852 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:04.793540 24574 raft_consensus.cc:493] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f450d3cd998e454da3e92d0381791c02)
I20260709 13:07:04.804857 24574 raft_consensus.cc:515] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.811739 24574 leader_election.cc:290] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:07:04.812383 24574 raft_consensus.cc:493] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2ae85b9b08454b42a0d9b3059aaa1304)
I20260709 13:07:04.813674 24574 raft_consensus.cc:515] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.815876 24574 leader_election.cc:290] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:07:04.817121 24574 raft_consensus.cc:493] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2ae85b9b08454b42a0d9b3059aaa1304)
I20260709 13:07:04.817663 24574 raft_consensus.cc:515] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } }
I20260709 13:07:04.819715 24574 leader_election.cc:290] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:07:04.821419 24574 raft_consensus.cc:493] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2ae85b9b08454b42a0d9b3059aaa1304)
I20260709 13:07:04.822048 24574 raft_consensus.cc:515] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.825098 24574 leader_election.cc:290] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:07:04.826812 24580 raft_consensus.cc:493] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2ae85b9b08454b42a0d9b3059aaa1304)
I20260709 13:07:04.827389 24580 raft_consensus.cc:515] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.828999 24321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc38bf75745341a08fa2ca990c4d199d" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:07:04.829905 24321 raft_consensus.cc:2468] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 1.
I20260709 13:07:04.830206 24395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3249533cb3842de91db1dc4cb6a4bcc" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02" is_pre_election: true
I20260709 13:07:04.830931 24395 raft_consensus.cc:2468] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 2.
I20260709 13:07:04.831673 24321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3249533cb3842de91db1dc4cb6a4bcc" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:07:04.832698 24211 leader_election.cc:304] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:07:04.833380 24321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83da7aa0dd1047908cfc3f6e16337706" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:07:04.834280 24577 raft_consensus.cc:493] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f450d3cd998e454da3e92d0381791c02)
I20260709 13:07:04.834664 24319 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f"
dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
mode: GRACEFUL
new_leader_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
 from {username='slave'} at 127.0.0.1:56340
I20260709 13:07:04.835059 24574 raft_consensus.cc:2804] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:07:04.834805 24577 raft_consensus.cc:515] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.835427 24319 raft_consensus.cc:606] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 LEADER]: Received request to transfer leadership to TS b81d16b3ad324fefa4ad3c3b5f4b3311
I20260709 13:07:04.837280 24577 leader_election.cc:290] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:07:04.837108 24211 leader_election.cc:304] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311; no voters: 
I20260709 13:07:04.838212 24320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:07:04.838985 24319 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "76d88f7a9bc84cbcb633d2433ddad860"
dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
mode: GRACEFUL
new_leader_uuid: "f450d3cd998e454da3e92d0381791c02"
 from {username='slave'} at 127.0.0.1:56340
I20260709 13:07:04.839217 24576 raft_consensus.cc:2804] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:07:04.839556 24319 raft_consensus.cc:606] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Received request to transfer leadership to TS f450d3cd998e454da3e92d0381791c02
I20260709 13:07:04.839792 24576 raft_consensus.cc:493] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f450d3cd998e454da3e92d0381791c02)
I20260709 13:07:04.840350 24576 raft_consensus.cc:3060] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:04.835458 24574 raft_consensus.cc:493] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2ae85b9b08454b42a0d9b3059aaa1304)
I20260709 13:07:04.843240 24574 raft_consensus.cc:3060] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:04.843101 24321 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "83da7aa0dd1047908cfc3f6e16337706"
dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
mode: GRACEFUL
new_leader_uuid: "f450d3cd998e454da3e92d0381791c02"
 from {username='slave'} at 127.0.0.1:56340
I20260709 13:07:04.843963 24321 raft_consensus.cc:606] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Received request to transfer leadership to TS f450d3cd998e454da3e92d0381791c02
I20260709 13:07:04.844143 24582 raft_consensus.cc:493] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2ae85b9b08454b42a0d9b3059aaa1304)
I20260709 13:07:04.844684 24582 raft_consensus.cc:515] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.847155 24321 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b3249533cb3842de91db1dc4cb6a4bcc"
dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
mode: GRACEFUL
new_leader_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
 from {username='slave'} at 127.0.0.1:56340
I20260709 13:07:04.847905 24321 raft_consensus.cc:606] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Received request to transfer leadership to TS b81d16b3ad324fefa4ad3c3b5f4b3311
I20260709 13:07:04.848474 24581 raft_consensus.cc:493] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2ae85b9b08454b42a0d9b3059aaa1304)
I20260709 13:07:04.849048 24581 raft_consensus.cc:515] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.852522 24549 raft_consensus.cc:493] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2ae85b9b08454b42a0d9b3059aaa1304)
I20260709 13:07:04.853384 24321 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a"
dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
mode: GRACEFUL
new_leader_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
 from {username='slave'} at 127.0.0.1:56340
I20260709 13:07:04.853495 24549 raft_consensus.cc:515] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.854152 24321 raft_consensus.cc:606] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 4 LEADER]: Received request to transfer leadership to TS b81d16b3ad324fefa4ad3c3b5f4b3311
I20260709 13:07:04.854573 24582 leader_election.cc:290] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:07:04.855067 24582 raft_consensus.cc:493] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2ae85b9b08454b42a0d9b3059aaa1304)
I20260709 13:07:04.855638 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:07:04.855742 24549 leader_election.cc:290] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:07:04.856284 24549 raft_consensus.cc:493] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2ae85b9b08454b42a0d9b3059aaa1304)
I20260709 13:07:04.856454 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:07:04.857065 24581 leader_election.cc:290] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:07:04.857854 24582 raft_consensus.cc:515] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.859939 24582 leader_election.cc:290] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:07:04.860402 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:07:04.865108 24247 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc38bf75745341a08fa2ca990c4d199d" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" is_pre_election: true
I20260709 13:07:04.856715 24549 raft_consensus.cc:515] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.868034 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9587f033fa334d328ccd9149b5b22bcf" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:07:04.869414 24549 leader_election.cc:290] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:07:04.869925 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:07:04.870958 24322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3249533cb3842de91db1dc4cb6a4bcc" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:07:04.872385 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:07:04.873337 24248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3249533cb3842de91db1dc4cb6a4bcc" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" is_pre_election: true
I20260709 13:07:04.873366 24321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:07:04.877240 24244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9587f033fa334d328ccd9149b5b22bcf" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" is_pre_election: true
I20260709 13:07:04.877532 24248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" is_pre_election: true
I20260709 13:07:04.879108 24243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e395d0ca9a1f4f08bb3fed809205f52d" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" is_pre_election: true
I20260709 13:07:04.884364 24394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02" is_pre_election: true
I20260709 13:07:04.886003 24362 leader_election.cc:304] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f450d3cd998e454da3e92d0381791c02; no voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311
I20260709 13:07:04.886966 24243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" is_pre_election: true
I20260709 13:07:04.887235 24362 leader_election.cc:304] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f450d3cd998e454da3e92d0381791c02; no voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311
I20260709 13:07:04.887768 24243 raft_consensus.cc:2468] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 4.
I20260709 13:07:04.888408 24578 raft_consensus.cc:993] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304: : Instructing follower b81d16b3ad324fefa4ad3c3b5f4b3311 to start an election
I20260709 13:07:04.888877 24578 raft_consensus.cc:1081] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 LEADER]: Signalling peer b81d16b3ad324fefa4ad3c3b5f4b3311 to start an election
I20260709 13:07:04.888398 24362 leader_election.cc:304] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f450d3cd998e454da3e92d0381791c02; no voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311
I20260709 13:07:04.890394 24396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc38bf75745341a08fa2ca990c4d199d" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02" is_pre_election: true
I20260709 13:07:04.890895 24362 leader_election.cc:304] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f450d3cd998e454da3e92d0381791c02; no voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311
I20260709 13:07:04.892056 24580 leader_election.cc:290] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:07:04.892584 24580 raft_consensus.cc:493] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2ae85b9b08454b42a0d9b3059aaa1304)
I20260709 13:07:04.892395 24549 raft_consensus.cc:2749] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:07:04.893116 24582 raft_consensus.cc:2749] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:07:04.894757 24397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83da7aa0dd1047908cfc3f6e16337706" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02" is_pre_election: true
I20260709 13:07:04.895120 24585 raft_consensus.cc:493] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2ae85b9b08454b42a0d9b3059aaa1304)
I20260709 13:07:04.895432 24397 raft_consensus.cc:2468] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 2.
I20260709 13:07:04.896366 24582 raft_consensus.cc:2749] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:07:04.895597 24585 raft_consensus.cc:515] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.885244 24394 raft_consensus.cc:2468] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 3.
I20260709 13:07:04.900656 24585 leader_election.cc:290] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:07:04.896991 24582 raft_consensus.cc:493] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2ae85b9b08454b42a0d9b3059aaa1304)
I20260709 13:07:04.902732 24243 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f"
dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
 from {username='slave'} at 127.0.0.1:39430
I20260709 13:07:04.902832 24582 raft_consensus.cc:515] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.903501 24243 raft_consensus.cc:493] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:04.903909 24243 raft_consensus.cc:3060] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:04.904526 24395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02" is_pre_election: true
I20260709 13:07:04.905184 24395 raft_consensus.cc:2468] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 4.
I20260709 13:07:04.907367 24211 leader_election.cc:304] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:07:04.908233 24211 leader_election.cc:304] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:07:04.908406 24574 raft_consensus.cc:515] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.909504 24245 raft_consensus.cc:1240] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Rejecting Update request from peer 2ae85b9b08454b42a0d9b3059aaa1304 for earlier term 2. Current term is 3. Ops: []
I20260709 13:07:04.909524 24583 raft_consensus.cc:2804] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:07:04.910601 24583 raft_consensus.cc:493] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2ae85b9b08454b42a0d9b3059aaa1304)
I20260709 13:07:04.910990 24583 raft_consensus.cc:3060] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:04.914094 24575 raft_consensus.cc:993] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304: : Instructing follower b81d16b3ad324fefa4ad3c3b5f4b3311 to start an election
I20260709 13:07:04.914633 24575 raft_consensus.cc:1081] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 4 LEADER]: Signalling peer b81d16b3ad324fefa4ad3c3b5f4b3311 to start an election
I20260709 13:07:04.915659 24579 consensus_queue.cc:1059] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:04.915943 24362 leader_election.cc:304] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 2ae85b9b08454b42a0d9b3059aaa1304
I20260709 13:07:04.916923 24575 raft_consensus.cc:3055] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:04.917467 24575 raft_consensus.cc:740] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Running, Role: LEADER
I20260709 13:07:04.918282 24583 raft_consensus.cc:515] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } }
I20260709 13:07:04.918713 24245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51f37fd5f5c34b13885b93a9eee3bc8f" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" is_pre_election: true
I20260709 13:07:04.914958 24580 raft_consensus.cc:515] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.919346 24575 consensus_queue.cc:260] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.918006 24585 raft_consensus.cc:2804] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:07:04.918094 24248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d88f7a9bc84cbcb633d2433ddad860" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" is_pre_election: true
I20260709 13:07:04.920905 24582 leader_election.cc:290] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:07:04.921253 24549 raft_consensus.cc:2749] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:07:04.921408 24580 leader_election.cc:290] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:07:04.921484 24585 raft_consensus.cc:493] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 2ae85b9b08454b42a0d9b3059aaa1304)
I20260709 13:07:04.922245 24585 raft_consensus.cc:3060] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:07:04.923933 24321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51f37fd5f5c34b13885b93a9eee3bc8f" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:07:04.923485 24574 leader_election.cc:290] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 3 election: Requested vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:07:04.924795 24392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc38bf75745341a08fa2ca990c4d199d" candidate_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02" is_pre_election: true
I20260709 13:07:04.922246 24576 raft_consensus.cc:515] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.925624 24248 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a"
dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
 from {username='slave'} at 127.0.0.1:39430
I20260709 13:07:04.928146 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d88f7a9bc84cbcb633d2433ddad860" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:07:04.920382 24583 leader_election.cc:290] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 3 election: Requested vote from peers f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:07:04.941325 24247 raft_consensus.cc:2393] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2ae85b9b08454b42a0d9b3059aaa1304 in current term 2: Already voted for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in this term.
I20260709 13:07:04.945245 24576 leader_election.cc:290] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 2 election: Requested vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:07:04.948427 24575 raft_consensus.cc:3060] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:04.956633 24360 leader_election.cc:304] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f450d3cd998e454da3e92d0381791c02; no voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311
I20260709 13:07:04.957679 24581 raft_consensus.cc:2749] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:07:04.949273 24248 raft_consensus.cc:493] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:04.958453 24248 raft_consensus.cc:3060] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:07:04.914170 24243 raft_consensus.cc:515] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.964764 24360 leader_election.cc:304] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f450d3cd998e454da3e92d0381791c02; no voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311
W20260709 13:07:04.962476 24211 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:35329, user_credentials={real_user=slave}} blocked reactor thread for 54275.2us
I20260709 13:07:04.967298 24211 leader_election.cc:304] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 2ae85b9b08454b42a0d9b3059aaa1304
I20260709 13:07:04.970252 24246 raft_consensus.cc:1240] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Rejecting Update request from peer f450d3cd998e454da3e92d0381791c02 for earlier term 1. Current term is 2. Ops: []
I20260709 13:07:04.971539 24392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83da7aa0dd1047908cfc3f6e16337706" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:07:04.973938 24581 raft_consensus.cc:2749] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:07:04.974313 24549 consensus_queue.cc:1059] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:04.975613 24581 raft_consensus.cc:3055] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:04.976069 24581 raft_consensus.cc:740] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Running, Role: LEADER
I20260709 13:07:04.977152 24581 consensus_queue.cc:260] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.978747 24581 raft_consensus.cc:3060] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:04.980404 24580 raft_consensus.cc:2764] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 13:07:04.982637 24243 leader_election.cc:290] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 4 election: Requested vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:07:04.986049 24585 raft_consensus.cc:515] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.989814 24243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:07:04.990743 24321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51f37fd5f5c34b13885b93a9eee3bc8f" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" is_pre_election: true
I20260709 13:07:04.991820 24392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51f37fd5f5c34b13885b93a9eee3bc8f" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02" is_pre_election: true
I20260709 13:07:04.985473 24248 raft_consensus.cc:515] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:04.992031 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83da7aa0dd1047908cfc3f6e16337706" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
W20260709 13:07:04.992501 24286 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:35329, user_credentials={real_user=slave}} blocked reactor thread for 63011.9us
I20260709 13:07:04.993024 24586 raft_consensus.cc:2764] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:07:04.993749 24248 leader_election.cc:290] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 5 election: Requested vote from peers 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417), f450d3cd998e454da3e92d0381791c02 (127.23.140.67:35329)
I20260709 13:07:04.994344 24585 leader_election.cc:290] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 5 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:07:04.995651 24588 raft_consensus.cc:993] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304: : Instructing follower f450d3cd998e454da3e92d0381791c02 to start an election
I20260709 13:07:04.995594 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:07:04.996201 24578 raft_consensus.cc:1081] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Signalling peer f450d3cd998e454da3e92d0381791c02 to start an election
I20260709 13:07:04.996366 24579 raft_consensus.cc:993] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304: : Instructing follower f450d3cd998e454da3e92d0381791c02 to start an election
I20260709 13:07:04.996821 24588 raft_consensus.cc:1081] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Signalling peer f450d3cd998e454da3e92d0381791c02 to start an election
I20260709 13:07:04.996963 24286 leader_election.cc:304] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304; no voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02
I20260709 13:07:04.997212 24360 leader_election.cc:304] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f450d3cd998e454da3e92d0381791c02; no voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311
I20260709 13:07:04.997848 24588 raft_consensus.cc:3060] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:04.997896 24585 raft_consensus.cc:2749] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:07:04.998987 24392 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "76d88f7a9bc84cbcb633d2433ddad860"
dest_uuid: "f450d3cd998e454da3e92d0381791c02"
 from {username='slave'} at 127.0.0.1:36802
I20260709 13:07:04.999346 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3249533cb3842de91db1dc4cb6a4bcc" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:07:04.999534 24392 raft_consensus.cc:493] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:04.999922 24392 raft_consensus.cc:3060] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:05.000280 24321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc38bf75745341a08fa2ca990c4d199d" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:07:05.003350 24397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc38bf75745341a08fa2ca990c4d199d" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:07:05.003705 24396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3249533cb3842de91db1dc4cb6a4bcc" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:07:05.006246 24321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:07:05.006650 24588 raft_consensus.cc:2749] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:07:05.006783 24321 raft_consensus.cc:3055] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:07:05.007145 24321 raft_consensus.cc:740] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Running, Role: LEADER
I20260709 13:07:05.007804 24396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:07:05.007891 24321 consensus_queue.cc:260] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:05.008363 24396 raft_consensus.cc:3060] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:05.008829 24321 raft_consensus.cc:3060] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:05.006773 24323 raft_consensus.cc:2468] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 3.
I20260709 13:07:05.011209 24394 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "83da7aa0dd1047908cfc3f6e16337706"
dest_uuid: "f450d3cd998e454da3e92d0381791c02"
 from {username='slave'} at 127.0.0.1:36802
I20260709 13:07:05.011742 24394 raft_consensus.cc:493] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:05.012074 24394 raft_consensus.cc:3060] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:05.014564 24321 raft_consensus.cc:2468] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 4.
I20260709 13:07:05.017695 24321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:07:05.018081 24393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df51c4376a72401495d95fe4c2df882a" candidate_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "f450d3cd998e454da3e92d0381791c02"
I20260709 13:07:05.018277 24321 raft_consensus.cc:3055] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:07:05.018625 24321 raft_consensus.cc:740] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Running, Role: LEADER
I20260709 13:07:05.018906 24393 raft_consensus.cc:2393] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in current term 5: Already voted for candidate f450d3cd998e454da3e92d0381791c02 in this term.
I20260709 13:07:05.018853 24394 raft_consensus.cc:515] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } }
I20260709 13:07:05.019613 24321 consensus_queue.cc:260] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:05.020259 24396 raft_consensus.cc:2468] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 4.
I20260709 13:07:05.020915 24321 raft_consensus.cc:3060] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:07:05.021796 24248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83da7aa0dd1047908cfc3f6e16337706" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:07:05.022234 24323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83da7aa0dd1047908cfc3f6e16337706" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:07:05.022528 24248 raft_consensus.cc:2393] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f450d3cd998e454da3e92d0381791c02 in current term 3: Already voted for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in this term.
I20260709 13:07:05.022773 24323 raft_consensus.cc:3055] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:05.023079 24323 raft_consensus.cc:740] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Running, Role: LEADER
I20260709 13:07:05.023149 24397 raft_consensus.cc:2468] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 2.
I20260709 13:07:05.023950 24323 consensus_queue.cc:260] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } }
I20260709 13:07:05.025058 24323 raft_consensus.cc:3060] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:05.025573 24211 leader_election.cc:304] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311; no voters: 
I20260709 13:07:05.026350 24586 raft_consensus.cc:2804] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:05.026723 24321 raft_consensus.cc:2468] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81d16b3ad324fefa4ad3c3b5f4b3311 in term 5.
I20260709 13:07:05.026790 24586 raft_consensus.cc:697] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 LEADER]: Becoming Leader. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Running, Role: LEADER
I20260709 13:07:05.027755 24586 consensus_queue.cc:237] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:05.029326 24394 leader_election.cc:290] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 3 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:07:05.029853 24392 raft_consensus.cc:515] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:05.032550 24248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d88f7a9bc84cbcb633d2433ddad860" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:07:05.033073 24321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d88f7a9bc84cbcb633d2433ddad860" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:07:05.033272 24248 raft_consensus.cc:3060] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:05.033772 24321 raft_consensus.cc:3055] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:05.034142 24321 raft_consensus.cc:740] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2ae85b9b08454b42a0d9b3059aaa1304, State: Running, Role: LEADER
I20260709 13:07:05.035044 24321 consensus_queue.cc:260] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:05.036134 24321 raft_consensus.cc:3060] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:05.036311 24211 leader_election.cc:304] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311; no voters: 
I20260709 13:07:05.037081 24586 raft_consensus.cc:2804] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:07:05.037238 24211 leader_election.cc:304] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, b81d16b3ad324fefa4ad3c3b5f4b3311; no voters: 
I20260709 13:07:05.037750 24586 raft_consensus.cc:697] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 LEADER]: Becoming Leader. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Running, Role: LEADER
I20260709 13:07:05.038091 24580 raft_consensus.cc:2804] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:07:05.038578 24580 raft_consensus.cc:697] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 5 LEADER]: Becoming Leader. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Running, Role: LEADER
I20260709 13:07:05.038949 24248 raft_consensus.cc:2468] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 3.
I20260709 13:07:05.038619 24586 consensus_queue.cc:237] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:05.039427 24580 consensus_queue.cc:237] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:05.040067 24362 leader_election.cc:304] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:07:05.041280 24581 raft_consensus.cc:2804] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:05.042120 24581 raft_consensus.cc:697] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 3 LEADER]: Becoming Leader. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Running, Role: LEADER
W20260709 13:07:05.043063 24344 debug-util.cc:398] Leaking SignalData structure 0x7b080012e060 after lost signal to thread 24116
I20260709 13:07:05.043367 24321 raft_consensus.cc:2468] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 3.
W20260709 13:07:05.043996 24344 debug-util.cc:398] Leaking SignalData structure 0x7b080012e740 after lost signal to thread 24179
I20260709 13:07:05.043434 24581 consensus_queue.cc:237] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:05.044409 24211 leader_election.cc:304] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311; no voters: 2ae85b9b08454b42a0d9b3059aaa1304, f450d3cd998e454da3e92d0381791c02
I20260709 13:07:05.032042 24392 leader_election.cc:290] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 3 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
W20260709 13:07:05.045034 24344 debug-util.cc:398] Leaking SignalData structure 0x7b08001171e0 after lost signal to thread 24272
W20260709 13:07:05.045980 24344 debug-util.cc:398] Leaking SignalData structure 0x7b0800095360 after lost signal to thread 24347
I20260709 13:07:05.046156 24583 raft_consensus.cc:2749] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:07:05.046931 24344 debug-util.cc:398] Leaking SignalData structure 0x7b0800194500 after lost signal to thread 24421
I20260709 13:07:05.047299 24211 leader_election.cc:304] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311; no voters: 2ae85b9b08454b42a0d9b3059aaa1304, f450d3cd998e454da3e92d0381791c02
I20260709 13:07:05.049268 24583 raft_consensus.cc:2749] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:07:05.049242 24323 raft_consensus.cc:2468] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 3.
I20260709 13:07:05.051067 24360 leader_election.cc:304] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, f450d3cd998e454da3e92d0381791c02; no voters: b81d16b3ad324fefa4ad3c3b5f4b3311
I20260709 13:07:05.052031 24585 raft_consensus.cc:2804] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:05.052114 24211 leader_election.cc:304] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b81d16b3ad324fefa4ad3c3b5f4b3311, f450d3cd998e454da3e92d0381791c02; no voters: 2ae85b9b08454b42a0d9b3059aaa1304
I20260709 13:07:05.052561 24585 raft_consensus.cc:697] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [term 3 LEADER]: Becoming Leader. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Running, Role: LEADER
I20260709 13:07:05.052827 24580 raft_consensus.cc:2804] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:05.053521 24585 consensus_queue.cc:237] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } }
I20260709 13:07:05.053546 24580 raft_consensus.cc:697] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 LEADER]: Becoming Leader. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Running, Role: LEADER
I20260709 13:07:05.059054 24580 consensus_queue.cc:237] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:05.059068 24145 catalog_manager.cc:5697] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 reported cstate change: term changed from 2 to 3, leader changed from 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66) to b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65). New cstate: current_term: 3 leader_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:05.078264 24144 catalog_manager.cc:5697] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 reported cstate change: term changed from 2 to 3, leader changed from 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66) to f450d3cd998e454da3e92d0381791c02 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "f450d3cd998e454da3e92d0381791c02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:05.090718 24144 catalog_manager.cc:5697] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 reported cstate change: term changed from 3 to 4, leader changed from 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66) to b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65). New cstate: current_term: 4 leader_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:05.092314 24144 catalog_manager.cc:5697] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 reported cstate change: term changed from 4 to 5, leader changed from 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66) to b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65). New cstate: current_term: 5 leader_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:05.093153 24145 catalog_manager.cc:5697] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 reported cstate change: term changed from 2 to 3, leader changed from 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66) to f450d3cd998e454da3e92d0381791c02 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "f450d3cd998e454da3e92d0381791c02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:05.093835 24144 catalog_manager.cc:5697] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 reported cstate change: term changed from 1 to 2, leader changed from f450d3cd998e454da3e92d0381791c02 (127.23.140.67) to b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:05.460721 24323 raft_consensus.cc:1275] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Refusing update from remote peer f450d3cd998e454da3e92d0381791c02: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:05.462306 24581 consensus_queue.cc:1048] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:05.462625 24323 raft_consensus.cc:1275] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 4 FOLLOWER]: Refusing update from remote peer b81d16b3ad324fefa4ad3c3b5f4b3311: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:07:05.463861 24580 consensus_queue.cc:1048] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:05.473552 24248 raft_consensus.cc:1275] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Refusing update from remote peer f450d3cd998e454da3e92d0381791c02: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:05.476269 24549 consensus_queue.cc:1048] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:05.484560 24322 raft_consensus.cc:1275] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Refusing update from remote peer b81d16b3ad324fefa4ad3c3b5f4b3311: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:05.486199 24580 consensus_queue.cc:1048] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:05.494598 24243 raft_consensus.cc:1275] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Refusing update from remote peer f450d3cd998e454da3e92d0381791c02: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:05.500523 24392 raft_consensus.cc:1275] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Refusing update from remote peer b81d16b3ad324fefa4ad3c3b5f4b3311: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:07:05.503764 24394 raft_consensus.cc:1275] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 5 FOLLOWER]: Refusing update from remote peer b81d16b3ad324fefa4ad3c3b5f4b3311: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:07:05.505554 24580 consensus_queue.cc:1048] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:05.507236 24586 consensus_queue.cc:1048] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:05.508708 24581 consensus_queue.cc:1048] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:05.523131 24322 raft_consensus.cc:1275] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Refusing update from remote peer f450d3cd998e454da3e92d0381791c02: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:05.525388 24581 consensus_queue.cc:1048] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:05.529693 24321 raft_consensus.cc:1275] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 5 FOLLOWER]: Refusing update from remote peer b81d16b3ad324fefa4ad3c3b5f4b3311: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:07:05.538115 24396 raft_consensus.cc:3060] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:05.545831 24606 consensus_queue.cc:1048] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:05.547513 24396 raft_consensus.cc:1275] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Refusing update from remote peer b81d16b3ad324fefa4ad3c3b5f4b3311: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:05.550846 24606 consensus_queue.cc:1048] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:05.623963 24394 raft_consensus.cc:1275] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Refusing update from remote peer b81d16b3ad324fefa4ad3c3b5f4b3311: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:05.625345 24606 consensus_queue.cc:1048] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:05.635742 24322 raft_consensus.cc:1275] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Refusing update from remote peer b81d16b3ad324fefa4ad3c3b5f4b3311: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:05.637904 24586 consensus_queue.cc:1048] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:05.857551 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:05.864684 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:05.872686 24243 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f"
dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
mode: GRACEFUL
new_leader_uuid: "f450d3cd998e454da3e92d0381791c02"
 from {username='slave'} at 127.0.0.1:39444
I20260709 13:07:05.873358 24243 raft_consensus.cc:606] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 LEADER]: Received request to transfer leadership to TS f450d3cd998e454da3e92d0381791c02
I20260709 13:07:05.874608 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260709 13:07:05.875011 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:07:06.122605 24606 raft_consensus.cc:993] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311: : Instructing follower f450d3cd998e454da3e92d0381791c02 to start an election
I20260709 13:07:06.123035 24586 raft_consensus.cc:1081] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 LEADER]: Signalling peer f450d3cd998e454da3e92d0381791c02 to start an election
I20260709 13:07:06.124374 24394 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f"
dest_uuid: "f450d3cd998e454da3e92d0381791c02"
 from {username='slave'} at 127.0.0.1:36786
I20260709 13:07:06.124912 24394 raft_consensus.cc:493] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:06.125295 24394 raft_consensus.cc:3060] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:07:06.130517 24394 raft_consensus.cc:515] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:06.132251 24394 leader_election.cc:290] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 5 election: Requested vote from peers b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615), 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417)
I20260709 13:07:06.132901 24243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311"
I20260709 13:07:06.133311 24322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c6de3b5e014474b86031c03d56f84f" candidate_uuid: "f450d3cd998e454da3e92d0381791c02" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "2ae85b9b08454b42a0d9b3059aaa1304"
I20260709 13:07:06.133648 24243 raft_consensus.cc:3055] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:07:06.133823 24322 raft_consensus.cc:3060] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:07:06.134109 24243 raft_consensus.cc:740] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b81d16b3ad324fefa4ad3c3b5f4b3311, State: Running, Role: LEADER
I20260709 13:07:06.134967 24243 consensus_queue.cc:260] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:06.136186 24243 raft_consensus.cc:3060] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:07:06.139202 24322 raft_consensus.cc:2468] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 5.
I20260709 13:07:06.140087 24360 leader_election.cc:304] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae85b9b08454b42a0d9b3059aaa1304, f450d3cd998e454da3e92d0381791c02; no voters: 
I20260709 13:07:06.140691 24617 raft_consensus.cc:2804] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:07:06.141120 24617 raft_consensus.cc:697] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 5 LEADER]: Becoming Leader. State: Replica: f450d3cd998e454da3e92d0381791c02, State: Running, Role: LEADER
I20260709 13:07:06.141989 24617 consensus_queue.cc:237] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } }
I20260709 13:07:06.144121 24243 raft_consensus.cc:2468] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f450d3cd998e454da3e92d0381791c02 in term 5.
I20260709 13:07:06.151721 24144 catalog_manager.cc:5697] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 reported cstate change: term changed from 4 to 5, leader changed from b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65) to f450d3cd998e454da3e92d0381791c02 (127.23.140.67). New cstate: current_term: 5 leader_uuid: "f450d3cd998e454da3e92d0381791c02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f450d3cd998e454da3e92d0381791c02" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35329 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:06.561091 24243 raft_consensus.cc:1275] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 5 FOLLOWER]: Refusing update from remote peer f450d3cd998e454da3e92d0381791c02: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:07:06.562481 24617 consensus_queue.cc:1048] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81d16b3ad324fefa4ad3c3b5f4b3311" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:06.582154 24322 raft_consensus.cc:1275] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 5 FOLLOWER]: Refusing update from remote peer f450d3cd998e454da3e92d0381791c02: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:07:06.583437 24616 consensus_queue.cc:1048] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae85b9b08454b42a0d9b3059aaa1304" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:06.875880 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:06.884752 24113 tablet_server.cc:179] TabletServer@127.23.140.65:0 shutting down...
I20260709 13:07:06.915791 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:07:06.916558 24113 tablet_replica.cc:333] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311: stopping tablet replica
I20260709 13:07:06.917325 24113 raft_consensus.cc:2243] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:06.917883 24113 raft_consensus.cc:2272] T 76d88f7a9bc84cbcb633d2433ddad860 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:06.920670 24113 tablet_replica.cc:333] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311: stopping tablet replica
I20260709 13:07:06.921388 24113 raft_consensus.cc:2243] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:06.921800 24113 raft_consensus.cc:2272] T 51f37fd5f5c34b13885b93a9eee3bc8f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:06.923496 24113 tablet_replica.cc:333] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311: stopping tablet replica
I20260709 13:07:06.923965 24113 raft_consensus.cc:2243] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:06.924371 24113 raft_consensus.cc:2272] T 9587f033fa334d328ccd9149b5b22bcf P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:06.926033 24113 tablet_replica.cc:333] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311: stopping tablet replica
I20260709 13:07:06.926496 24113 raft_consensus.cc:2243] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:07:06.927150 24113 raft_consensus.cc:2272] T b3249533cb3842de91db1dc4cb6a4bcc P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:06.928717 24113 tablet_replica.cc:333] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311: stopping tablet replica
I20260709 13:07:06.929255 24113 raft_consensus.cc:2243] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 5 LEADER]: Raft consensus shutting down.
I20260709 13:07:06.929975 24113 raft_consensus.cc:2272] T df51c4376a72401495d95fe4c2df882a P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:06.931414 24113 tablet_replica.cc:333] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311: stopping tablet replica
I20260709 13:07:06.931890 24113 raft_consensus.cc:2243] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:06.932530 24113 raft_consensus.cc:2272] T dc38bf75745341a08fa2ca990c4d199d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:06.934111 24113 tablet_replica.cc:333] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311: stopping tablet replica
I20260709 13:07:06.934562 24113 raft_consensus.cc:2243] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:06.935003 24113 raft_consensus.cc:2272] T 69c6de3b5e014474b86031c03d56f84f P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:06.937454 24113 tablet_replica.cc:333] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311: stopping tablet replica
I20260709 13:07:06.937891 24113 raft_consensus.cc:2243] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:06.938313 24113 raft_consensus.cc:2272] T e395d0ca9a1f4f08bb3fed809205f52d P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:06.939807 24113 tablet_replica.cc:333] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311: stopping tablet replica
I20260709 13:07:06.940220 24113 raft_consensus.cc:2243] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:06.940606 24113 raft_consensus.cc:2272] T 83da7aa0dd1047908cfc3f6e16337706 P b81d16b3ad324fefa4ad3c3b5f4b3311 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 13:07:07.081599 24288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.140.65:41615: connect: Connection refused (error 111)
I20260709 13:07:07.085521 24113 tablet_server.cc:196] TabletServer@127.23.140.65:0 shutdown complete.
W20260709 13:07:07.087381 24288 consensus_peers.cc:597] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 -> Peer b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615): Couldn't send request to peer b81d16b3ad324fefa4ad3c3b5f4b3311. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:41615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:07:07.087802 24288 consensus_peers.cc:597] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 -> Peer b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615): Couldn't send request to peer b81d16b3ad324fefa4ad3c3b5f4b3311. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:41615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:07:07.111820 24362 consensus_peers.cc:597] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 -> Peer b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615): Couldn't send request to peer b81d16b3ad324fefa4ad3c3b5f4b3311. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:41615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:07.114130 24113 tablet_server.cc:179] TabletServer@127.23.140.66:0 shutting down...
I20260709 13:07:07.142609 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:07:07.143361 24113 tablet_replica.cc:333] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304: stopping tablet replica
I20260709 13:07:07.144357 24113 raft_consensus.cc:2243] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:07.145067 24113 raft_consensus.cc:2272] T 83da7aa0dd1047908cfc3f6e16337706 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:07.148258 24113 tablet_replica.cc:333] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304: stopping tablet replica
I20260709 13:07:07.149349 24113 raft_consensus.cc:2243] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:07:07.150454 24113 raft_consensus.cc:2272] T e395d0ca9a1f4f08bb3fed809205f52d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:07.153673 24113 tablet_replica.cc:333] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304: stopping tablet replica
I20260709 13:07:07.154704 24113 raft_consensus.cc:2243] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:07.155381 24113 raft_consensus.cc:2272] T 76d88f7a9bc84cbcb633d2433ddad860 P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 13:07:07.155400 24360 consensus_peers.cc:597] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 -> Peer 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417): Couldn't send request to peer 2ae85b9b08454b42a0d9b3059aaa1304. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:07.157742 24113 tablet_replica.cc:333] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304: stopping tablet replica
I20260709 13:07:07.158267 24113 raft_consensus.cc:2243] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:07.158942 24113 raft_consensus.cc:2272] T 51f37fd5f5c34b13885b93a9eee3bc8f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:07.160580 24113 tablet_replica.cc:333] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304: stopping tablet replica
I20260709 13:07:07.161056 24113 raft_consensus.cc:2243] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:07.161482 24113 raft_consensus.cc:2272] T dc38bf75745341a08fa2ca990c4d199d P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:07.162986 24113 tablet_replica.cc:333] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304: stopping tablet replica
I20260709 13:07:07.163431 24113 raft_consensus.cc:2243] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:07.163813 24113 raft_consensus.cc:2272] T 69c6de3b5e014474b86031c03d56f84f P 2ae85b9b08454b42a0d9b3059aaa1304 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:07.166337 24113 tablet_replica.cc:333] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304: stopping tablet replica
I20260709 13:07:07.166802 24113 raft_consensus.cc:2243] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:07.167178 24113 raft_consensus.cc:2272] T b3249533cb3842de91db1dc4cb6a4bcc P 2ae85b9b08454b42a0d9b3059aaa1304 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:07.169118 24113 tablet_replica.cc:333] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304: stopping tablet replica
I20260709 13:07:07.169632 24113 raft_consensus.cc:2243] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:07.170007 24113 raft_consensus.cc:2272] T df51c4376a72401495d95fe4c2df882a P 2ae85b9b08454b42a0d9b3059aaa1304 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:07.171459 24113 tablet_replica.cc:333] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304: stopping tablet replica
I20260709 13:07:07.171860 24113 raft_consensus.cc:2243] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:07.172533 24113 raft_consensus.cc:2272] T 9587f033fa334d328ccd9149b5b22bcf P 2ae85b9b08454b42a0d9b3059aaa1304 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:07:07.198467 24362 consensus_peers.cc:597] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 -> Peer b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615): Couldn't send request to peer b81d16b3ad324fefa4ad3c3b5f4b3311. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:41615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:07:07.309055 24362 consensus_peers.cc:597] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 -> Peer b81d16b3ad324fefa4ad3c3b5f4b3311 (127.23.140.65:41615): Couldn't send request to peer b81d16b3ad324fefa4ad3c3b5f4b3311. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:41615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:07:07.438894 24360 consensus_peers.cc:597] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 -> Peer 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417): Couldn't send request to peer 2ae85b9b08454b42a0d9b3059aaa1304. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42417: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:07:07.478787 24360 consensus_peers.cc:597] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 -> Peer 2ae85b9b08454b42a0d9b3059aaa1304 (127.23.140.66:42417): Couldn't send request to peer 2ae85b9b08454b42a0d9b3059aaa1304. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42417: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:07.574357 24113 tablet_server.cc:196] TabletServer@127.23.140.66:0 shutdown complete.
I20260709 13:07:07.593189 24113 tablet_server.cc:179] TabletServer@127.23.140.67:0 shutting down...
I20260709 13:07:07.616745 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:07:07.617419 24113 tablet_replica.cc:333] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02: stopping tablet replica
I20260709 13:07:07.618037 24113 raft_consensus.cc:2243] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:07:07.618813 24113 raft_consensus.cc:2272] T 76d88f7a9bc84cbcb633d2433ddad860 P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:07.620535 24113 tablet_replica.cc:333] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02: stopping tablet replica
I20260709 13:07:07.621058 24113 raft_consensus.cc:2243] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:07.621577 24113 raft_consensus.cc:2272] T 51f37fd5f5c34b13885b93a9eee3bc8f P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:07.623133 24113 tablet_replica.cc:333] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02: stopping tablet replica
I20260709 13:07:07.623616 24113 raft_consensus.cc:2243] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:07.624010 24113 raft_consensus.cc:2272] T df51c4376a72401495d95fe4c2df882a P f450d3cd998e454da3e92d0381791c02 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:07.625661 24113 tablet_replica.cc:333] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02: stopping tablet replica
I20260709 13:07:07.626148 24113 raft_consensus.cc:2243] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:07.626540 24113 raft_consensus.cc:2272] T b3249533cb3842de91db1dc4cb6a4bcc P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:07.628058 24113 tablet_replica.cc:333] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02: stopping tablet replica
I20260709 13:07:07.628547 24113 raft_consensus.cc:2243] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:07.628966 24113 raft_consensus.cc:2272] T 9587f033fa334d328ccd9149b5b22bcf P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:07.630651 24113 tablet_replica.cc:333] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02: stopping tablet replica
I20260709 13:07:07.631196 24113 raft_consensus.cc:2243] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:07.631593 24113 raft_consensus.cc:2272] T e395d0ca9a1f4f08bb3fed809205f52d P f450d3cd998e454da3e92d0381791c02 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:07.633224 24113 tablet_replica.cc:333] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02: stopping tablet replica
I20260709 13:07:07.633661 24113 raft_consensus.cc:2243] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:07:07.634339 24113 raft_consensus.cc:2272] T 83da7aa0dd1047908cfc3f6e16337706 P f450d3cd998e454da3e92d0381791c02 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:07.635952 24113 tablet_replica.cc:333] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02: stopping tablet replica
I20260709 13:07:07.636399 24113 raft_consensus.cc:2243] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:07.636777 24113 raft_consensus.cc:2272] T dc38bf75745341a08fa2ca990c4d199d P f450d3cd998e454da3e92d0381791c02 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:07.638307 24113 tablet_replica.cc:333] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02: stopping tablet replica
I20260709 13:07:07.638727 24113 raft_consensus.cc:2243] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 5 LEADER]: Raft consensus shutting down.
I20260709 13:07:07.639405 24113 raft_consensus.cc:2272] T 69c6de3b5e014474b86031c03d56f84f P f450d3cd998e454da3e92d0381791c02 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:07.667099 24113 tablet_server.cc:196] TabletServer@127.23.140.67:0 shutdown complete.
I20260709 13:07:07.684535 24113 master.cc:562] Master@127.23.140.126:37707 shutting down...
I20260709 13:07:07.796243 24200 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Closing
I20260709 13:07:07.820075 24113 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:07:07.820566 24113 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:07.820909 24113 tablet_replica.cc:333] T 00000000000000000000000000000000 P 2357efedd24947459e4bb0fd26f6a96d: stopping tablet replica
I20260709 13:07:07.839236 24113 master.cc:584] Master@127.23.140.126:37707 shutdown complete.
2ae85b9b08454b42a0d9b3059aaa1304  4
b81d16b3ad324fefa4ad3c3b5f4b3311  4
f450d3cd998e454da3e92d0381791c02  1
[       OK ] LeaderRebalancerTest.FunctionalTestForDivided (14707 ms)
[ RUN      ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260709 13:07:07.877343 24113 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.140.126:41765
I20260709 13:07:07.878491 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:07:07.884158 24633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:07.884380 24634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:07.885564 24636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:07.885788 24113 server_base.cc:1061] running on GCE node
I20260709 13:07:07.886690 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:07:07.886900 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:07:07.887058 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602427887042 us; error 0 us; skew 500 ppm
I20260709 13:07:07.887568 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:07:07.889909 24113 webserver.cc:533] Webserver started at http://127.23.140.126:45863/ using document root <none> and password file <none>
I20260709 13:07:07.890405 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:07:07.890599 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:07:07.890873 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:07:07.892554 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/master-0-root/instance:
uuid: "36709ee497be48979944e8465cd39dea"
format_stamp: "Formatted at 2026-07-09 13:07:07 on dist-test-slave-k73n"
I20260709 13:07:07.896733 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 13:07:07.899858 24641 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:07.900738 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 13:07:07.900962 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/master-0-root
uuid: "36709ee497be48979944e8465cd39dea"
format_stamp: "Formatted at 2026-07-09 13:07:07 on dist-test-slave-k73n"
I20260709 13:07:07.901286 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:07:07.920709 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:07:07.921902 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:07:07.959280 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.126:41765
I20260709 13:07:07.959371 24692 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.126:41765 every 8 connection(s)
I20260709 13:07:07.963435 24693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:07.972560 24693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea: Bootstrap starting.
I20260709 13:07:07.977245 24693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:07.981909 24693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea: No bootstrap required, opened a new log
I20260709 13:07:07.983894 24693 raft_consensus.cc:359] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36709ee497be48979944e8465cd39dea" member_type: VOTER }
I20260709 13:07:07.984282 24693 raft_consensus.cc:385] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:07.984513 24693 raft_consensus.cc:740] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36709ee497be48979944e8465cd39dea, State: Initialized, Role: FOLLOWER
I20260709 13:07:07.985176 24693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36709ee497be48979944e8465cd39dea" member_type: VOTER }
I20260709 13:07:07.985623 24693 raft_consensus.cc:399] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:07:07.985839 24693 raft_consensus.cc:493] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:07:07.986131 24693 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:07.990001 24693 raft_consensus.cc:515] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36709ee497be48979944e8465cd39dea" member_type: VOTER }
I20260709 13:07:07.990494 24693 leader_election.cc:304] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36709ee497be48979944e8465cd39dea; no voters: 
I20260709 13:07:07.991549 24693 leader_election.cc:290] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:07:07.991835 24696 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:07.993160 24696 raft_consensus.cc:697] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [term 1 LEADER]: Becoming Leader. State: Replica: 36709ee497be48979944e8465cd39dea, State: Running, Role: LEADER
I20260709 13:07:07.993827 24696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36709ee497be48979944e8465cd39dea" member_type: VOTER }
I20260709 13:07:07.994441 24693 sys_catalog.cc:565] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:07:07.998584 24698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36709ee497be48979944e8465cd39dea. Latest consensus state: current_term: 1 leader_uuid: "36709ee497be48979944e8465cd39dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36709ee497be48979944e8465cd39dea" member_type: VOTER } }
I20260709 13:07:07.998755 24697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36709ee497be48979944e8465cd39dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36709ee497be48979944e8465cd39dea" member_type: VOTER } }
I20260709 13:07:07.999316 24698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [sys.catalog]: This master's current role is: LEADER
I20260709 13:07:07.999461 24697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [sys.catalog]: This master's current role is: LEADER
I20260709 13:07:08.003144 24703 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:07:08.007994 24703 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:07:08.010298 24113 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:07:08.016372 24703 catalog_manager.cc:1383] Generated new cluster ID: 35d743720dca43729edbd58eeb31522d
I20260709 13:07:08.016665 24703 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:07:08.039206 24703 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:07:08.040374 24703 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:07:08.051659 24703 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea: Generated new TSK 0
I20260709 13:07:08.052222 24703 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:07:08.076327 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:07:08.081894 24714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:08.083055 24715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:08.084226 24717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:08.084525 24113 server_base.cc:1061] running on GCE node
I20260709 13:07:08.085434 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:07:08.085604 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:07:08.085727 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602428085717 us; error 0 us; skew 500 ppm
I20260709 13:07:08.086169 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:07:08.088227 24113 webserver.cc:533] Webserver started at http://127.23.140.65:33371/ using document root <none> and password file <none>
I20260709 13:07:08.088644 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:07:08.088799 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:07:08.089003 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:07:08.090030 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/ts-0-root/instance:
uuid: "04c981c15b3b481293007baaeeca8a94"
format_stamp: "Formatted at 2026-07-09 13:07:08 on dist-test-slave-k73n"
I20260709 13:07:08.094033 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 13:07:08.096989 24722 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:08.097748 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 13:07:08.098110 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/ts-0-root
uuid: "04c981c15b3b481293007baaeeca8a94"
format_stamp: "Formatted at 2026-07-09 13:07:08 on dist-test-slave-k73n"
I20260709 13:07:08.098340 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:07:08.114564 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:07:08.115643 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:07:08.116953 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:07:08.119122 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:07:08.119284 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:08.119573 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:07:08.119750 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:08.158170 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.65:46481
I20260709 13:07:08.158274 24784 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.65:46481 every 8 connection(s)
I20260709 13:07:08.162917 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:07:08.170605 24789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:08.172317 24790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:08.175379 24792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:08.178157 24113 server_base.cc:1061] running on GCE node
I20260709 13:07:08.178256 24785 heartbeater.cc:344] Connected to a master server at 127.23.140.126:41765
I20260709 13:07:08.178649 24785 heartbeater.cc:461] Registering TS with master...
I20260709 13:07:08.179286 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:07:08.179543 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:07:08.179515 24785 heartbeater.cc:507] Master 127.23.140.126:41765 requested a full tablet report, sending...
I20260709 13:07:08.179854 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602428179835 us; error 0 us; skew 500 ppm
I20260709 13:07:08.180579 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:07:08.182063 24658 ts_manager.cc:194] Registered new tserver with Master: 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481)
I20260709 13:07:08.183022 24113 webserver.cc:533] Webserver started at http://127.23.140.66:43563/ using document root <none> and password file <none>
I20260709 13:07:08.183564 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:07:08.183816 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:07:08.184175 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:07:08.184224 24658 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46884
I20260709 13:07:08.185535 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/ts-1-root/instance:
uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
format_stamp: "Formatted at 2026-07-09 13:07:08 on dist-test-slave-k73n"
I20260709 13:07:08.189946 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.004s
I20260709 13:07:08.192945 24797 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:08.193670 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260709 13:07:08.193898 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/ts-1-root
uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
format_stamp: "Formatted at 2026-07-09 13:07:08 on dist-test-slave-k73n"
I20260709 13:07:08.194118 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:07:08.210032 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:07:08.211081 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:07:08.212493 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:07:08.214632 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:07:08.214797 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:08.214965 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:07:08.215085 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:08.253489 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.66:33759
I20260709 13:07:08.253597 24859 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.66:33759 every 8 connection(s)
I20260709 13:07:08.258025 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:07:08.264832 24863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:08.265971 24864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:08.270066 24866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:08.270798 24113 server_base.cc:1061] running on GCE node
I20260709 13:07:08.271157 24860 heartbeater.cc:344] Connected to a master server at 127.23.140.126:41765
I20260709 13:07:08.271531 24860 heartbeater.cc:461] Registering TS with master...
I20260709 13:07:08.271575 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:07:08.271867 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:07:08.272038 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602428272021 us; error 0 us; skew 500 ppm
I20260709 13:07:08.272385 24860 heartbeater.cc:507] Master 127.23.140.126:41765 requested a full tablet report, sending...
I20260709 13:07:08.272631 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:07:08.274399 24658 ts_manager.cc:194] Registered new tserver with Master: 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:08.275146 24113 webserver.cc:533] Webserver started at http://127.23.140.67:36123/ using document root <none> and password file <none>
I20260709 13:07:08.275743 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:07:08.275975 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:07:08.276126 24658 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46898
I20260709 13:07:08.276291 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:07:08.277704 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/ts-2-root/instance:
uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
format_stamp: "Formatted at 2026-07-09 13:07:08 on dist-test-slave-k73n"
I20260709 13:07:08.281819 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20260709 13:07:08.284703 24871 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:08.285427 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260709 13:07:08.285705 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/ts-2-root
uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
format_stamp: "Formatted at 2026-07-09 13:07:08 on dist-test-slave-k73n"
I20260709 13:07:08.285984 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602412999163-24113-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:07:08.298604 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:07:08.299703 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:07:08.301111 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:07:08.303263 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:07:08.303491 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:08.303681 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:07:08.303908 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:08.343308 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.67:37083
I20260709 13:07:08.343401 24933 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.67:37083 every 8 connection(s)
I20260709 13:07:08.356719 24934 heartbeater.cc:344] Connected to a master server at 127.23.140.126:41765
I20260709 13:07:08.357118 24934 heartbeater.cc:461] Registering TS with master...
I20260709 13:07:08.357827 24934 heartbeater.cc:507] Master 127.23.140.126:41765 requested a full tablet report, sending...
I20260709 13:07:08.359709 24658 ts_manager.cc:194] Registered new tserver with Master: 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:08.360486 24113 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014252427s
I20260709 13:07:08.361202 24658 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46900
I20260709 13:07:08.383931 24658 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46912:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:07:08.386644 24658 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:07:08.455170 24750 tablet_service.cc:1511] Processing CreateTablet for tablet 519702cc31404b58a9f7c00e5e012da5 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 13:07:08.455639 24748 tablet_service.cc:1511] Processing CreateTablet for tablet e1196843f61e4f5a92a158a876ead792 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 13:07:08.456621 24750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 519702cc31404b58a9f7c00e5e012da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.457819 24748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1196843f61e4f5a92a158a876ead792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.457739 24747 tablet_service.cc:1511] Processing CreateTablet for tablet 40603ccb1d484c139ca4806440e17898 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 13:07:08.459162 24747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40603ccb1d484c139ca4806440e17898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.458781 24749 tablet_service.cc:1511] Processing CreateTablet for tablet 22c9ddb498a04387bd6e576b0cce53b8 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 13:07:08.469923 24749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22c9ddb498a04387bd6e576b0cce53b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.481650 24825 tablet_service.cc:1511] Processing CreateTablet for tablet 519702cc31404b58a9f7c00e5e012da5 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 13:07:08.483331 24825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 519702cc31404b58a9f7c00e5e012da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.489894 24748 tablet_service.cc:1511] Processing CreateTablet for tablet aa3c2d8eadd9456a8490eb4671f18b82 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 13:07:08.491205 24748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa3c2d8eadd9456a8490eb4671f18b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.496030 24949 tablet_bootstrap.cc:492] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94: Bootstrap starting.
I20260709 13:07:08.496373 24750 tablet_service.cc:1511] Processing CreateTablet for tablet e31dbaadd96443bcafd21ec6a5cd19c4 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 13:07:08.496737 24824 tablet_service.cc:1511] Processing CreateTablet for tablet 22c9ddb498a04387bd6e576b0cce53b8 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 13:07:08.497635 24750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e31dbaadd96443bcafd21ec6a5cd19c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.498064 24824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22c9ddb498a04387bd6e576b0cce53b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.498895 24823 tablet_service.cc:1511] Processing CreateTablet for tablet e1196843f61e4f5a92a158a876ead792 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 13:07:08.500139 24823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1196843f61e4f5a92a158a876ead792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.510816 24822 tablet_service.cc:1511] Processing CreateTablet for tablet 40603ccb1d484c139ca4806440e17898 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 13:07:08.511576 24949 tablet_bootstrap.cc:654] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:08.512107 24822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40603ccb1d484c139ca4806440e17898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.515060 24899 tablet_service.cc:1511] Processing CreateTablet for tablet 519702cc31404b58a9f7c00e5e012da5 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 13:07:08.516261 24899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 519702cc31404b58a9f7c00e5e012da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.523175 24898 tablet_service.cc:1511] Processing CreateTablet for tablet 22c9ddb498a04387bd6e576b0cce53b8 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 13:07:08.524914 24898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22c9ddb498a04387bd6e576b0cce53b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.520364 24747 tablet_service.cc:1511] Processing CreateTablet for tablet e5e8e388810f4364b62faefb3c1ec45f (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 13:07:08.526087 24747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e8e388810f4364b62faefb3c1ec45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.531993 24749 tablet_service.cc:1511] Processing CreateTablet for tablet 24b03eb9381040d4b92e89e870caa867 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 13:07:08.533288 24749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24b03eb9381040d4b92e89e870caa867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.543485 24897 tablet_service.cc:1511] Processing CreateTablet for tablet e1196843f61e4f5a92a158a876ead792 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 13:07:08.545425 24897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1196843f61e4f5a92a158a876ead792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.545790 24746 tablet_service.cc:1511] Processing CreateTablet for tablet 4425bffab9a34e1aa5b95565fe6808f4 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 13:07:08.547086 24746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4425bffab9a34e1aa5b95565fe6808f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.555744 24896 tablet_service.cc:1511] Processing CreateTablet for tablet 40603ccb1d484c139ca4806440e17898 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 13:07:08.559880 24896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40603ccb1d484c139ca4806440e17898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.562597 24950 tablet_bootstrap.cc:492] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f: Bootstrap starting.
I20260709 13:07:08.565440 24895 tablet_service.cc:1511] Processing CreateTablet for tablet aa3c2d8eadd9456a8490eb4671f18b82 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 13:07:08.566792 24895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa3c2d8eadd9456a8490eb4671f18b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.568511 24949 tablet_bootstrap.cc:492] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94: No bootstrap required, opened a new log
I20260709 13:07:08.568945 24949 ts_tablet_manager.cc:1403] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94: Time spent bootstrapping tablet: real 0.073s	user 0.008s	sys 0.006s
I20260709 13:07:08.571808 24949 raft_consensus.cc:359] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.572508 24949 raft_consensus.cc:385] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:08.572798 24949 raft_consensus.cc:740] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Initialized, Role: FOLLOWER
I20260709 13:07:08.576726 24824 tablet_service.cc:1511] Processing CreateTablet for tablet aa3c2d8eadd9456a8490eb4671f18b82 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 13:07:08.576818 24750 tablet_service.cc:1511] Processing CreateTablet for tablet fab2dc5fe6b24dd180b77f547c8471c5 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 13:07:08.578168 24824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa3c2d8eadd9456a8490eb4671f18b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.578250 24750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab2dc5fe6b24dd180b77f547c8471c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.579447 24825 tablet_service.cc:1511] Processing CreateTablet for tablet e31dbaadd96443bcafd21ec6a5cd19c4 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 13:07:08.580785 24825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e31dbaadd96443bcafd21ec6a5cd19c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.581472 24950 tablet_bootstrap.cc:654] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:08.576128 24949 consensus_queue.cc:260] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.583680 24823 tablet_service.cc:1511] Processing CreateTablet for tablet e5e8e388810f4364b62faefb3c1ec45f (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 13:07:08.585255 24823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e8e388810f4364b62faefb3c1ec45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.586122 24896 tablet_service.cc:1511] Processing CreateTablet for tablet e31dbaadd96443bcafd21ec6a5cd19c4 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 13:07:08.587391 24896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e31dbaadd96443bcafd21ec6a5cd19c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.597309 24949 ts_tablet_manager.cc:1434] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94: Time spent starting tablet: real 0.028s	user 0.008s	sys 0.004s
I20260709 13:07:08.599891 24785 heartbeater.cc:499] Master 127.23.140.126:41765 was elected leader, sending a full tablet report...
I20260709 13:07:08.600306 24949 tablet_bootstrap.cc:492] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94: Bootstrap starting.
I20260709 13:07:08.608078 24949 tablet_bootstrap.cc:654] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:08.610522 24894 tablet_service.cc:1511] Processing CreateTablet for tablet e5e8e388810f4364b62faefb3c1ec45f (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 13:07:08.610949 24821 tablet_service.cc:1511] Processing CreateTablet for tablet 24b03eb9381040d4b92e89e870caa867 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 13:07:08.612125 24894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e8e388810f4364b62faefb3c1ec45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.612299 24821 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24b03eb9381040d4b92e89e870caa867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.623862 24950 tablet_bootstrap.cc:492] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f: No bootstrap required, opened a new log
I20260709 13:07:08.624461 24950 ts_tablet_manager.cc:1403] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent bootstrapping tablet: real 0.062s	user 0.015s	sys 0.000s
I20260709 13:07:08.629230 24824 tablet_service.cc:1511] Processing CreateTablet for tablet 4425bffab9a34e1aa5b95565fe6808f4 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 13:07:08.630927 24824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4425bffab9a34e1aa5b95565fe6808f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.639607 24820 tablet_service.cc:1511] Processing CreateTablet for tablet fab2dc5fe6b24dd180b77f547c8471c5 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 13:07:08.640370 24893 tablet_service.cc:1511] Processing CreateTablet for tablet fab2dc5fe6b24dd180b77f547c8471c5 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 13:07:08.640929 24820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab2dc5fe6b24dd180b77f547c8471c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.641597 24893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab2dc5fe6b24dd180b77f547c8471c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.648286 24950 raft_consensus.cc:359] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.660341 24954 raft_consensus.cc:493] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:08.660928 24952 tablet_bootstrap.cc:492] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Bootstrap starting.
I20260709 13:07:08.660820 24954 raft_consensus.cc:515] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.665647 24894 tablet_service.cc:1511] Processing CreateTablet for tablet 4425bffab9a34e1aa5b95565fe6808f4 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 13:07:08.667279 24952 tablet_bootstrap.cc:654] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:08.676051 24894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4425bffab9a34e1aa5b95565fe6808f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.668073 24954 leader_election.cc:290] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:08.660357 24950 raft_consensus.cc:385] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:08.674822 24949 tablet_bootstrap.cc:492] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94: No bootstrap required, opened a new log
I20260709 13:07:08.678612 24949 ts_tablet_manager.cc:1403] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94: Time spent bootstrapping tablet: real 0.079s	user 0.007s	sys 0.005s
I20260709 13:07:08.681463 24949 raft_consensus.cc:359] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.681559 24897 tablet_service.cc:1511] Processing CreateTablet for tablet 24b03eb9381040d4b92e89e870caa867 (DEFAULT_TABLE table=test-workload [id=f6d62284c27f4de089e4dd780a43c614]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 13:07:08.682241 24949 raft_consensus.cc:385] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:08.682574 24949 raft_consensus.cc:740] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Initialized, Role: FOLLOWER
I20260709 13:07:08.682889 24897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24b03eb9381040d4b92e89e870caa867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:08.683450 24949 consensus_queue.cc:260] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.685866 24949 ts_tablet_manager.cc:1434] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 13:07:08.686761 24949 tablet_bootstrap.cc:492] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94: Bootstrap starting.
I20260709 13:07:08.689332 24950 raft_consensus.cc:740] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Initialized, Role: FOLLOWER
I20260709 13:07:08.690160 24950 consensus_queue.cc:260] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.693028 24949 tablet_bootstrap.cc:654] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:08.694826 24952 tablet_bootstrap.cc:492] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4: No bootstrap required, opened a new log
I20260709 13:07:08.695351 24952 ts_tablet_manager.cc:1403] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent bootstrapping tablet: real 0.035s	user 0.008s	sys 0.004s
I20260709 13:07:08.696827 24860 heartbeater.cc:499] Master 127.23.140.126:41765 was elected leader, sending a full tablet report...
I20260709 13:07:08.698371 24952 raft_consensus.cc:359] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } }
I20260709 13:07:08.699077 24952 raft_consensus.cc:385] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:08.699363 24952 raft_consensus.cc:740] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Initialized, Role: FOLLOWER
I20260709 13:07:08.701535 24950 ts_tablet_manager.cc:1434] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent starting tablet: real 0.077s	user 0.006s	sys 0.005s
I20260709 13:07:08.702558 24950 tablet_bootstrap.cc:492] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f: Bootstrap starting.
I20260709 13:07:08.710506 24950 tablet_bootstrap.cc:654] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:08.710417 24952 consensus_queue.cc:260] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } }
I20260709 13:07:08.716310 24934 heartbeater.cc:499] Master 127.23.140.126:41765 was elected leader, sending a full tablet report...
I20260709 13:07:08.717741 24952 ts_tablet_manager.cc:1434] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.001s
I20260709 13:07:08.718699 24952 tablet_bootstrap.cc:492] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Bootstrap starting.
I20260709 13:07:08.719564 24949 tablet_bootstrap.cc:492] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94: No bootstrap required, opened a new log
I20260709 13:07:08.720010 24949 ts_tablet_manager.cc:1403] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94: Time spent bootstrapping tablet: real 0.033s	user 0.017s	sys 0.001s
I20260709 13:07:08.720515 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40603ccb1d484c139ca4806440e17898" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" is_pre_election: true
I20260709 13:07:08.721822 24909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40603ccb1d484c139ca4806440e17898" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" is_pre_election: true
I20260709 13:07:08.722838 24949 raft_consensus.cc:359] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.723546 24949 raft_consensus.cc:385] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:08.723838 24949 raft_consensus.cc:740] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Initialized, Role: FOLLOWER
I20260709 13:07:08.724546 24949 consensus_queue.cc:260] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
W20260709 13:07:08.725323 24723 leader_election.cc:343] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:08.725986 24723 leader_election.cc:343] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:08.726418 24723 leader_election.cc:304] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94; no voters: 1b0ad47f96d64c6f96c0f303d8b68fd4, 9a1e5371bf904722bfabfdb1c97a5f7f
I20260709 13:07:08.726693 24949 ts_tablet_manager.cc:1434] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 13:07:08.727646 24949 tablet_bootstrap.cc:492] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94: Bootstrap starting.
I20260709 13:07:08.727876 24954 raft_consensus.cc:2749] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:08.730561 24950 tablet_bootstrap.cc:492] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f: No bootstrap required, opened a new log
I20260709 13:07:08.725956 24952 tablet_bootstrap.cc:654] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:08.731108 24950 ts_tablet_manager.cc:1403] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent bootstrapping tablet: real 0.029s	user 0.013s	sys 0.000s
I20260709 13:07:08.734012 24950 raft_consensus.cc:359] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.734807 24950 raft_consensus.cc:385] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:08.735103 24949 tablet_bootstrap.cc:654] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:08.735122 24950 raft_consensus.cc:740] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Initialized, Role: FOLLOWER
I20260709 13:07:08.736176 24950 consensus_queue.cc:260] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.736793 24952 tablet_bootstrap.cc:492] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4: No bootstrap required, opened a new log
I20260709 13:07:08.737695 24952 ts_tablet_manager.cc:1403] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent bootstrapping tablet: real 0.019s	user 0.008s	sys 0.004s
I20260709 13:07:08.738806 24950 ts_tablet_manager.cc:1434] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:07:08.739521 24950 tablet_bootstrap.cc:492] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f: Bootstrap starting.
I20260709 13:07:08.741259 24952 raft_consensus.cc:359] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.742174 24952 raft_consensus.cc:385] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:08.742530 24952 raft_consensus.cc:740] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Initialized, Role: FOLLOWER
I20260709 13:07:08.743467 24952 consensus_queue.cc:260] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.744987 24950 tablet_bootstrap.cc:654] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:08.746220 24952 ts_tablet_manager.cc:1434] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:07:08.747411 24952 tablet_bootstrap.cc:492] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Bootstrap starting.
I20260709 13:07:08.755496 24960 raft_consensus.cc:493] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:08.755990 24952 tablet_bootstrap.cc:654] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:08.756116 24960 raft_consensus.cc:515] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } }
I20260709 13:07:08.770587 24952 tablet_bootstrap.cc:492] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4: No bootstrap required, opened a new log
I20260709 13:07:08.771203 24952 ts_tablet_manager.cc:1403] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent bootstrapping tablet: real 0.024s	user 0.012s	sys 0.008s
I20260709 13:07:08.774858 24952 raft_consensus.cc:359] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.775806 24952 raft_consensus.cc:385] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:08.776165 24952 raft_consensus.cc:740] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Initialized, Role: FOLLOWER
I20260709 13:07:08.777499 24952 consensus_queue.cc:260] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.780264 24952 ts_tablet_manager.cc:1434] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 13:07:08.781436 24952 tablet_bootstrap.cc:492] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4: Bootstrap starting.
I20260709 13:07:08.786185 24960 leader_election.cc:290] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:08.788757 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "519702cc31404b58a9f7c00e5e012da5" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" is_pre_election: true
W20260709 13:07:08.791590 24872 leader_election.cc:343] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:08.794566 24958 raft_consensus.cc:493] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:08.789611 24952 tablet_bootstrap.cc:654] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:08.795033 24958 raft_consensus.cc:515] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.797838 24958 leader_election.cc:290] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:08.823733 24949 tablet_bootstrap.cc:492] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94: No bootstrap required, opened a new log
I20260709 13:07:08.824391 24949 ts_tablet_manager.cc:1403] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94: Time spent bootstrapping tablet: real 0.097s	user 0.034s	sys 0.032s
I20260709 13:07:08.835647 24949 raft_consensus.cc:359] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } }
I20260709 13:07:08.836403 24949 raft_consensus.cc:385] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:08.836712 24949 raft_consensus.cc:740] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Initialized, Role: FOLLOWER
I20260709 13:07:08.840214 24760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1196843f61e4f5a92a158a876ead792" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94" is_pre_election: true
I20260709 13:07:08.841224 24760 raft_consensus.cc:2468] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 0.
I20260709 13:07:08.842584 24800 leader_election.cc:304] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:08.842917 24952 tablet_bootstrap.cc:492] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4: No bootstrap required, opened a new log
I20260709 13:07:08.843395 24952 ts_tablet_manager.cc:1403] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent bootstrapping tablet: real 0.062s	user 0.019s	sys 0.014s
I20260709 13:07:08.843494 24958 raft_consensus.cc:2804] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:08.843851 24958 raft_consensus.cc:493] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:08.844570 24958 raft_consensus.cc:3060] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:08.844326 24949 consensus_queue.cc:260] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } }
I20260709 13:07:08.846794 24952 raft_consensus.cc:359] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.847605 24952 raft_consensus.cc:385] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:08.847940 24952 raft_consensus.cc:740] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Initialized, Role: FOLLOWER
I20260709 13:07:08.846786 24950 tablet_bootstrap.cc:492] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f: No bootstrap required, opened a new log
I20260709 13:07:08.848590 24950 ts_tablet_manager.cc:1403] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent bootstrapping tablet: real 0.109s	user 0.043s	sys 0.061s
I20260709 13:07:08.848932 24952 consensus_queue.cc:260] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.851495 24958 raft_consensus.cc:515] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.851398 24950 raft_consensus.cc:359] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } }
I20260709 13:07:08.852219 24950 raft_consensus.cc:385] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:08.852526 24950 raft_consensus.cc:740] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Initialized, Role: FOLLOWER
I20260709 13:07:08.853291 24950 consensus_queue.cc:260] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } }
I20260709 13:07:08.854074 24952 ts_tablet_manager.cc:1434] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.001s
I20260709 13:07:08.854210 24760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1196843f61e4f5a92a158a876ead792" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:08.856109 24760 raft_consensus.cc:3060] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:08.858387 24759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "519702cc31404b58a9f7c00e5e012da5" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94" is_pre_election: true
I20260709 13:07:08.859035 24759 raft_consensus.cc:2468] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 0.
I20260709 13:07:08.860198 24874 leader_election.cc:304] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 1b0ad47f96d64c6f96c0f303d8b68fd4; no voters: 9a1e5371bf904722bfabfdb1c97a5f7f
I20260709 13:07:08.860709 24958 leader_election.cc:290] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:08.861387 24966 raft_consensus.cc:2804] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:08.861788 24966 raft_consensus.cc:493] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:08.862136 24966 raft_consensus.cc:3060] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:08.863171 24960 raft_consensus.cc:493] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:08.863624 24960 raft_consensus.cc:515] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.865674 24960 leader_election.cc:290] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:08.866496 24759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40603ccb1d484c139ca4806440e17898" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94" is_pre_election: true
I20260709 13:07:08.866791 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40603ccb1d484c139ca4806440e17898" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" is_pre_election: true
I20260709 13:07:08.867089 24759 raft_consensus.cc:2468] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 0.
I20260709 13:07:08.866979 24909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1196843f61e4f5a92a158a876ead792" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" is_pre_election: true
I20260709 13:07:08.867890 24909 raft_consensus.cc:2468] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 0.
W20260709 13:07:08.868150 24872 leader_election.cc:343] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:08.869472 24952 tablet_bootstrap.cc:492] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Bootstrap starting.
I20260709 13:07:08.869338 24908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1196843f61e4f5a92a158a876ead792" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:08.869693 24874 leader_election.cc:304] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 1b0ad47f96d64c6f96c0f303d8b68fd4; no voters: 9a1e5371bf904722bfabfdb1c97a5f7f
I20260709 13:07:08.870162 24908 raft_consensus.cc:3060] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:08.870730 24960 raft_consensus.cc:2804] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:08.871109 24960 raft_consensus.cc:493] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:08.871443 24960 raft_consensus.cc:3060] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:08.876652 24952 tablet_bootstrap.cc:654] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:08.883004 24908 raft_consensus.cc:2468] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 1.
I20260709 13:07:08.883246 24966 raft_consensus.cc:515] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } }
I20260709 13:07:08.884814 24798 leader_election.cc:304] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:08.885998 24958 raft_consensus.cc:2804] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:08.886593 24958 raft_consensus.cc:697] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Becoming Leader. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Running, Role: LEADER
I20260709 13:07:08.856159 24949 ts_tablet_manager.cc:1434] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94: Time spent starting tablet: real 0.031s	user 0.006s	sys 0.003s
I20260709 13:07:08.887701 24958 consensus_queue.cc:237] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.891944 24949 tablet_bootstrap.cc:492] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94: Bootstrap starting.
I20260709 13:07:08.887537 24960 raft_consensus.cc:515] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.897210 24759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "519702cc31404b58a9f7c00e5e012da5" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:08.894908 24760 raft_consensus.cc:2468] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 1.
I20260709 13:07:08.897962 24759 raft_consensus.cc:3060] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:08.908854 24657 catalog_manager.cc:5697] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:08.916982 24958 raft_consensus.cc:493] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:08.917641 24958 raft_consensus.cc:515] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.919209 24759 raft_consensus.cc:2468] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 1.
I20260709 13:07:08.919219 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "519702cc31404b58a9f7c00e5e012da5" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:08.919855 24835 raft_consensus.cc:3060] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:08.920004 24966 leader_election.cc:290] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:08.921401 24874 leader_election.cc:304] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 1b0ad47f96d64c6f96c0f303d8b68fd4; no voters: 
I20260709 13:07:08.922755 24760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40603ccb1d484c139ca4806440e17898" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:08.923482 24760 raft_consensus.cc:3060] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:08.925282 24966 raft_consensus.cc:2804] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:08.927663 24834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40603ccb1d484c139ca4806440e17898" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:08.928369 24958 leader_election.cc:290] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
W20260709 13:07:08.929248 24872 leader_election.cc:343] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:08.929925 24835 raft_consensus.cc:2468] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 1.
I20260709 13:07:08.930357 24908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22c9ddb498a04387bd6e576b0cce53b8" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" is_pre_election: true
W20260709 13:07:08.932760 24798 leader_election.cc:343] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:08.933605 24960 leader_election.cc:290] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:08.934509 24760 raft_consensus.cc:2468] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 1.
I20260709 13:07:08.935905 24874 leader_election.cc:304] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 1b0ad47f96d64c6f96c0f303d8b68fd4; no voters: 9a1e5371bf904722bfabfdb1c97a5f7f
I20260709 13:07:08.936177 24759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22c9ddb498a04387bd6e576b0cce53b8" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94" is_pre_election: true
I20260709 13:07:08.936830 24950 ts_tablet_manager.cc:1434] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent starting tablet: real 0.088s	user 0.041s	sys 0.033s
I20260709 13:07:08.936966 24759 raft_consensus.cc:2468] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 0.
I20260709 13:07:08.937884 24950 tablet_bootstrap.cc:492] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f: Bootstrap starting.
I20260709 13:07:08.938493 24800 leader_election.cc:304] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 1b0ad47f96d64c6f96c0f303d8b68fd4
I20260709 13:07:08.939838 24960 raft_consensus.cc:2804] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:08.940876 24972 raft_consensus.cc:2804] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:08.941319 24972 raft_consensus.cc:493] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:08.941569 24976 raft_consensus.cc:493] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:08.942122 24960 raft_consensus.cc:697] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Running, Role: LEADER
I20260709 13:07:08.942276 24976 raft_consensus.cc:515] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.942982 24960 consensus_queue.cc:237] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.943807 24950 tablet_bootstrap.cc:654] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:08.945185 24966 raft_consensus.cc:697] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Running, Role: LEADER
I20260709 13:07:08.945843 24760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e8e388810f4364b62faefb3c1ec45f" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94" is_pre_election: true
I20260709 13:07:08.946313 24966 consensus_queue.cc:237] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } }
W20260709 13:07:08.947310 24874 leader_election.cc:343] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:08.950115 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e8e388810f4364b62faefb3c1ec45f" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" is_pre_election: true
W20260709 13:07:08.951941 24872 leader_election.cc:343] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:08.952491 24872 leader_election.cc:304] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4; no voters: 04c981c15b3b481293007baaeeca8a94, 9a1e5371bf904722bfabfdb1c97a5f7f
I20260709 13:07:08.953610 24975 raft_consensus.cc:2749] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:08.941666 24972 raft_consensus.cc:3060] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:08.924971 24949 tablet_bootstrap.cc:654] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:08.955075 24976 leader_election.cc:290] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:08.955335 24950 tablet_bootstrap.cc:492] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f: No bootstrap required, opened a new log
I20260709 13:07:08.955770 24950 ts_tablet_manager.cc:1403] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent bootstrapping tablet: real 0.018s	user 0.004s	sys 0.011s
I20260709 13:07:08.958724 24950 raft_consensus.cc:359] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.959442 24950 raft_consensus.cc:385] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:08.959746 24950 raft_consensus.cc:740] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Initialized, Role: FOLLOWER
I20260709 13:07:08.961019 24950 consensus_queue.cc:260] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.963429 24972 raft_consensus.cc:515] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.966378 24950 ts_tablet_manager.cc:1434] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 13:07:08.966136 24657 catalog_manager.cc:5697] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:08.967659 24950 tablet_bootstrap.cc:492] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f: Bootstrap starting.
I20260709 13:07:08.976213 24950 tablet_bootstrap.cc:654] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:08.977211 24952 tablet_bootstrap.cc:492] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4: No bootstrap required, opened a new log
I20260709 13:07:08.977669 24952 ts_tablet_manager.cc:1403] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent bootstrapping tablet: real 0.108s	user 0.033s	sys 0.032s
I20260709 13:07:08.980490 24952 raft_consensus.cc:359] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.981477 24952 raft_consensus.cc:385] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:08.981774 24952 raft_consensus.cc:740] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Initialized, Role: FOLLOWER
I20260709 13:07:08.982504 24952 consensus_queue.cc:260] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.983111 24950 tablet_bootstrap.cc:492] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f: No bootstrap required, opened a new log
I20260709 13:07:08.983620 24950 ts_tablet_manager.cc:1403] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent bootstrapping tablet: real 0.016s	user 0.012s	sys 0.000s
I20260709 13:07:08.983024 24657 catalog_manager.cc:5697] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:08.984736 24952 ts_tablet_manager.cc:1434] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 13:07:08.985653 24952 tablet_bootstrap.cc:492] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Bootstrap starting.
I20260709 13:07:08.986071 24950 raft_consensus.cc:359] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:08.986797 24950 raft_consensus.cc:385] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:08.986793 24760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22c9ddb498a04387bd6e576b0cce53b8" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:08.987243 24950 raft_consensus.cc:740] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Initialized, Role: FOLLOWER
I20260709 13:07:08.987526 24760 raft_consensus.cc:3060] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:08.987857 24908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22c9ddb498a04387bd6e576b0cce53b8" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:08.988183 24950 consensus_queue.cc:260] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
W20260709 13:07:08.989387 24798 leader_election.cc:343] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:08.989981 24972 leader_election.cc:290] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:08.992023 24950 ts_tablet_manager.cc:1434] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:07:08.992780 24950 tablet_bootstrap.cc:492] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f: Bootstrap starting.
I20260709 13:07:08.994335 24760 raft_consensus.cc:2468] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 1.
I20260709 13:07:08.995087 24952 tablet_bootstrap.cc:654] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:08.995517 24800 leader_election.cc:304] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 1b0ad47f96d64c6f96c0f303d8b68fd4
I20260709 13:07:08.997687 24949 tablet_bootstrap.cc:492] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94: No bootstrap required, opened a new log
I20260709 13:07:08.998081 24950 tablet_bootstrap.cc:654] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:08.998263 24949 ts_tablet_manager.cc:1403] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94: Time spent bootstrapping tablet: real 0.107s	user 0.011s	sys 0.000s
I20260709 13:07:09.000128 24972 raft_consensus.cc:2804] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:09.001080 24972 raft_consensus.cc:697] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Becoming Leader. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Running, Role: LEADER
I20260709 13:07:09.002139 24972 consensus_queue.cc:237] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.003126 24960 raft_consensus.cc:493] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:09.007588 24950 tablet_bootstrap.cc:492] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f: No bootstrap required, opened a new log
I20260709 13:07:09.007521 24960 raft_consensus.cc:515] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.008060 24950 ts_tablet_manager.cc:1403] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent bootstrapping tablet: real 0.015s	user 0.004s	sys 0.009s
W20260709 13:07:09.009043 24706 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:07:09.009711 24960 leader_election.cc:290] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:09.011034 24952 tablet_bootstrap.cc:492] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4: No bootstrap required, opened a new log
I20260709 13:07:09.010842 24950 raft_consensus.cc:359] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.011477 24952 ts_tablet_manager.cc:1403] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent bootstrapping tablet: real 0.026s	user 0.013s	sys 0.003s
I20260709 13:07:09.011547 24950 raft_consensus.cc:385] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:09.012032 24950 raft_consensus.cc:740] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Initialized, Role: FOLLOWER
I20260709 13:07:09.011616 24949 raft_consensus.cc:359] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.012480 24949 raft_consensus.cc:385] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:09.012771 24949 raft_consensus.cc:740] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Initialized, Role: FOLLOWER
I20260709 13:07:09.012869 24950 consensus_queue.cc:260] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.013604 24949 consensus_queue.cc:260] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.014473 24952 raft_consensus.cc:359] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.015142 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e31dbaadd96443bcafd21ec6a5cd19c4" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" is_pre_election: true
I20260709 13:07:09.016219 24950 ts_tablet_manager.cc:1434] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.005s
W20260709 13:07:09.016505 24872 leader_election.cc:343] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:09.015132 24952 raft_consensus.cc:385] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:09.017210 24950 tablet_bootstrap.cc:492] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f: Bootstrap starting.
I20260709 13:07:09.017334 24952 raft_consensus.cc:740] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Initialized, Role: FOLLOWER
I20260709 13:07:09.018122 24952 consensus_queue.cc:260] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.021029 24952 ts_tablet_manager.cc:1434] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 13:07:09.021607 24760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e31dbaadd96443bcafd21ec6a5cd19c4" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94" is_pre_election: true
I20260709 13:07:09.021926 24952 tablet_bootstrap.cc:492] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Bootstrap starting.
W20260709 13:07:09.022874 24874 leader_election.cc:343] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:09.023294 24874 leader_election.cc:304] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4; no voters: 04c981c15b3b481293007baaeeca8a94, 9a1e5371bf904722bfabfdb1c97a5f7f
I20260709 13:07:09.023804 24950 tablet_bootstrap.cc:654] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:09.024338 24960 raft_consensus.cc:2749] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:09.027750 24949 ts_tablet_manager.cc:1434] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94: Time spent starting tablet: real 0.029s	user 0.008s	sys 0.000s
I20260709 13:07:09.028393 24952 tablet_bootstrap.cc:654] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:09.030846 24949 tablet_bootstrap.cc:492] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94: Bootstrap starting.
I20260709 13:07:09.039000 24949 tablet_bootstrap.cc:654] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:09.048764 24655 catalog_manager.cc:5697] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:09.077250 24949 tablet_bootstrap.cc:492] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94: No bootstrap required, opened a new log
I20260709 13:07:09.077739 24949 ts_tablet_manager.cc:1403] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94: Time spent bootstrapping tablet: real 0.047s	user 0.019s	sys 0.027s
I20260709 13:07:09.078035 24950 tablet_bootstrap.cc:492] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f: No bootstrap required, opened a new log
I20260709 13:07:09.078449 24950 ts_tablet_manager.cc:1403] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent bootstrapping tablet: real 0.061s	user 0.032s	sys 0.028s
I20260709 13:07:09.079020 24952 tablet_bootstrap.cc:492] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4: No bootstrap required, opened a new log
I20260709 13:07:09.079452 24952 ts_tablet_manager.cc:1403] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent bootstrapping tablet: real 0.058s	user 0.025s	sys 0.028s
I20260709 13:07:09.080513 24950 raft_consensus.cc:359] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.080550 24949 raft_consensus.cc:359] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.081120 24950 raft_consensus.cc:385] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:09.081336 24949 raft_consensus.cc:385] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:09.081528 24950 raft_consensus.cc:740] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Initialized, Role: FOLLOWER
I20260709 13:07:09.081694 24949 raft_consensus.cc:740] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Initialized, Role: FOLLOWER
I20260709 13:07:09.082103 24952 raft_consensus.cc:359] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.082345 24950 consensus_queue.cc:260] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.082783 24952 raft_consensus.cc:385] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:09.082448 24949 consensus_queue.cc:260] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.083132 24952 raft_consensus.cc:740] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Initialized, Role: FOLLOWER
I20260709 13:07:09.083962 24952 consensus_queue.cc:260] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.084468 24950 ts_tablet_manager.cc:1434] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 13:07:09.085453 24950 tablet_bootstrap.cc:492] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f: Bootstrap starting.
I20260709 13:07:09.086140 24952 ts_tablet_manager.cc:1434] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.001s
I20260709 13:07:09.087360 24952 tablet_bootstrap.cc:492] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Bootstrap starting.
I20260709 13:07:09.091703 24950 tablet_bootstrap.cc:654] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:09.093464 24952 tablet_bootstrap.cc:654] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:09.102202 24949 ts_tablet_manager.cc:1434] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94: Time spent starting tablet: real 0.024s	user 0.007s	sys 0.000s
I20260709 13:07:09.102593 24950 tablet_bootstrap.cc:492] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f: No bootstrap required, opened a new log
I20260709 13:07:09.103036 24950 ts_tablet_manager.cc:1403] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent bootstrapping tablet: real 0.018s	user 0.015s	sys 0.000s
I20260709 13:07:09.103386 24949 tablet_bootstrap.cc:492] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94: Bootstrap starting.
I20260709 13:07:09.104878 24952 tablet_bootstrap.cc:492] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4: No bootstrap required, opened a new log
I20260709 13:07:09.105324 24952 ts_tablet_manager.cc:1403] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent bootstrapping tablet: real 0.018s	user 0.010s	sys 0.006s
I20260709 13:07:09.105664 24950 raft_consensus.cc:359] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.106309 24950 raft_consensus.cc:385] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:09.106568 24950 raft_consensus.cc:740] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Initialized, Role: FOLLOWER
I20260709 13:07:09.107474 24972 raft_consensus.cc:493] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:09.107373 24950 consensus_queue.cc:260] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.107771 24952 raft_consensus.cc:359] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.107896 24972 raft_consensus.cc:515] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.108464 24952 raft_consensus.cc:385] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:09.108754 24952 raft_consensus.cc:740] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Initialized, Role: FOLLOWER
I20260709 13:07:09.109583 24950 ts_tablet_manager.cc:1434] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.001s
I20260709 13:07:09.109710 24949 tablet_bootstrap.cc:654] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:09.109613 24952 consensus_queue.cc:260] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.110577 24950 tablet_bootstrap.cc:492] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f: Bootstrap starting.
I20260709 13:07:09.112035 24952 ts_tablet_manager.cc:1434] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:07:09.112874 24952 tablet_bootstrap.cc:492] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Bootstrap starting.
I20260709 13:07:09.113368 24958 raft_consensus.cc:493] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:09.113792 24958 raft_consensus.cc:515] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.114279 24972 leader_election.cc:290] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:09.115733 24958 leader_election.cc:290] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:09.115758 24908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b03eb9381040d4b92e89e870caa867" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" is_pre_election: true
I20260709 13:07:09.117930 24952 tablet_bootstrap.cc:654] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Neither blocks nor log segments found. Creating new log.
W20260709 13:07:09.118139 24798 leader_election.cc:343] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:09.119791 24760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b03eb9381040d4b92e89e870caa867" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94" is_pre_election: true
I20260709 13:07:09.120064 24908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e8e388810f4364b62faefb3c1ec45f" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" is_pre_election: true
I20260709 13:07:09.120700 24908 raft_consensus.cc:2468] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 0.
I20260709 13:07:09.121059 24759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e8e388810f4364b62faefb3c1ec45f" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94" is_pre_election: true
I20260709 13:07:09.121901 24798 leader_election.cc:304] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:09.122740 24958 raft_consensus.cc:2804] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:09.123085 24958 raft_consensus.cc:493] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:09.123431 24958 raft_consensus.cc:3060] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:09.123481 24800 leader_election.cc:343] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:09.124132 24800 leader_election.cc:304] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 04c981c15b3b481293007baaeeca8a94, 1b0ad47f96d64c6f96c0f303d8b68fd4
W20260709 13:07:09.125535 24800 leader_election.cc:343] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:09.126538 24972 raft_consensus.cc:2749] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:09.130231 24958 raft_consensus.cc:515] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.132817 24759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e8e388810f4364b62faefb3c1ec45f" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:09.132997 24908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e8e388810f4364b62faefb3c1ec45f" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:09.133847 24908 raft_consensus.cc:3060] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:09.134394 24800 leader_election.cc:343] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:09.135962 24950 tablet_bootstrap.cc:654] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:09.137444 24949 tablet_bootstrap.cc:492] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94: No bootstrap required, opened a new log
I20260709 13:07:09.137907 24949 ts_tablet_manager.cc:1403] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94: Time spent bootstrapping tablet: real 0.035s	user 0.010s	sys 0.004s
I20260709 13:07:09.138792 24952 tablet_bootstrap.cc:492] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4: No bootstrap required, opened a new log
I20260709 13:07:09.139249 24952 ts_tablet_manager.cc:1403] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent bootstrapping tablet: real 0.027s	user 0.018s	sys 0.005s
I20260709 13:07:09.136096 24958 leader_election.cc:290] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:09.140679 24949 raft_consensus.cc:359] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.141098 24908 raft_consensus.cc:2468] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 1.
I20260709 13:07:09.141379 24949 raft_consensus.cc:385] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:09.141791 24949 raft_consensus.cc:740] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Initialized, Role: FOLLOWER
I20260709 13:07:09.142123 24952 raft_consensus.cc:359] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.142798 24952 raft_consensus.cc:385] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:09.142549 24798 leader_election.cc:304] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 04c981c15b3b481293007baaeeca8a94
I20260709 13:07:09.142598 24949 consensus_queue.cc:260] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.143231 24952 raft_consensus.cc:740] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Initialized, Role: FOLLOWER
I20260709 13:07:09.143756 24958 raft_consensus.cc:2804] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:09.144254 24958 raft_consensus.cc:697] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Becoming Leader. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Running, Role: LEADER
I20260709 13:07:09.144117 24952 consensus_queue.cc:260] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.145043 24958 consensus_queue.cc:237] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.146353 24952 ts_tablet_manager.cc:1434] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 13:07:09.147763 24949 ts_tablet_manager.cc:1434] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.001s
I20260709 13:07:09.148054 24952 tablet_bootstrap.cc:492] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Bootstrap starting.
I20260709 13:07:09.148665 24949 tablet_bootstrap.cc:492] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94: Bootstrap starting.
I20260709 13:07:09.153951 24952 tablet_bootstrap.cc:654] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:09.168040 24950 tablet_bootstrap.cc:492] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f: No bootstrap required, opened a new log
I20260709 13:07:09.168493 24950 ts_tablet_manager.cc:1403] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent bootstrapping tablet: real 0.058s	user 0.009s	sys 0.012s
I20260709 13:07:09.169159 24952 tablet_bootstrap.cc:492] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4: No bootstrap required, opened a new log
I20260709 13:07:09.168525 24658 catalog_manager.cc:5697] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:09.169579 24952 ts_tablet_manager.cc:1403] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent bootstrapping tablet: real 0.022s	user 0.013s	sys 0.004s
I20260709 13:07:09.171407 24949 tablet_bootstrap.cc:654] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:09.171113 24950 raft_consensus.cc:359] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.171849 24950 raft_consensus.cc:385] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:09.172137 24950 raft_consensus.cc:740] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Initialized, Role: FOLLOWER
I20260709 13:07:09.171986 24952 raft_consensus.cc:359] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.172640 24952 raft_consensus.cc:385] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:09.172912 24952 raft_consensus.cc:740] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Initialized, Role: FOLLOWER
I20260709 13:07:09.172827 24950 consensus_queue.cc:260] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.173725 24952 consensus_queue.cc:260] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.174558 24966 raft_consensus.cc:493] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:09.174814 24950 ts_tablet_manager.cc:1434] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:07:09.175000 24966 raft_consensus.cc:515] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.175693 24950 tablet_bootstrap.cc:492] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f: Bootstrap starting.
I20260709 13:07:09.176932 24966 leader_election.cc:290] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:09.180074 24759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22c9ddb498a04387bd6e576b0cce53b8" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94" is_pre_election: true
I20260709 13:07:09.180820 24759 raft_consensus.cc:2393] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in current term 1: Already voted for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in this term.
I20260709 13:07:09.181512 24950 tablet_bootstrap.cc:654] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:09.181447 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22c9ddb498a04387bd6e576b0cce53b8" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" is_pre_election: true
I20260709 13:07:09.183797 24949 tablet_bootstrap.cc:492] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94: No bootstrap required, opened a new log
I20260709 13:07:09.184305 24949 ts_tablet_manager.cc:1403] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94: Time spent bootstrapping tablet: real 0.036s	user 0.014s	sys 0.003s
I20260709 13:07:09.183909 24874 leader_election.cc:304] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4; no voters: 04c981c15b3b481293007baaeeca8a94, 9a1e5371bf904722bfabfdb1c97a5f7f
I20260709 13:07:09.185328 24966 raft_consensus.cc:3060] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:09.186088 24952 ts_tablet_manager.cc:1434] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20260709 13:07:09.191946 24966 raft_consensus.cc:2749] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:09.194031 24966 raft_consensus.cc:493] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:09.194469 24966 raft_consensus.cc:515] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.196437 24949 raft_consensus.cc:359] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.197256 24759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4425bffab9a34e1aa5b95565fe6808f4" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94" is_pre_election: true
I20260709 13:07:09.197417 24949 raft_consensus.cc:385] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:09.197928 24949 raft_consensus.cc:740] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Initialized, Role: FOLLOWER
W20260709 13:07:09.198659 24874 leader_election.cc:343] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:09.199098 24949 consensus_queue.cc:260] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.199748 24966 leader_election.cc:290] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:09.200836 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4425bffab9a34e1aa5b95565fe6808f4" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" is_pre_election: true
I20260709 13:07:09.201465 24835 raft_consensus.cc:2468] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 0.
I20260709 13:07:09.202193 24949 ts_tablet_manager.cc:1434] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 13:07:09.202622 24872 leader_election.cc:304] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 04c981c15b3b481293007baaeeca8a94
I20260709 13:07:09.203421 24966 raft_consensus.cc:2804] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:09.203588 24949 tablet_bootstrap.cc:492] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94: Bootstrap starting.
I20260709 13:07:09.203820 24966 raft_consensus.cc:493] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:09.204288 24966 raft_consensus.cc:3060] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:09.210978 24966 raft_consensus.cc:515] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.212083 24962 raft_consensus.cc:493] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:09.212884 24949 tablet_bootstrap.cc:654] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:09.213701 24759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4425bffab9a34e1aa5b95565fe6808f4" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:09.214154 24950 tablet_bootstrap.cc:492] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f: No bootstrap required, opened a new log
I20260709 13:07:09.214636 24950 ts_tablet_manager.cc:1403] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent bootstrapping tablet: real 0.039s	user 0.026s	sys 0.012s
W20260709 13:07:09.215068 24874 leader_election.cc:343] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:09.212585 24962 raft_consensus.cc:515] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.216837 24966 leader_election.cc:290] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:09.218187 24962 leader_election.cc:290] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:09.219286 24834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e31dbaadd96443bcafd21ec6a5cd19c4" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" is_pre_election: true
I20260709 13:07:09.219623 24908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e31dbaadd96443bcafd21ec6a5cd19c4" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" is_pre_election: true
I20260709 13:07:09.220042 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4425bffab9a34e1aa5b95565fe6808f4" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:09.220217 24908 raft_consensus.cc:2468] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 0.
I20260709 13:07:09.220587 24835 raft_consensus.cc:3060] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:09.221171 24723 leader_election.cc:343] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:09.222090 24723 leader_election.cc:304] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 1b0ad47f96d64c6f96c0f303d8b68fd4; no voters: 9a1e5371bf904722bfabfdb1c97a5f7f
I20260709 13:07:09.222577 24950 raft_consensus.cc:359] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.223083 24962 raft_consensus.cc:2804] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:09.223254 24950 raft_consensus.cc:385] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:09.223588 24962 raft_consensus.cc:493] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:09.223613 24950 raft_consensus.cc:740] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Initialized, Role: FOLLOWER
I20260709 13:07:09.223976 24949 tablet_bootstrap.cc:492] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94: No bootstrap required, opened a new log
I20260709 13:07:09.224092 24962 raft_consensus.cc:3060] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:09.224463 24949 ts_tablet_manager.cc:1403] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94: Time spent bootstrapping tablet: real 0.021s	user 0.018s	sys 0.000s
I20260709 13:07:09.224489 24950 consensus_queue.cc:260] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.226653 24950 ts_tablet_manager.cc:1434] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20260709 13:07:09.227154 24949 raft_consensus.cc:359] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.227778 24949 raft_consensus.cc:385] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:09.228022 24949 raft_consensus.cc:740] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Initialized, Role: FOLLOWER
I20260709 13:07:09.229427 24835 raft_consensus.cc:2468] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 1.
I20260709 13:07:09.229332 24949 consensus_queue.cc:260] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.230611 24872 leader_election.cc:304] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 04c981c15b3b481293007baaeeca8a94
I20260709 13:07:09.231642 24966 raft_consensus.cc:2804] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:09.231796 24949 ts_tablet_manager.cc:1434] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:07:09.232108 24966 raft_consensus.cc:697] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Running, Role: LEADER
I20260709 13:07:09.232848 24949 tablet_bootstrap.cc:492] T fab2dc5fe6b24dd180b77f547c8471c5 P 04c981c15b3b481293007baaeeca8a94: Bootstrap starting.
I20260709 13:07:09.232903 24966 consensus_queue.cc:237] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.241226 24962 raft_consensus.cc:515] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.242467 24949 tablet_bootstrap.cc:654] T fab2dc5fe6b24dd180b77f547c8471c5 P 04c981c15b3b481293007baaeeca8a94: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:09.242899 24655 catalog_manager.cc:5697] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:09.243721 24834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e31dbaadd96443bcafd21ec6a5cd19c4" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:09.245174 24834 raft_consensus.cc:3060] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:09.248461 24949 tablet_bootstrap.cc:492] T fab2dc5fe6b24dd180b77f547c8471c5 P 04c981c15b3b481293007baaeeca8a94: No bootstrap required, opened a new log
I20260709 13:07:09.248878 24949 ts_tablet_manager.cc:1403] T fab2dc5fe6b24dd180b77f547c8471c5 P 04c981c15b3b481293007baaeeca8a94: Time spent bootstrapping tablet: real 0.016s	user 0.009s	sys 0.000s
I20260709 13:07:09.249325 24962 leader_election.cc:290] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 1 election: Requested vote from peers 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:09.250409 24908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e31dbaadd96443bcafd21ec6a5cd19c4" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:09.250964 24908 raft_consensus.cc:3060] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:09.251396 24949 raft_consensus.cc:359] T fab2dc5fe6b24dd180b77f547c8471c5 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.252049 24949 raft_consensus.cc:385] T fab2dc5fe6b24dd180b77f547c8471c5 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:09.251912 24834 raft_consensus.cc:2468] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 1.
I20260709 13:07:09.252434 24949 raft_consensus.cc:740] T fab2dc5fe6b24dd180b77f547c8471c5 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Initialized, Role: FOLLOWER
I20260709 13:07:09.253269 24949 consensus_queue.cc:260] T fab2dc5fe6b24dd180b77f547c8471c5 P 04c981c15b3b481293007baaeeca8a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.253590 24723 leader_election.cc:304] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:09.254581 24998 raft_consensus.cc:2804] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:09.255000 24998 raft_consensus.cc:697] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 1 LEADER]: Becoming Leader. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Running, Role: LEADER
I20260709 13:07:09.255205 24949 ts_tablet_manager.cc:1434] T fab2dc5fe6b24dd180b77f547c8471c5 P 04c981c15b3b481293007baaeeca8a94: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:07:09.255795 24998 consensus_queue.cc:237] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.257380 24908 raft_consensus.cc:2468] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 1.
I20260709 13:07:09.263710 24658 catalog_manager.cc:5697] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04c981c15b3b481293007baaeeca8a94 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "04c981c15b3b481293007baaeeca8a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:09.269896 24966 raft_consensus.cc:493] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:09.270397 24966 raft_consensus.cc:515] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.271971 24966 leader_election.cc:290] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:09.272733 24759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94" is_pre_election: true
I20260709 13:07:09.272919 24834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" is_pre_election: true
I20260709 13:07:09.273366 24759 raft_consensus.cc:2468] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 0.
I20260709 13:07:09.273478 24834 raft_consensus.cc:2468] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 0.
I20260709 13:07:09.274420 24872 leader_election.cc:304] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:09.274629 24958 raft_consensus.cc:493] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:09.275111 24966 raft_consensus.cc:2804] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:09.275081 24958 raft_consensus.cc:515] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.275522 24966 raft_consensus.cc:493] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:09.275965 24966 raft_consensus.cc:3060] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:09.277230 24958 leader_election.cc:290] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:09.277933 24759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fab2dc5fe6b24dd180b77f547c8471c5" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94" is_pre_election: true
I20260709 13:07:09.278555 24759 raft_consensus.cc:2468] T fab2dc5fe6b24dd180b77f547c8471c5 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 0.
I20260709 13:07:09.278692 24908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fab2dc5fe6b24dd180b77f547c8471c5" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" is_pre_election: true
I20260709 13:07:09.279289 24908 raft_consensus.cc:2468] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 0.
I20260709 13:07:09.279462 24800 leader_election.cc:304] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:09.280208 24958 raft_consensus.cc:2804] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:09.280556 24958 raft_consensus.cc:493] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:09.280875 24958 raft_consensus.cc:3060] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:09.282904 24966 raft_consensus.cc:515] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.284911 24966 leader_election.cc:290] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:09.285617 24759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:09.286091 24759 raft_consensus.cc:3060] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:09.286968 24958 raft_consensus.cc:515] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.285944 24834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:09.287731 24834 raft_consensus.cc:3060] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:09.289486 24760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fab2dc5fe6b24dd180b77f547c8471c5" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:09.289927 24760 raft_consensus.cc:3060] T fab2dc5fe6b24dd180b77f547c8471c5 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:09.290055 24958 leader_election.cc:290] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:09.289942 24908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fab2dc5fe6b24dd180b77f547c8471c5" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:09.290730 24908 raft_consensus.cc:3060] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:09.293385 24834 raft_consensus.cc:2468] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 1.
I20260709 13:07:09.294315 24872 leader_election.cc:304] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:09.294962 24966 raft_consensus.cc:2804] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:09.295398 24966 raft_consensus.cc:697] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Running, Role: LEADER
I20260709 13:07:09.296156 24966 consensus_queue.cc:237] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.297005 24908 raft_consensus.cc:2468] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 1.
I20260709 13:07:09.298023 24798 leader_election.cc:304] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:09.298854 24759 raft_consensus.cc:2468] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 1.
I20260709 13:07:09.299350 24972 raft_consensus.cc:2804] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:09.299983 24972 raft_consensus.cc:697] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Becoming Leader. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Running, Role: LEADER
I20260709 13:07:09.300190 24760 raft_consensus.cc:2468] T fab2dc5fe6b24dd180b77f547c8471c5 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 1.
I20260709 13:07:09.300894 24972 consensus_queue.cc:237] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.302922 24958 consensus_queue.cc:1048] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:09.310132 24658 catalog_manager.cc:5697] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:09.311575 24655 catalog_manager.cc:5697] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:09.327163 24958 consensus_queue.cc:1048] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:09.334398 24966 raft_consensus.cc:493] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:09.334897 24966 raft_consensus.cc:515] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.336860 24966 leader_election.cc:290] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:09.337643 24760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b03eb9381040d4b92e89e870caa867" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94" is_pre_election: true
I20260709 13:07:09.338287 24760 raft_consensus.cc:2468] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 0.
I20260709 13:07:09.338397 24834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b03eb9381040d4b92e89e870caa867" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" is_pre_election: true
I20260709 13:07:09.339093 24834 raft_consensus.cc:2468] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 0.
I20260709 13:07:09.340104 24874 leader_election.cc:304] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 1b0ad47f96d64c6f96c0f303d8b68fd4; no voters: 
I20260709 13:07:09.341107 24966 raft_consensus.cc:2804] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:09.341466 24966 raft_consensus.cc:493] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:09.341794 24966 raft_consensus.cc:3060] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:09.345422 24958 raft_consensus.cc:493] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:09.345916 24958 raft_consensus.cc:515] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.347596 24958 leader_election.cc:290] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:09.348306 24759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40603ccb1d484c139ca4806440e17898" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94" is_pre_election: true
I20260709 13:07:09.349018 24759 raft_consensus.cc:2393] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a1e5371bf904722bfabfdb1c97a5f7f in current term 1: Already voted for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in this term.
I20260709 13:07:09.348966 24908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40603ccb1d484c139ca4806440e17898" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" is_pre_election: true
I20260709 13:07:09.350687 24798 leader_election.cc:304] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 04c981c15b3b481293007baaeeca8a94, 1b0ad47f96d64c6f96c0f303d8b68fd4
I20260709 13:07:09.350612 24966 raft_consensus.cc:515] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.351613 24958 raft_consensus.cc:3060] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:09.352644 24966 leader_election.cc:290] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:09.353368 24759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b03eb9381040d4b92e89e870caa867" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:09.353911 24759 raft_consensus.cc:3060] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:09.358094 24958 raft_consensus.cc:2749] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:09.359758 24834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b03eb9381040d4b92e89e870caa867" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:09.360164 24759 raft_consensus.cc:2468] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 1.
I20260709 13:07:09.360335 24834 raft_consensus.cc:3060] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:09.361323 24874 leader_election.cc:304] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 1b0ad47f96d64c6f96c0f303d8b68fd4; no voters: 
I20260709 13:07:09.362108 24966 raft_consensus.cc:2804] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:09.362641 24966 raft_consensus.cc:697] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Running, Role: LEADER
I20260709 13:07:09.366245 24960 consensus_queue.cc:1048] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:09.367614 24834 raft_consensus.cc:2468] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 1.
I20260709 13:07:09.367499 24966 consensus_queue.cc:237] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.372699 24966 consensus_queue.cc:1048] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:09.379302 24655 catalog_manager.cc:5697] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:09.461580 24975 consensus_queue.cc:1048] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:09.471107 24976 consensus_queue.cc:1048] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:09.525628 24759 raft_consensus.cc:3060] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:09.532612 24958 consensus_queue.cc:1048] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:09.541728 24972 consensus_queue.cc:1048] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:09.549968 25007 consensus_queue.cc:1048] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:09.559931 25007 consensus_queue.cc:1048] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:09.591976 24909 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "519702cc31404b58a9f7c00e5e012da5"
dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
mode: GRACEFUL
new_leader_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:40054
I20260709 13:07:09.592527 24909 raft_consensus.cc:606] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Received request to transfer leadership to TS 04c981c15b3b481293007baaeeca8a94
I20260709 13:07:09.604493 24834 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "22c9ddb498a04387bd6e576b0cce53b8"
dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
mode: GRACEFUL
new_leader_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:44474
I20260709 13:07:09.605037 24834 raft_consensus.cc:606] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Received request to transfer leadership to TS 04c981c15b3b481293007baaeeca8a94
I20260709 13:07:09.610072 24834 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e1196843f61e4f5a92a158a876ead792"
dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
mode: GRACEFUL
new_leader_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:44474
I20260709 13:07:09.610533 24834 raft_consensus.cc:606] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Received request to transfer leadership to TS 04c981c15b3b481293007baaeeca8a94
I20260709 13:07:09.614874 24909 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "40603ccb1d484c139ca4806440e17898"
dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
mode: GRACEFUL
new_leader_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:40054
I20260709 13:07:09.615278 24909 raft_consensus.cc:606] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Received request to transfer leadership to TS 04c981c15b3b481293007baaeeca8a94
I20260709 13:07:09.619411 24909 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82"
dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
mode: GRACEFUL
new_leader_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:40054
I20260709 13:07:09.619804 24909 raft_consensus.cc:606] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Received request to transfer leadership to TS 04c981c15b3b481293007baaeeca8a94
I20260709 13:07:09.631274 24759 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e31dbaadd96443bcafd21ec6a5cd19c4"
dest_uuid: "04c981c15b3b481293007baaeeca8a94"
mode: GRACEFUL
new_leader_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
 from {username='slave'} at 127.0.0.1:49364
I20260709 13:07:09.631757 24759 raft_consensus.cc:606] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 1 LEADER]: Received request to transfer leadership to TS 9a1e5371bf904722bfabfdb1c97a5f7f
I20260709 13:07:09.638545 24909 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "24b03eb9381040d4b92e89e870caa867"
dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
mode: GRACEFUL
new_leader_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
 from {username='slave'} at 127.0.0.1:40054
I20260709 13:07:09.638969 24909 raft_consensus.cc:606] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Received request to transfer leadership to TS 9a1e5371bf904722bfabfdb1c97a5f7f
I20260709 13:07:09.642933 24998 raft_consensus.cc:493] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:09.643036 24909 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4425bffab9a34e1aa5b95565fe6808f4"
dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
mode: GRACEFUL
new_leader_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
 from {username='slave'} at 127.0.0.1:40054
I20260709 13:07:09.643586 24909 raft_consensus.cc:606] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Received request to transfer leadership to TS 9a1e5371bf904722bfabfdb1c97a5f7f
I20260709 13:07:09.643368 24998 raft_consensus.cc:515] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.645450 24998 leader_election.cc:290] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:09.646013 24834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4425bffab9a34e1aa5b95565fe6808f4" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" is_pre_election: true
I20260709 13:07:09.646472 24909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4425bffab9a34e1aa5b95565fe6808f4" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" is_pre_election: true
I20260709 13:07:09.646736 24834 raft_consensus.cc:2393] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 04c981c15b3b481293007baaeeca8a94 in current term 1: Already voted for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in this term.
I20260709 13:07:09.647850 24723 leader_election.cc:304] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94; no voters: 1b0ad47f96d64c6f96c0f303d8b68fd4, 9a1e5371bf904722bfabfdb1c97a5f7f
I20260709 13:07:09.648502 24834 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fab2dc5fe6b24dd180b77f547c8471c5"
dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
mode: GRACEFUL
new_leader_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
 from {username='slave'} at 127.0.0.1:44474
I20260709 13:07:09.648597 24998 raft_consensus.cc:3060] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:09.648998 24834 raft_consensus.cc:606] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Received request to transfer leadership to TS 1b0ad47f96d64c6f96c0f303d8b68fd4
I20260709 13:07:09.653440 24998 raft_consensus.cc:2749] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:09.665308 24976 consensus_queue.cc:1048] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:09.677011 24975 consensus_queue.cc:1048] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:09.677325 24998 consensus_queue.cc:1048] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:09.686625 24966 raft_consensus.cc:993] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4: : Instructing follower 9a1e5371bf904722bfabfdb1c97a5f7f to start an election
I20260709 13:07:09.686699 24962 raft_consensus.cc:993] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94: : Instructing follower 9a1e5371bf904722bfabfdb1c97a5f7f to start an election
I20260709 13:07:09.687126 24966 raft_consensus.cc:1081] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Signalling peer 9a1e5371bf904722bfabfdb1c97a5f7f to start an election
I20260709 13:07:09.688828 24835 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4425bffab9a34e1aa5b95565fe6808f4"
dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
 from {username='slave'} at 127.0.0.1:44470
I20260709 13:07:09.688771 24966 consensus_queue.cc:1048] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:09.689993 24835 raft_consensus.cc:493] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:09.690364 24835 raft_consensus.cc:3060] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.696663 24998 raft_consensus.cc:1081] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 1 LEADER]: Signalling peer 9a1e5371bf904722bfabfdb1c97a5f7f to start an election
I20260709 13:07:09.698232 24833 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e31dbaadd96443bcafd21ec6a5cd19c4"
dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
 from {username='slave'} at 127.0.0.1:44466
I20260709 13:07:09.698277 24958 consensus_queue.cc:1048] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:09.698964 24833 raft_consensus.cc:493] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:09.699409 24833 raft_consensus.cc:3060] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.706770 24835 raft_consensus.cc:515] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.710049 24833 raft_consensus.cc:515] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.711689 24760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4425bffab9a34e1aa5b95565fe6808f4" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:09.712267 24760 raft_consensus.cc:3060] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.713986 24958 consensus_queue.cc:1048] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:09.718063 24835 leader_election.cc:290] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 2 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:09.719416 24759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e31dbaadd96443bcafd21ec6a5cd19c4" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:09.719992 24759 raft_consensus.cc:3055] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:09.719923 24760 raft_consensus.cc:2468] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 2.
I20260709 13:07:09.720379 24759 raft_consensus.cc:740] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Running, Role: LEADER
I20260709 13:07:09.720781 24907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4425bffab9a34e1aa5b95565fe6808f4" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:09.721419 24907 raft_consensus.cc:3055] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:09.721791 24907 raft_consensus.cc:740] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Running, Role: LEADER
I20260709 13:07:09.721333 24759 consensus_queue.cc:260] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.722635 24907 consensus_queue.cc:260] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.724772 24907 raft_consensus.cc:3060] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.722496 24759 raft_consensus.cc:3060] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.727522 24800 leader_election.cc:304] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:09.729466 24958 raft_consensus.cc:2804] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:09.729938 24958 raft_consensus.cc:697] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 LEADER]: Becoming Leader. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Running, Role: LEADER
I20260709 13:07:09.730207 24908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e31dbaadd96443bcafd21ec6a5cd19c4" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:09.730726 24908 raft_consensus.cc:3060] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.730805 24958 consensus_queue.cc:237] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.733908 24976 consensus_queue.cc:1048] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:09.737428 24908 raft_consensus.cc:2468] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 2.
I20260709 13:07:09.738744 24798 leader_election.cc:304] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:09.739594 24958 raft_consensus.cc:2804] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:09.740223 24958 raft_consensus.cc:697] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 LEADER]: Becoming Leader. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Running, Role: LEADER
I20260709 13:07:09.741027 24958 consensus_queue.cc:237] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.746417 24907 raft_consensus.cc:2468] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 2.
I20260709 13:07:09.747329 24658 catalog_manager.cc:5697] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f reported cstate change: term changed from 1 to 2, leader changed from 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67) to 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66). New cstate: current_term: 2 leader_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:09.727876 24833 leader_election.cc:290] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 2 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:09.753612 24972 raft_consensus.cc:993] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f: : Instructing follower 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:09.754245 24972 raft_consensus.cc:1081] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Signalling peer 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:09.754436 24958 raft_consensus.cc:993] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f: : Instructing follower 1b0ad47f96d64c6f96c0f303d8b68fd4 to start an election
I20260709 13:07:09.755260 24972 raft_consensus.cc:1081] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Signalling peer 1b0ad47f96d64c6f96c0f303d8b68fd4 to start an election
I20260709 13:07:09.758248 24758 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e1196843f61e4f5a92a158a876ead792"
dest_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:49346
I20260709 13:07:09.758934 24758 raft_consensus.cc:493] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:09.759392 24758 raft_consensus.cc:3060] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.776328 24654 catalog_manager.cc:5697] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f reported cstate change: term changed from 1 to 2, leader changed from 04c981c15b3b481293007baaeeca8a94 (127.23.140.65) to 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66). New cstate: current_term: 2 leader_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:09.779675 24759 raft_consensus.cc:2468] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 2.
I20260709 13:07:09.787159 24976 raft_consensus.cc:993] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4: : Instructing follower 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:09.787634 24960 raft_consensus.cc:1081] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Signalling peer 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:09.788317 24758 raft_consensus.cc:515] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.790431 24758 leader_election.cc:290] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 2 election: Requested vote from peers 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:09.791026 24834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1196843f61e4f5a92a158a876ead792" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:09.792081 24759 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82"
dest_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:49356
I20260709 13:07:09.792495 24834 raft_consensus.cc:3055] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:09.792659 24759 raft_consensus.cc:493] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:09.792912 24834 raft_consensus.cc:740] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Running, Role: LEADER
I20260709 13:07:09.793107 24759 raft_consensus.cc:3060] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.791561 24909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1196843f61e4f5a92a158a876ead792" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:09.793799 24909 raft_consensus.cc:3060] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.793779 24834 consensus_queue.cc:260] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.794613 24960 consensus_queue.cc:1048] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:09.795171 24834 raft_consensus.cc:3060] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.799849 24907 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fab2dc5fe6b24dd180b77f547c8471c5"
dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
 from {username='slave'} at 127.0.0.1:40046
I20260709 13:07:09.800403 24907 raft_consensus.cc:493] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:09.800768 24907 raft_consensus.cc:3060] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.801965 24834 raft_consensus.cc:2468] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 2.
I20260709 13:07:09.803104 24723 leader_election.cc:304] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:09.803906 24962 raft_consensus.cc:2804] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:09.804330 24962 raft_consensus.cc:697] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Becoming Leader. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Running, Role: LEADER
I20260709 13:07:09.805151 24962 consensus_queue.cc:237] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.810338 24909 raft_consensus.cc:2468] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 2.
I20260709 13:07:09.813280 24976 raft_consensus.cc:993] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4: : Instructing follower 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:09.813687 24966 raft_consensus.cc:1081] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Signalling peer 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:09.814775 24654 catalog_manager.cc:5697] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 reported cstate change: term changed from 1 to 2, leader changed from 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66) to 04c981c15b3b481293007baaeeca8a94 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "04c981c15b3b481293007baaeeca8a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:09.815434 24758 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "40603ccb1d484c139ca4806440e17898"
dest_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:49356
I20260709 13:07:09.815968 24758 raft_consensus.cc:493] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:09.816327 24758 raft_consensus.cc:3060] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.817277 24907 raft_consensus.cc:515] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.817577 24976 consensus_queue.cc:1048] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:09.819319 24907 leader_election.cc:290] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 2 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:09.820969 24833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fab2dc5fe6b24dd180b77f547c8471c5" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:09.821549 24833 raft_consensus.cc:3055] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:09.821856 24833 raft_consensus.cc:740] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Running, Role: LEADER
I20260709 13:07:09.822592 24833 consensus_queue.cc:260] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.822721 24759 raft_consensus.cc:515] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.823658 24833 raft_consensus.cc:3060] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.825227 24759 leader_election.cc:290] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 2 election: Requested vote from peers 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:09.826532 24758 raft_consensus.cc:515] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.827054 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:09.827649 24835 raft_consensus.cc:3060] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.829682 24758 leader_election.cc:290] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 2 election: Requested vote from peers 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:09.829842 24832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40603ccb1d484c139ca4806440e17898" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:09.830478 24832 raft_consensus.cc:3060] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.833954 24835 raft_consensus.cc:2468] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 2.
I20260709 13:07:09.835631 24723 leader_election.cc:304] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:09.836479 24962 raft_consensus.cc:2804] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:09.835760 24757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fab2dc5fe6b24dd180b77f547c8471c5" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:09.837148 24962 raft_consensus.cc:697] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Becoming Leader. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Running, Role: LEADER
I20260709 13:07:09.839013 24909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40603ccb1d484c139ca4806440e17898" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:09.839561 24909 raft_consensus.cc:3055] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:09.839447 24962 consensus_queue.cc:237] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.839960 24909 raft_consensus.cc:740] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Running, Role: LEADER
I20260709 13:07:09.839619 24757 raft_consensus.cc:3060] T fab2dc5fe6b24dd180b77f547c8471c5 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.837628 24907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:09.840756 24909 consensus_queue.cc:260] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.845435 24907 raft_consensus.cc:3055] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:09.845773 24907 raft_consensus.cc:740] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Running, Role: LEADER
I20260709 13:07:09.845836 24909 raft_consensus.cc:3060] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.846652 24907 consensus_queue.cc:260] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.846853 24833 raft_consensus.cc:2468] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 2.
I20260709 13:07:09.847823 24907 raft_consensus.cc:3060] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.848552 24872 leader_election.cc:304] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:09.849756 24976 raft_consensus.cc:2804] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:09.851343 24976 raft_consensus.cc:697] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 LEADER]: Becoming Leader. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Running, Role: LEADER
I20260709 13:07:09.852348 24909 raft_consensus.cc:2468] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 2.
I20260709 13:07:09.852522 24976 consensus_queue.cc:237] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.853530 24723 leader_election.cc:304] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 1b0ad47f96d64c6f96c0f303d8b68fd4; no voters: 
I20260709 13:07:09.856034 24654 catalog_manager.cc:5697] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 reported cstate change: term changed from 1 to 2, leader changed from 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67) to 04c981c15b3b481293007baaeeca8a94 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "04c981c15b3b481293007baaeeca8a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:09.837460 24832 raft_consensus.cc:2468] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 2.
I20260709 13:07:09.857312 24962 raft_consensus.cc:2804] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:09.859669 24962 raft_consensus.cc:697] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Becoming Leader. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Running, Role: LEADER
I20260709 13:07:09.861855 24907 raft_consensus.cc:2468] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 2.
I20260709 13:07:09.863369 24757 raft_consensus.cc:2468] T fab2dc5fe6b24dd180b77f547c8471c5 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 2.
I20260709 13:07:09.863839 24962 consensus_queue.cc:237] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.865743 24655 catalog_manager.cc:5697] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 reported cstate change: term changed from 1 to 2, leader changed from 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66) to 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67). New cstate: current_term: 2 leader_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:09.869303 24975 raft_consensus.cc:993] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4: : Instructing follower 9a1e5371bf904722bfabfdb1c97a5f7f to start an election
I20260709 13:07:09.869716 24976 raft_consensus.cc:1081] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Signalling peer 9a1e5371bf904722bfabfdb1c97a5f7f to start an election
I20260709 13:07:09.871253 24834 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "24b03eb9381040d4b92e89e870caa867"
dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
 from {username='slave'} at 127.0.0.1:44470
I20260709 13:07:09.871793 24834 raft_consensus.cc:493] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:09.872148 24834 raft_consensus.cc:3060] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.879202 24834 raft_consensus.cc:515] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.881348 24975 raft_consensus.cc:993] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4: : Instructing follower 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:09.881779 24975 raft_consensus.cc:1081] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Signalling peer 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:09.881860 24757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b03eb9381040d4b92e89e870caa867" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:09.882465 24757 raft_consensus.cc:3060] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.882617 24909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b03eb9381040d4b92e89e870caa867" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:09.883196 24909 raft_consensus.cc:3055] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:09.883502 24909 raft_consensus.cc:740] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Running, Role: LEADER
I20260709 13:07:09.883822 24758 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "519702cc31404b58a9f7c00e5e012da5"
dest_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:49356
I20260709 13:07:09.884354 24758 raft_consensus.cc:493] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:09.884862 24758 raft_consensus.cc:3060] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.880300 24655 catalog_manager.cc:5697] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 reported cstate change: term changed from 1 to 2, leader changed from 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67) to 04c981c15b3b481293007baaeeca8a94 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "04c981c15b3b481293007baaeeca8a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:09.881979 24834 leader_election.cc:290] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 2 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:09.884320 24909 consensus_queue.cc:260] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.889153 24757 raft_consensus.cc:2468] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 2.
I20260709 13:07:09.889376 24909 raft_consensus.cc:3060] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.890425 24800 leader_election.cc:304] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:09.891300 25031 raft_consensus.cc:2804] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:09.891741 25031 raft_consensus.cc:697] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 LEADER]: Becoming Leader. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Running, Role: LEADER
I20260709 13:07:09.892570 25031 consensus_queue.cc:237] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:09.895540 24909 raft_consensus.cc:2468] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 2.
I20260709 13:07:09.897087 24758 raft_consensus.cc:515] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } }
I20260709 13:07:09.898788 24758 leader_election.cc:290] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 2 election: Requested vote from peers 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:09.899852 24834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "519702cc31404b58a9f7c00e5e012da5" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:09.900000 24909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "519702cc31404b58a9f7c00e5e012da5" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:09.900321 24834 raft_consensus.cc:3060] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.900580 24909 raft_consensus.cc:3055] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:09.900923 24909 raft_consensus.cc:740] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Running, Role: LEADER
I20260709 13:07:09.900398 24655 catalog_manager.cc:5697] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f reported cstate change: term changed from 1 to 2, leader changed from 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67) to 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66). New cstate: current_term: 2 leader_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:09.901721 24909 consensus_queue.cc:260] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } }
I20260709 13:07:09.902909 24909 raft_consensus.cc:3060] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:09.905519 24834 raft_consensus.cc:2468] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 2.
I20260709 13:07:09.906634 24723 leader_election.cc:304] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:09.908572 24909 raft_consensus.cc:2468] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 2.
I20260709 13:07:09.908546 24962 raft_consensus.cc:2804] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:09.909267 24962 raft_consensus.cc:697] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Becoming Leader. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Running, Role: LEADER
I20260709 13:07:09.910085 24962 consensus_queue.cc:237] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } }
I20260709 13:07:09.915911 24655 catalog_manager.cc:5697] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 reported cstate change: term changed from 1 to 2, leader changed from 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67) to 04c981c15b3b481293007baaeeca8a94 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "04c981c15b3b481293007baaeeca8a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:10.022500 25031 raft_consensus.cc:993] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f: : Instructing follower 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:10.023027 25007 raft_consensus.cc:1081] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Signalling peer 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:10.024688 24758 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "22c9ddb498a04387bd6e576b0cce53b8"
dest_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:49346
I20260709 13:07:10.025256 24758 raft_consensus.cc:493] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:10.025605 24758 raft_consensus.cc:3060] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:10.030066 24758 raft_consensus.cc:515] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:10.031558 24758 leader_election.cc:290] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 2 election: Requested vote from peers 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:10.032362 24834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22c9ddb498a04387bd6e576b0cce53b8" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:10.032536 24909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22c9ddb498a04387bd6e576b0cce53b8" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:10.032948 24834 raft_consensus.cc:3055] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:10.033017 24909 raft_consensus.cc:3060] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:10.033388 24834 raft_consensus.cc:740] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Running, Role: LEADER
I20260709 13:07:10.034199 24834 consensus_queue.cc:260] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:10.035347 24834 raft_consensus.cc:3060] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:10.039592 24909 raft_consensus.cc:2468] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 2.
I20260709 13:07:10.040652 24723 leader_election.cc:304] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 1b0ad47f96d64c6f96c0f303d8b68fd4; no voters: 
I20260709 13:07:10.041347 24998 raft_consensus.cc:2804] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:10.041508 24834 raft_consensus.cc:2468] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 2.
I20260709 13:07:10.041786 24998 raft_consensus.cc:697] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Becoming Leader. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Running, Role: LEADER
I20260709 13:07:10.042590 24998 consensus_queue.cc:237] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:10.047991 24655 catalog_manager.cc:5697] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 reported cstate change: term changed from 1 to 2, leader changed from 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66) to 04c981c15b3b481293007baaeeca8a94 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "04c981c15b3b481293007baaeeca8a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:10.193457 24758 raft_consensus.cc:1275] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Refusing update from remote peer 9a1e5371bf904722bfabfdb1c97a5f7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:10.194630 24958 consensus_queue.cc:1048] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:10.203241 24909 raft_consensus.cc:1275] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Refusing update from remote peer 9a1e5371bf904722bfabfdb1c97a5f7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:10.204664 24958 consensus_queue.cc:1048] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:10.210382 24907 raft_consensus.cc:1275] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Refusing update from remote peer 9a1e5371bf904722bfabfdb1c97a5f7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:10.211642 24958 consensus_queue.cc:1048] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:10.222244 24758 raft_consensus.cc:1275] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Refusing update from remote peer 9a1e5371bf904722bfabfdb1c97a5f7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:10.223601 24972 consensus_queue.cc:1048] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:10.249078 24907 raft_consensus.cc:1275] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:10.250253 24998 consensus_queue.cc:1048] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:10.256946 24834 raft_consensus.cc:1275] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:10.258479 24962 consensus_queue.cc:1048] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:10.297310 24834 raft_consensus.cc:1275] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:10.298419 24962 consensus_queue.cc:1048] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:10.301457 24907 raft_consensus.cc:1275] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:10.302662 24962 consensus_queue.cc:1048] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:10.306257 24907 raft_consensus.cc:1275] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:10.309564 24834 raft_consensus.cc:1275] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Refusing update from remote peer 1b0ad47f96d64c6f96c0f303d8b68fd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:10.310714 25035 consensus_queue.cc:1048] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:10.313114 24966 consensus_queue.cc:1048] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:10.315412 24832 raft_consensus.cc:1275] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:10.318235 24962 consensus_queue.cc:1048] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:10.323376 24758 raft_consensus.cc:1275] T fab2dc5fe6b24dd180b77f547c8471c5 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Refusing update from remote peer 1b0ad47f96d64c6f96c0f303d8b68fd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:10.325117 24976 consensus_queue.cc:1048] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:10.445283 24832 raft_consensus.cc:1275] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:10.446563 24962 consensus_queue.cc:1048] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:10.450304 24907 raft_consensus.cc:1275] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:10.451937 24998 consensus_queue.cc:1048] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:10.485713 24907 raft_consensus.cc:1275] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Refusing update from remote peer 9a1e5371bf904722bfabfdb1c97a5f7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:10.486924 24958 consensus_queue.cc:1048] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:10.493983 24758 raft_consensus.cc:1275] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Refusing update from remote peer 9a1e5371bf904722bfabfdb1c97a5f7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:10.495527 24972 consensus_queue.cc:1048] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:10.581616 24832 raft_consensus.cc:1275] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:10.582821 24998 consensus_queue.cc:1048] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:10.591954 24907 raft_consensus.cc:1275] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:10.593595 25026 consensus_queue.cc:1048] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:12.679441 24113 auto_leader_rebalancer-test.cc:348] The leader distribution is 
I20260709 13:07:12.680122 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:12.701105 24758 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "40603ccb1d484c139ca4806440e17898"
dest_uuid: "04c981c15b3b481293007baaeeca8a94"
mode: GRACEFUL
new_leader_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
 from {username='slave'} at 127.0.0.1:49366
I20260709 13:07:12.701714 24758 raft_consensus.cc:606] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Received request to transfer leadership to TS 1b0ad47f96d64c6f96c0f303d8b68fd4
I20260709 13:07:12.704618 24758 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82"
dest_uuid: "04c981c15b3b481293007baaeeca8a94"
mode: GRACEFUL
new_leader_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
 from {username='slave'} at 127.0.0.1:49366
I20260709 13:07:12.705185 24758 raft_consensus.cc:606] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Received request to transfer leadership to TS 1b0ad47f96d64c6f96c0f303d8b68fd4
I20260709 13:07:12.706246 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:07:12.706620 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:07:13.063866 25040 raft_consensus.cc:993] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94: : Instructing follower 1b0ad47f96d64c6f96c0f303d8b68fd4 to start an election
I20260709 13:07:13.064291 25045 raft_consensus.cc:1081] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Signalling peer 1b0ad47f96d64c6f96c0f303d8b68fd4 to start an election
I20260709 13:07:13.065762 24909 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "40603ccb1d484c139ca4806440e17898"
dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
 from {username='slave'} at 127.0.0.1:40042
I20260709 13:07:13.066185 24909 raft_consensus.cc:493] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:13.066424 24909 raft_consensus.cc:3060] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:13.070811 24909 raft_consensus.cc:515] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:13.072314 24909 leader_election.cc:290] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 3 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:13.073055 24758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40603ccb1d484c139ca4806440e17898" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:13.073349 24832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40603ccb1d484c139ca4806440e17898" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:13.073690 24758 raft_consensus.cc:3055] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:13.073885 24832 raft_consensus.cc:3060] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:13.074044 24758 raft_consensus.cc:740] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Running, Role: LEADER
I20260709 13:07:13.074784 24758 consensus_queue.cc:260] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:13.075662 24758 raft_consensus.cc:3060] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:13.079047 24832 raft_consensus.cc:2468] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 3.
I20260709 13:07:13.080125 24872 leader_election.cc:304] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:13.080945 25039 raft_consensus.cc:2804] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:13.081269 24758 raft_consensus.cc:2468] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 3.
I20260709 13:07:13.082280 25039 raft_consensus.cc:697] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 LEADER]: Becoming Leader. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Running, Role: LEADER
I20260709 13:07:13.083072 25039 consensus_queue.cc:237] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:13.089485 24654 catalog_manager.cc:5697] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 reported cstate change: term changed from 2 to 3, leader changed from 04c981c15b3b481293007baaeeca8a94 (127.23.140.65) to 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:13.098073 25040 raft_consensus.cc:993] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94: : Instructing follower 1b0ad47f96d64c6f96c0f303d8b68fd4 to start an election
I20260709 13:07:13.098567 25050 raft_consensus.cc:1081] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Signalling peer 1b0ad47f96d64c6f96c0f303d8b68fd4 to start an election
I20260709 13:07:13.100297 24909 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82"
dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
 from {username='slave'} at 127.0.0.1:40042
I20260709 13:07:13.100857 24909 raft_consensus.cc:493] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:13.101226 24909 raft_consensus.cc:3060] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:13.106238 24909 raft_consensus.cc:515] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:13.107726 24909 leader_election.cc:290] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 3 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:13.108588 24758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:13.108780 24832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:13.109176 24758 raft_consensus.cc:3055] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:13.109378 24832 raft_consensus.cc:3060] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:13.109512 24758 raft_consensus.cc:740] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Running, Role: LEADER
I20260709 13:07:13.110323 24758 consensus_queue.cc:260] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:13.111208 24758 raft_consensus.cc:3060] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:13.114675 24832 raft_consensus.cc:2468] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 3.
I20260709 13:07:13.115720 24872 leader_election.cc:304] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:13.116387 25039 raft_consensus.cc:2804] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:13.116851 25039 raft_consensus.cc:697] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 LEADER]: Becoming Leader. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Running, Role: LEADER
I20260709 13:07:13.117296 24758 raft_consensus.cc:2468] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 3.
I20260709 13:07:13.117755 25039 consensus_queue.cc:237] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:13.124496 24654 catalog_manager.cc:5697] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 reported cstate change: term changed from 2 to 3, leader changed from 04c981c15b3b481293007baaeeca8a94 (127.23.140.65) to 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:13.610011 24758 raft_consensus.cc:1275] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Refusing update from remote peer 1b0ad47f96d64c6f96c0f303d8b68fd4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:13.610563 24832 raft_consensus.cc:1275] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Refusing update from remote peer 1b0ad47f96d64c6f96c0f303d8b68fd4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:13.611928 25039 consensus_queue.cc:1048] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:13.612840 25046 consensus_queue.cc:1048] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:13.635715 24834 raft_consensus.cc:1275] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Refusing update from remote peer 1b0ad47f96d64c6f96c0f303d8b68fd4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:13.637359 25057 consensus_queue.cc:1048] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:13.655864 24758 raft_consensus.cc:1275] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Refusing update from remote peer 1b0ad47f96d64c6f96c0f303d8b68fd4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:13.657629 25039 consensus_queue.cc:1048] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:13.707506 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:13.725957 24907 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "40603ccb1d484c139ca4806440e17898"
dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
mode: GRACEFUL
new_leader_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:40054
I20260709 13:07:13.726548 24907 raft_consensus.cc:606] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 LEADER]: Received request to transfer leadership to TS 04c981c15b3b481293007baaeeca8a94
I20260709 13:07:13.730933 24907 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82"
dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
mode: GRACEFUL
new_leader_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:40054
I20260709 13:07:13.731387 24907 raft_consensus.cc:606] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 LEADER]: Received request to transfer leadership to TS 04c981c15b3b481293007baaeeca8a94
I20260709 13:07:13.735637 24834 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e31dbaadd96443bcafd21ec6a5cd19c4"
dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
mode: GRACEFUL
new_leader_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:44474
I20260709 13:07:13.736104 24834 raft_consensus.cc:606] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 LEADER]: Received request to transfer leadership to TS 04c981c15b3b481293007baaeeca8a94
I20260709 13:07:13.740330 24834 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e5e8e388810f4364b62faefb3c1ec45f"
dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
mode: GRACEFUL
new_leader_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:44474
I20260709 13:07:13.740788 24834 raft_consensus.cc:606] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Received request to transfer leadership to TS 04c981c15b3b481293007baaeeca8a94
I20260709 13:07:13.744910 24834 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "24b03eb9381040d4b92e89e870caa867"
dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
mode: GRACEFUL
new_leader_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:44474
I20260709 13:07:13.745374 24834 raft_consensus.cc:606] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 LEADER]: Received request to transfer leadership to TS 04c981c15b3b481293007baaeeca8a94
I20260709 13:07:13.958148 25041 raft_consensus.cc:993] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f: : Instructing follower 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:13.958788 25064 raft_consensus.cc:1081] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 LEADER]: Signalling peer 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:13.960317 24758 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e31dbaadd96443bcafd21ec6a5cd19c4"
dest_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:49346
I20260709 13:07:13.960816 24758 raft_consensus.cc:493] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:13.961149 24758 raft_consensus.cc:3060] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:13.966269 24758 raft_consensus.cc:515] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:13.967928 24758 leader_election.cc:290] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 3 election: Requested vote from peers 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:13.968791 24834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e31dbaadd96443bcafd21ec6a5cd19c4" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:13.969164 24907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e31dbaadd96443bcafd21ec6a5cd19c4" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:13.969465 24834 raft_consensus.cc:3055] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:13.969744 24907 raft_consensus.cc:3060] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:13.969795 24834 raft_consensus.cc:740] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Running, Role: LEADER
I20260709 13:07:13.970697 24834 consensus_queue.cc:260] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:13.971647 24834 raft_consensus.cc:3060] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:13.975495 24907 raft_consensus.cc:2468] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 3.
I20260709 13:07:13.976627 24834 raft_consensus.cc:2468] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 3.
I20260709 13:07:13.976796 24723 leader_election.cc:304] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 1b0ad47f96d64c6f96c0f303d8b68fd4; no voters: 
I20260709 13:07:13.977813 25050 raft_consensus.cc:2804] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:13.978901 25050 raft_consensus.cc:697] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 3 LEADER]: Becoming Leader. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Running, Role: LEADER
I20260709 13:07:13.979720 25050 consensus_queue.cc:237] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:13.986774 24655 catalog_manager.cc:5697] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 reported cstate change: term changed from 2 to 3, leader changed from 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66) to 04c981c15b3b481293007baaeeca8a94 (127.23.140.65). New cstate: current_term: 3 leader_uuid: "04c981c15b3b481293007baaeeca8a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:14.043514 25057 raft_consensus.cc:993] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4: : Instructing follower 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:14.043990 25046 raft_consensus.cc:1081] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 LEADER]: Signalling peer 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:14.045306 24758 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "40603ccb1d484c139ca4806440e17898"
dest_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:49356
I20260709 13:07:14.045881 24758 raft_consensus.cc:493] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:14.046257 24758 raft_consensus.cc:3060] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:14.051234 24758 raft_consensus.cc:515] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:14.052713 24758 leader_election.cc:290] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 4 election: Requested vote from peers 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:14.053529 24834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40603ccb1d484c139ca4806440e17898" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:14.053745 24907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40603ccb1d484c139ca4806440e17898" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:14.054095 24834 raft_consensus.cc:3060] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:14.054255 24907 raft_consensus.cc:3055] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:07:14.054610 24907 raft_consensus.cc:740] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Running, Role: LEADER
I20260709 13:07:14.055279 24907 consensus_queue.cc:260] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:14.056165 24907 raft_consensus.cc:3060] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:14.058789 24834 raft_consensus.cc:2468] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 4.
I20260709 13:07:14.059763 24723 leader_election.cc:304] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:14.060428 25050 raft_consensus.cc:2804] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:07:14.060843 25050 raft_consensus.cc:697] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 4 LEADER]: Becoming Leader. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Running, Role: LEADER
I20260709 13:07:14.061949 24907 raft_consensus.cc:2468] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 4.
I20260709 13:07:14.061776 25050 consensus_queue.cc:237] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:14.068176 24655 catalog_manager.cc:5697] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 reported cstate change: term changed from 3 to 4, leader changed from 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67) to 04c981c15b3b481293007baaeeca8a94 (127.23.140.65). New cstate: current_term: 4 leader_uuid: "04c981c15b3b481293007baaeeca8a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:14.155939 25057 raft_consensus.cc:993] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4: : Instructing follower 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:14.156427 25046 raft_consensus.cc:1081] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 LEADER]: Signalling peer 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:14.158427 24758 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82"
dest_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:49356
I20260709 13:07:14.159240 24758 raft_consensus.cc:493] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:14.159754 24758 raft_consensus.cc:3060] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:14.164088 24758 raft_consensus.cc:515] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:14.165525 24758 leader_election.cc:290] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 4 election: Requested vote from peers 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:14.166317 24834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:14.166566 24907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:14.166806 24834 raft_consensus.cc:3060] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:14.167181 24907 raft_consensus.cc:3055] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:07:14.167593 24907 raft_consensus.cc:740] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Running, Role: LEADER
I20260709 13:07:14.168399 24907 consensus_queue.cc:260] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:14.169906 24907 raft_consensus.cc:3060] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:14.172958 24834 raft_consensus.cc:2468] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 4.
I20260709 13:07:14.174050 24723 leader_election.cc:304] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:14.174721 25050 raft_consensus.cc:2804] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:07:14.175078 24907 raft_consensus.cc:2468] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 4.
I20260709 13:07:14.175233 25050 raft_consensus.cc:697] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 4 LEADER]: Becoming Leader. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Running, Role: LEADER
I20260709 13:07:14.176086 25050 consensus_queue.cc:237] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:14.181515 24655 catalog_manager.cc:5697] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 reported cstate change: term changed from 3 to 4, leader changed from 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67) to 04c981c15b3b481293007baaeeca8a94 (127.23.140.65). New cstate: current_term: 4 leader_uuid: "04c981c15b3b481293007baaeeca8a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:14.228005 25041 raft_consensus.cc:993] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f: : Instructing follower 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:14.228539 25064 raft_consensus.cc:1081] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 LEADER]: Signalling peer 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:14.229846 25064 raft_consensus.cc:993] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f: : Instructing follower 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:14.230075 24758 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "24b03eb9381040d4b92e89e870caa867"
dest_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:49346
I20260709 13:07:14.230290 25041 raft_consensus.cc:1081] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Signalling peer 04c981c15b3b481293007baaeeca8a94 to start an election
I20260709 13:07:14.230633 24758 raft_consensus.cc:493] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:14.231000 24758 raft_consensus.cc:3060] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:14.231926 24757 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e5e8e388810f4364b62faefb3c1ec45f"
dest_uuid: "04c981c15b3b481293007baaeeca8a94"
 from {username='slave'} at 127.0.0.1:49346
I20260709 13:07:14.232393 24757 raft_consensus.cc:493] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:14.232685 24757 raft_consensus.cc:3060] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:14.235245 24758 raft_consensus.cc:515] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:14.236768 24758 leader_election.cc:290] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 3 election: Requested vote from peers 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:14.237852 24834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b03eb9381040d4b92e89e870caa867" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:14.237891 24757 raft_consensus.cc:515] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:14.238550 24834 raft_consensus.cc:3055] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:14.237926 24907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b03eb9381040d4b92e89e870caa867" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:14.239100 24834 raft_consensus.cc:740] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Running, Role: LEADER
I20260709 13:07:14.239394 24907 raft_consensus.cc:3060] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:14.240258 24757 leader_election.cc:290] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 2 election: Requested vote from peers 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:14.240947 24832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e8e388810f4364b62faefb3c1ec45f" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:14.240819 24834 consensus_queue.cc:260] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:14.241555 24832 raft_consensus.cc:3055] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:14.241814 24832 raft_consensus.cc:740] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Running, Role: LEADER
I20260709 13:07:14.242142 24834 raft_consensus.cc:3060] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:14.242434 24832 consensus_queue.cc:260] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:14.243268 24832 raft_consensus.cc:3060] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:14.245067 24909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e8e388810f4364b62faefb3c1ec45f" candidate_uuid: "04c981c15b3b481293007baaeeca8a94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:14.245714 24909 raft_consensus.cc:3060] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:14.246328 24907 raft_consensus.cc:2468] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 3.
I20260709 13:07:14.247601 24723 leader_election.cc:304] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 1b0ad47f96d64c6f96c0f303d8b68fd4; no voters: 
I20260709 13:07:14.248667 25065 raft_consensus.cc:2804] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:14.251394 24834 raft_consensus.cc:2468] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 3.
I20260709 13:07:14.251529 24832 raft_consensus.cc:2468] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 2.
I20260709 13:07:14.252736 25065 raft_consensus.cc:697] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 3 LEADER]: Becoming Leader. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Running, Role: LEADER
I20260709 13:07:14.253156 24909 raft_consensus.cc:2468] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c981c15b3b481293007baaeeca8a94 in term 2.
I20260709 13:07:14.253227 24723 leader_election.cc:304] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c981c15b3b481293007baaeeca8a94, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:14.254151 25050 raft_consensus.cc:2804] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:14.253690 25065 consensus_queue.cc:237] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:14.254621 25050 raft_consensus.cc:697] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Becoming Leader. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Running, Role: LEADER
I20260709 13:07:14.255544 25050 consensus_queue.cc:237] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:14.261698 24655 catalog_manager.cc:5697] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 reported cstate change: term changed from 2 to 3, leader changed from 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66) to 04c981c15b3b481293007baaeeca8a94 (127.23.140.65). New cstate: current_term: 3 leader_uuid: "04c981c15b3b481293007baaeeca8a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:14.271876 24655 catalog_manager.cc:5697] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94 reported cstate change: term changed from 1 to 2, leader changed from 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66) to 04c981c15b3b481293007baaeeca8a94 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "04c981c15b3b481293007baaeeca8a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:14.462496 24909 raft_consensus.cc:1275] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:14.463742 25050 consensus_queue.cc:1048] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:14.473744 24832 raft_consensus.cc:1275] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:14.475468 25050 consensus_queue.cc:1048] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:14.637009 24909 raft_consensus.cc:1275] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 4 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:07:14.638430 25050 consensus_queue.cc:1048] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:14.642874 24907 raft_consensus.cc:1275] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:14.644313 25050 consensus_queue.cc:1048] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:14.645787 24909 raft_consensus.cc:1275] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 4 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:07:14.648818 25080 consensus_queue.cc:1048] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:07:14.649713 24834 raft_consensus.cc:1275] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 4 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:07:14.648769 24832 raft_consensus.cc:1275] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 4 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:07:14.653817 25071 consensus_queue.cc:1048] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 13:07:14.653919 25065 consensus_queue.cc:1048] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 13:07:14.679270 24907 raft_consensus.cc:1275] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:14.687836 24835 raft_consensus.cc:1275] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:14.692523 25083 consensus_queue.cc:1048] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:14.694903 25080 consensus_queue.cc:1048] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:14.723556 24832 raft_consensus.cc:1275] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Refusing update from remote peer 04c981c15b3b481293007baaeeca8a94: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:14.725649 25082 consensus_queue.cc:1048] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:16.751762 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:16.759039 24757 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "24b03eb9381040d4b92e89e870caa867"
dest_uuid: "04c981c15b3b481293007baaeeca8a94"
mode: GRACEFUL
new_leader_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
 from {username='slave'} at 127.0.0.1:49366
I20260709 13:07:16.759552 24757 raft_consensus.cc:606] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 3 LEADER]: Received request to transfer leadership to TS 1b0ad47f96d64c6f96c0f303d8b68fd4
I20260709 13:07:16.761785 24757 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "519702cc31404b58a9f7c00e5e012da5"
dest_uuid: "04c981c15b3b481293007baaeeca8a94"
mode: GRACEFUL
new_leader_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
 from {username='slave'} at 127.0.0.1:49366
I20260709 13:07:16.762215 24757 raft_consensus.cc:606] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Received request to transfer leadership to TS 1b0ad47f96d64c6f96c0f303d8b68fd4
I20260709 13:07:16.764084 24757 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82"
dest_uuid: "04c981c15b3b481293007baaeeca8a94"
mode: GRACEFUL
new_leader_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
 from {username='slave'} at 127.0.0.1:49366
I20260709 13:07:16.764500 24757 raft_consensus.cc:606] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 4 LEADER]: Received request to transfer leadership to TS 9a1e5371bf904722bfabfdb1c97a5f7f
I20260709 13:07:16.766407 24757 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e1196843f61e4f5a92a158a876ead792"
dest_uuid: "04c981c15b3b481293007baaeeca8a94"
mode: GRACEFUL
new_leader_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
 from {username='slave'} at 127.0.0.1:49366
I20260709 13:07:16.766836 24757 raft_consensus.cc:606] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Received request to transfer leadership to TS 9a1e5371bf904722bfabfdb1c97a5f7f
I20260709 13:07:16.767828 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 13:07:16.768256 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:07:16.953212 25090 raft_consensus.cc:993] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94: : Instructing follower 1b0ad47f96d64c6f96c0f303d8b68fd4 to start an election
I20260709 13:07:16.953648 25090 raft_consensus.cc:1081] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 3 LEADER]: Signalling peer 1b0ad47f96d64c6f96c0f303d8b68fd4 to start an election
I20260709 13:07:16.954982 24908 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "24b03eb9381040d4b92e89e870caa867"
dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
 from {username='slave'} at 127.0.0.1:40042
I20260709 13:07:16.955382 24908 raft_consensus.cc:493] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:16.955612 24908 raft_consensus.cc:3060] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:16.960433 24908 raft_consensus.cc:515] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:16.962090 24908 leader_election.cc:290] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 4 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:16.962966 24757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b03eb9381040d4b92e89e870caa867" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:16.963124 24832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b03eb9381040d4b92e89e870caa867" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:16.963528 24757 raft_consensus.cc:3055] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:07:16.963660 24832 raft_consensus.cc:3060] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:16.963860 24757 raft_consensus.cc:740] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Running, Role: LEADER
I20260709 13:07:16.964607 24757 consensus_queue.cc:260] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:16.965538 24757 raft_consensus.cc:3060] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:16.967972 24832 raft_consensus.cc:2468] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 4.
I20260709 13:07:16.969014 24872 leader_election.cc:304] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:16.969731 25046 raft_consensus.cc:2804] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:07:16.970281 24757 raft_consensus.cc:2468] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 4.
I20260709 13:07:16.970619 25046 raft_consensus.cc:697] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 4 LEADER]: Becoming Leader. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Running, Role: LEADER
I20260709 13:07:16.971525 25046 consensus_queue.cc:237] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:16.977547 24658 catalog_manager.cc:5697] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 reported cstate change: term changed from 3 to 4, leader changed from 04c981c15b3b481293007baaeeca8a94 (127.23.140.65) to 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67). New cstate: current_term: 4 leader_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:17.060304 25083 raft_consensus.cc:993] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94: : Instructing follower 9a1e5371bf904722bfabfdb1c97a5f7f to start an election
I20260709 13:07:17.060727 25090 raft_consensus.cc:1081] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Signalling peer 9a1e5371bf904722bfabfdb1c97a5f7f to start an election
I20260709 13:07:17.062000 24832 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e1196843f61e4f5a92a158a876ead792"
dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
 from {username='slave'} at 127.0.0.1:44466
I20260709 13:07:17.062428 24832 raft_consensus.cc:493] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:17.062723 24832 raft_consensus.cc:3060] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:17.066998 24832 raft_consensus.cc:515] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:17.068485 24832 leader_election.cc:290] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 3 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:17.069294 24757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1196843f61e4f5a92a158a876ead792" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:17.069561 24908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1196843f61e4f5a92a158a876ead792" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:17.069782 24757 raft_consensus.cc:3055] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:17.070103 24908 raft_consensus.cc:3060] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:17.070116 24757 raft_consensus.cc:740] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Running, Role: LEADER
I20260709 13:07:17.071177 24757 consensus_queue.cc:260] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:17.072072 24757 raft_consensus.cc:3060] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:17.074715 24908 raft_consensus.cc:2468] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 3.
I20260709 13:07:17.075832 24798 leader_election.cc:304] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:17.076519 25041 raft_consensus.cc:2804] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:17.077055 24757 raft_consensus.cc:2468] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 3.
I20260709 13:07:17.077775 25041 raft_consensus.cc:697] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 LEADER]: Becoming Leader. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Running, Role: LEADER
I20260709 13:07:17.078667 25041 consensus_queue.cc:237] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:17.085904 24658 catalog_manager.cc:5697] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f reported cstate change: term changed from 2 to 3, leader changed from 04c981c15b3b481293007baaeeca8a94 (127.23.140.65) to 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66). New cstate: current_term: 3 leader_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:17.112028 25090 raft_consensus.cc:993] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94: : Instructing follower 9a1e5371bf904722bfabfdb1c97a5f7f to start an election
I20260709 13:07:17.112529 25099 raft_consensus.cc:1081] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 4 LEADER]: Signalling peer 9a1e5371bf904722bfabfdb1c97a5f7f to start an election
I20260709 13:07:17.113816 24832 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82"
dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
 from {username='slave'} at 127.0.0.1:44466
I20260709 13:07:17.114233 24832 raft_consensus.cc:493] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:17.114490 24832 raft_consensus.cc:3060] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:07:17.118273 24832 raft_consensus.cc:515] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:17.119702 24832 leader_election.cc:290] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 5 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67:37083)
I20260709 13:07:17.120467 24757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:17.120723 24908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa3c2d8eadd9456a8490eb4671f18b82" candidate_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
I20260709 13:07:17.120999 24757 raft_consensus.cc:3055] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:07:17.121294 24908 raft_consensus.cc:3060] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:07:17.121335 24757 raft_consensus.cc:740] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Running, Role: LEADER
I20260709 13:07:17.122140 24757 consensus_queue.cc:260] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:17.123032 24757 raft_consensus.cc:3060] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:07:17.127012 24908 raft_consensus.cc:2468] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 5.
I20260709 13:07:17.128149 24798 leader_election.cc:304] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:17.129164 25041 raft_consensus.cc:2804] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:07:17.129658 25041 raft_consensus.cc:697] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 5 LEADER]: Becoming Leader. State: Replica: 9a1e5371bf904722bfabfdb1c97a5f7f, State: Running, Role: LEADER
I20260709 13:07:17.130427 25041 consensus_queue.cc:237] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } }
I20260709 13:07:17.129166 24757 raft_consensus.cc:2468] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1e5371bf904722bfabfdb1c97a5f7f in term 5.
I20260709 13:07:17.138509 24658 catalog_manager.cc:5697] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f reported cstate change: term changed from 4 to 5, leader changed from 04c981c15b3b481293007baaeeca8a94 (127.23.140.65) to 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66). New cstate: current_term: 5 leader_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:17.301378 25083 raft_consensus.cc:993] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94: : Instructing follower 1b0ad47f96d64c6f96c0f303d8b68fd4 to start an election
I20260709 13:07:17.301831 25099 raft_consensus.cc:1081] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Signalling peer 1b0ad47f96d64c6f96c0f303d8b68fd4 to start an election
I20260709 13:07:17.303222 24908 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "519702cc31404b58a9f7c00e5e012da5"
dest_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4"
 from {username='slave'} at 127.0.0.1:40042
I20260709 13:07:17.303670 24908 raft_consensus.cc:493] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:17.303969 24908 raft_consensus.cc:3060] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:17.308578 24908 raft_consensus.cc:515] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } }
I20260709 13:07:17.310148 24908 leader_election.cc:290] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 3 election: Requested vote from peers 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481), 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759)
I20260709 13:07:17.311003 24757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "519702cc31404b58a9f7c00e5e012da5" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "04c981c15b3b481293007baaeeca8a94"
I20260709 13:07:17.311156 24832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "519702cc31404b58a9f7c00e5e012da5" candidate_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f"
I20260709 13:07:17.311607 24757 raft_consensus.cc:3055] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:17.311661 24832 raft_consensus.cc:3060] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:17.311951 24757 raft_consensus.cc:740] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 04c981c15b3b481293007baaeeca8a94, State: Running, Role: LEADER
I20260709 13:07:17.312721 24757 consensus_queue.cc:260] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } }
I20260709 13:07:17.313612 24757 raft_consensus.cc:3060] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:17.315832 24832 raft_consensus.cc:2468] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 3.
I20260709 13:07:17.316704 24872 leader_election.cc:304] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0ad47f96d64c6f96c0f303d8b68fd4, 9a1e5371bf904722bfabfdb1c97a5f7f; no voters: 
I20260709 13:07:17.317399 25046 raft_consensus.cc:2804] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:17.317771 25046 raft_consensus.cc:697] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 LEADER]: Becoming Leader. State: Replica: 1b0ad47f96d64c6f96c0f303d8b68fd4, State: Running, Role: LEADER
I20260709 13:07:17.318346 24757 raft_consensus.cc:2468] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0ad47f96d64c6f96c0f303d8b68fd4 in term 3.
I20260709 13:07:17.318542 25046 consensus_queue.cc:237] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } }
I20260709 13:07:17.324875 24658 catalog_manager.cc:5697] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 reported cstate change: term changed from 2 to 3, leader changed from 04c981c15b3b481293007baaeeca8a94 (127.23.140.65) to 1b0ad47f96d64c6f96c0f303d8b68fd4 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:17.437002 24757 raft_consensus.cc:1275] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 4 FOLLOWER]: Refusing update from remote peer 1b0ad47f96d64c6f96c0f303d8b68fd4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:07:17.438108 25046 consensus_queue.cc:1048] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:17.447644 24832 raft_consensus.cc:1275] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 4 FOLLOWER]: Refusing update from remote peer 1b0ad47f96d64c6f96c0f303d8b68fd4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:07:17.448868 25046 consensus_queue.cc:1048] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:17.539762 24757 raft_consensus.cc:1275] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 5 FOLLOWER]: Refusing update from remote peer 9a1e5371bf904722bfabfdb1c97a5f7f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:07:17.540396 24908 raft_consensus.cc:1275] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Refusing update from remote peer 9a1e5371bf904722bfabfdb1c97a5f7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:17.540982 25041 consensus_queue.cc:1048] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:17.541743 25097 consensus_queue.cc:1048] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:17.556317 24907 raft_consensus.cc:1275] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 5 FOLLOWER]: Refusing update from remote peer 9a1e5371bf904722bfabfdb1c97a5f7f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:07:17.558528 25111 consensus_queue.cc:1048] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0ad47f96d64c6f96c0f303d8b68fd4" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:07:17.561106 24757 raft_consensus.cc:1275] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Refusing update from remote peer 9a1e5371bf904722bfabfdb1c97a5f7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:17.562673 25111 consensus_queue.cc:1048] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:17.704537 24757 raft_consensus.cc:1275] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Refusing update from remote peer 1b0ad47f96d64c6f96c0f303d8b68fd4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:17.705758 25046 consensus_queue.cc:1048] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c981c15b3b481293007baaeeca8a94" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:17.713569 24832 raft_consensus.cc:1275] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Refusing update from remote peer 1b0ad47f96d64c6f96c0f303d8b68fd4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:17.715124 25104 consensus_queue.cc:1048] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1e5371bf904722bfabfdb1c97a5f7f" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:17.769023 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:17.774766 24113 tablet_server.cc:179] TabletServer@127.23.140.65:0 shutting down...
I20260709 13:07:17.798391 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:07:17.799286 24113 tablet_replica.cc:333] T fab2dc5fe6b24dd180b77f547c8471c5 P 04c981c15b3b481293007baaeeca8a94: stopping tablet replica
I20260709 13:07:17.800361 24113 raft_consensus.cc:2243] T fab2dc5fe6b24dd180b77f547c8471c5 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:17.801074 24113 raft_consensus.cc:2272] T fab2dc5fe6b24dd180b77f547c8471c5 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.803917 24113 tablet_replica.cc:333] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94: stopping tablet replica
I20260709 13:07:17.804749 24113 raft_consensus.cc:2243] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:17.805269 24113 raft_consensus.cc:2272] T 24b03eb9381040d4b92e89e870caa867 P 04c981c15b3b481293007baaeeca8a94 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.807140 24113 tablet_replica.cc:333] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94: stopping tablet replica
I20260709 13:07:17.807578 24113 raft_consensus.cc:2243] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:17.808161 24113 raft_consensus.cc:2272] T e5e8e388810f4364b62faefb3c1ec45f P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.809705 24113 tablet_replica.cc:333] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94: stopping tablet replica
I20260709 13:07:17.810184 24113 raft_consensus.cc:2243] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:17.810505 24113 raft_consensus.cc:2272] T 519702cc31404b58a9f7c00e5e012da5 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.812456 24113 tablet_replica.cc:333] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94: stopping tablet replica
I20260709 13:07:17.812889 24113 raft_consensus.cc:2243] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:17.813575 24113 raft_consensus.cc:2272] T 22c9ddb498a04387bd6e576b0cce53b8 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.815503 24113 tablet_replica.cc:333] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94: stopping tablet replica
I20260709 13:07:17.816045 24113 raft_consensus.cc:2243] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:17.816501 24113 raft_consensus.cc:2272] T 4425bffab9a34e1aa5b95565fe6808f4 P 04c981c15b3b481293007baaeeca8a94 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.818138 24113 tablet_replica.cc:333] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94: stopping tablet replica
I20260709 13:07:17.818557 24113 raft_consensus.cc:2243] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:17.818877 24113 raft_consensus.cc:2272] T e1196843f61e4f5a92a158a876ead792 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.820605 24113 tablet_replica.cc:333] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94: stopping tablet replica
I20260709 13:07:17.821012 24113 raft_consensus.cc:2243] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:17.821396 24113 raft_consensus.cc:2272] T aa3c2d8eadd9456a8490eb4671f18b82 P 04c981c15b3b481293007baaeeca8a94 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.823024 24113 tablet_replica.cc:333] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94: stopping tablet replica
I20260709 13:07:17.823414 24113 raft_consensus.cc:2243] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:07:17.823992 24113 raft_consensus.cc:2272] T e31dbaadd96443bcafd21ec6a5cd19c4 P 04c981c15b3b481293007baaeeca8a94 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.825440 24113 tablet_replica.cc:333] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94: stopping tablet replica
I20260709 13:07:17.825805 24113 raft_consensus.cc:2243] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:07:17.826378 24113 raft_consensus.cc:2272] T 40603ccb1d484c139ca4806440e17898 P 04c981c15b3b481293007baaeeca8a94 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.854602 24113 tablet_server.cc:196] TabletServer@127.23.140.65:0 shutdown complete.
I20260709 13:07:17.873000 24113 tablet_server.cc:179] TabletServer@127.23.140.66:0 shutting down...
W20260709 13:07:17.883750 24872 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 9 similar messages]
W20260709 13:07:17.890781 24872 consensus_peers.cc:597] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 -> Peer 9a1e5371bf904722bfabfdb1c97a5f7f (127.23.140.66:33759): Couldn't send request to peer 9a1e5371bf904722bfabfdb1c97a5f7f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:17.899214 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:07:17.899988 24113 tablet_replica.cc:333] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f: stopping tablet replica
I20260709 13:07:17.900599 24113 raft_consensus.cc:2243] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:17.900977 24113 raft_consensus.cc:2272] T e31dbaadd96443bcafd21ec6a5cd19c4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.902667 24113 tablet_replica.cc:333] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f: stopping tablet replica
I20260709 13:07:17.903270 24113 raft_consensus.cc:2243] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:17.903625 24113 raft_consensus.cc:2272] T 40603ccb1d484c139ca4806440e17898 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.905228 24113 tablet_replica.cc:333] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f: stopping tablet replica
I20260709 13:07:17.905804 24113 raft_consensus.cc:2243] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:17.906145 24113 raft_consensus.cc:2272] T 24b03eb9381040d4b92e89e870caa867 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.908655 24113 tablet_replica.cc:333] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f: stopping tablet replica
I20260709 13:07:17.909233 24113 raft_consensus.cc:2243] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:17.909628 24113 raft_consensus.cc:2272] T fab2dc5fe6b24dd180b77f547c8471c5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.911114 24113 tablet_replica.cc:333] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f: stopping tablet replica
I20260709 13:07:17.911541 24113 raft_consensus.cc:2243] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 5 LEADER]: Raft consensus shutting down.
I20260709 13:07:17.912192 24113 raft_consensus.cc:2272] T aa3c2d8eadd9456a8490eb4671f18b82 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.914309 24113 tablet_replica.cc:333] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f: stopping tablet replica
I20260709 13:07:17.914736 24113 raft_consensus.cc:2243] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:17.915056 24113 raft_consensus.cc:2272] T e5e8e388810f4364b62faefb3c1ec45f P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.916651 24113 tablet_replica.cc:333] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f: stopping tablet replica
I20260709 13:07:17.917081 24113 raft_consensus.cc:2243] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:17.917511 24113 raft_consensus.cc:2272] T 519702cc31404b58a9f7c00e5e012da5 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.919327 24113 tablet_replica.cc:333] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f: stopping tablet replica
I20260709 13:07:17.919731 24113 raft_consensus.cc:2243] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:17.920343 24113 raft_consensus.cc:2272] T 4425bffab9a34e1aa5b95565fe6808f4 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.921854 24113 tablet_replica.cc:333] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f: stopping tablet replica
I20260709 13:07:17.922232 24113 raft_consensus.cc:2243] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:07:17.922850 24113 raft_consensus.cc:2272] T e1196843f61e4f5a92a158a876ead792 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.924497 24113 tablet_replica.cc:333] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f: stopping tablet replica
I20260709 13:07:17.924870 24113 raft_consensus.cc:2243] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:17.925257 24113 raft_consensus.cc:2272] T 22c9ddb498a04387bd6e576b0cce53b8 P 9a1e5371bf904722bfabfdb1c97a5f7f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:17.952886 24113 tablet_server.cc:196] TabletServer@127.23.140.66:0 shutdown complete.
I20260709 13:07:17.970228 24113 tablet_server.cc:179] TabletServer@127.23.140.67:0 shutting down...
I20260709 13:07:17.996868 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:07:17.997678 24113 tablet_replica.cc:333] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4: stopping tablet replica
W20260709 13:07:17.998271 24874 consensus_peers.cc:597] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 -> Peer 04c981c15b3b481293007baaeeca8a94 (127.23.140.65:46481): Couldn't send request to peer 04c981c15b3b481293007baaeeca8a94. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:46481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:17.998719 24113 raft_consensus.cc:2243] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:07:17.999620 24113 raft_consensus.cc:2272] T 24b03eb9381040d4b92e89e870caa867 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:18.001578 24113 tablet_replica.cc:333] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4: stopping tablet replica
I20260709 13:07:18.002038 24113 raft_consensus.cc:2243] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:18.002734 24113 raft_consensus.cc:2272] T fab2dc5fe6b24dd180b77f547c8471c5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:18.004356 24113 tablet_replica.cc:333] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4: stopping tablet replica
I20260709 13:07:18.004799 24113 raft_consensus.cc:2243] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:18.005235 24113 raft_consensus.cc:2272] T aa3c2d8eadd9456a8490eb4671f18b82 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:18.007004 24113 tablet_replica.cc:333] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4: stopping tablet replica
I20260709 13:07:18.007436 24113 raft_consensus.cc:2243] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:18.007803 24113 raft_consensus.cc:2272] T 22c9ddb498a04387bd6e576b0cce53b8 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:18.009362 24113 tablet_replica.cc:333] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4: stopping tablet replica
I20260709 13:07:18.009766 24113 raft_consensus.cc:2243] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:18.010072 24113 raft_consensus.cc:2272] T 40603ccb1d484c139ca4806440e17898 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:18.011482 24113 tablet_replica.cc:333] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4: stopping tablet replica
I20260709 13:07:18.011896 24113 raft_consensus.cc:2243] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:18.012244 24113 raft_consensus.cc:2272] T e31dbaadd96443bcafd21ec6a5cd19c4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:18.013746 24113 tablet_replica.cc:333] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4: stopping tablet replica
I20260709 13:07:18.014161 24113 raft_consensus.cc:2243] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:18.014523 24113 raft_consensus.cc:2272] T 4425bffab9a34e1aa5b95565fe6808f4 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:18.015980 24113 tablet_replica.cc:333] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4: stopping tablet replica
I20260709 13:07:18.016371 24113 raft_consensus.cc:2243] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:18.016705 24113 raft_consensus.cc:2272] T e1196843f61e4f5a92a158a876ead792 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:18.018587 24113 tablet_replica.cc:333] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4: stopping tablet replica
I20260709 13:07:18.019002 24113 raft_consensus.cc:2243] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:18.019320 24113 raft_consensus.cc:2272] T e5e8e388810f4364b62faefb3c1ec45f P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:18.020735 24113 tablet_replica.cc:333] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4: stopping tablet replica
I20260709 13:07:18.021102 24113 raft_consensus.cc:2243] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:07:18.021806 24113 raft_consensus.cc:2272] T 519702cc31404b58a9f7c00e5e012da5 P 1b0ad47f96d64c6f96c0f303d8b68fd4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:18.040381 24113 tablet_server.cc:196] TabletServer@127.23.140.67:0 shutdown complete.
I20260709 13:07:18.056946 24113 master.cc:562] Master@127.23.140.126:41765 shutting down...
I20260709 13:07:18.076784 24113 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:07:18.077378 24113 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:18.077723 24113 tablet_replica.cc:333] T 00000000000000000000000000000000 P 36709ee497be48979944e8465cd39dea: stopping tablet replica
I20260709 13:07:18.097664 24113 master.cc:584] Master@127.23.140.126:41765 shutdown complete.
04c981c15b3b481293007baaeeca8a94  5
1b0ad47f96d64c6f96c0f303d8b68fd4  1
9a1e5371bf904722bfabfdb1c97a5f7f  4
[       OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (10258 ms)
[ RUN      ] LeaderRebalancerTest.AddTserver
I20260709 13:07:18.135846 24113 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.140.126:33305
I20260709 13:07:18.136933 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:07:18.142616 25119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:18.142695 25120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:18.143319 25122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:18.144169 24113 server_base.cc:1061] running on GCE node
I20260709 13:07:18.144907 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:07:18.145152 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:07:18.145319 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602438145302 us; error 0 us; skew 500 ppm
I20260709 13:07:18.145831 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:07:18.148109 24113 webserver.cc:533] Webserver started at http://127.23.140.126:40865/ using document root <none> and password file <none>
I20260709 13:07:18.148584 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:07:18.148795 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:07:18.149068 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:07:18.150400 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/master-0-root/instance:
uuid: "4d591857b6ac45ad9cab82445b07ae03"
format_stamp: "Formatted at 2026-07-09 13:07:18 on dist-test-slave-k73n"
I20260709 13:07:18.154592 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 13:07:18.157642 25127 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:18.158335 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 13:07:18.158603 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/master-0-root
uuid: "4d591857b6ac45ad9cab82445b07ae03"
format_stamp: "Formatted at 2026-07-09 13:07:18 on dist-test-slave-k73n"
I20260709 13:07:18.158883 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:07:18.177248 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:07:18.178406 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:07:18.216210 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.126:33305
I20260709 13:07:18.216311 25178 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.126:33305 every 8 connection(s)
I20260709 13:07:18.220095 25179 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.230165 25179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Bootstrap starting.
I20260709 13:07:18.234607 25179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:18.238530 25179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: No bootstrap required, opened a new log
I20260709 13:07:18.240505 25179 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d591857b6ac45ad9cab82445b07ae03" member_type: VOTER }
I20260709 13:07:18.240885 25179 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:18.241119 25179 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d591857b6ac45ad9cab82445b07ae03, State: Initialized, Role: FOLLOWER
I20260709 13:07:18.241664 25179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d591857b6ac45ad9cab82445b07ae03" member_type: VOTER }
I20260709 13:07:18.242111 25179 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:07:18.242338 25179 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:07:18.242599 25179 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:18.246559 25179 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d591857b6ac45ad9cab82445b07ae03" member_type: VOTER }
I20260709 13:07:18.247077 25179 leader_election.cc:304] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d591857b6ac45ad9cab82445b07ae03; no voters: 
I20260709 13:07:18.248221 25179 leader_election.cc:290] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:07:18.248545 25182 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:18.249871 25182 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [term 1 LEADER]: Becoming Leader. State: Replica: 4d591857b6ac45ad9cab82445b07ae03, State: Running, Role: LEADER
I20260709 13:07:18.250523 25182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d591857b6ac45ad9cab82445b07ae03" member_type: VOTER }
I20260709 13:07:18.251164 25179 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:07:18.255046 25183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d591857b6ac45ad9cab82445b07ae03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d591857b6ac45ad9cab82445b07ae03" member_type: VOTER } }
I20260709 13:07:18.255738 25183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [sys.catalog]: This master's current role is: LEADER
I20260709 13:07:18.256619 25184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d591857b6ac45ad9cab82445b07ae03. Latest consensus state: current_term: 1 leader_uuid: "4d591857b6ac45ad9cab82445b07ae03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d591857b6ac45ad9cab82445b07ae03" member_type: VOTER } }
I20260709 13:07:18.257396 25184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [sys.catalog]: This master's current role is: LEADER
I20260709 13:07:18.260128 25190 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:07:18.267771 25190 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:07:18.268731 24113 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:07:18.276089 25190 catalog_manager.cc:1383] Generated new cluster ID: 906ea993108f452ba49e6ff4a97b2a70
I20260709 13:07:18.276394 25190 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:07:18.294857 25190 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:07:18.296033 25190 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:07:18.310391 25190 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Generated new TSK 0
I20260709 13:07:18.310936 25190 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:07:18.335369 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:07:18.341352 25200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:18.342519 25201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:18.345772 25203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:18.346416 24113 server_base.cc:1061] running on GCE node
I20260709 13:07:18.347293 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:07:18.347469 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:07:18.347585 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602438347575 us; error 0 us; skew 500 ppm
I20260709 13:07:18.348027 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:07:18.350216 24113 webserver.cc:533] Webserver started at http://127.23.140.65:42105/ using document root <none> and password file <none>
I20260709 13:07:18.350620 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:07:18.350770 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:07:18.350977 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:07:18.351948 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-0-root/instance:
uuid: "46de6919f85341c6a9b574dff112e35a"
format_stamp: "Formatted at 2026-07-09 13:07:18 on dist-test-slave-k73n"
I20260709 13:07:18.356215 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 13:07:18.359251 25208 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:18.360004 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.001s
I20260709 13:07:18.360343 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-0-root
uuid: "46de6919f85341c6a9b574dff112e35a"
format_stamp: "Formatted at 2026-07-09 13:07:18 on dist-test-slave-k73n"
I20260709 13:07:18.360692 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:07:18.385944 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:07:18.387159 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:07:18.388604 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:07:18.390821 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:07:18.390991 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:18.391278 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:07:18.391453 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:18.429980 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.65:45313
I20260709 13:07:18.430076 25270 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.65:45313 every 8 connection(s)
I20260709 13:07:18.434796 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:07:18.441871 25275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:18.442487 25276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:18.445441 24113 server_base.cc:1061] running on GCE node
W20260709 13:07:18.445869 25278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:18.446899 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:07:18.447137 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:07:18.447314 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602438447291 us; error 0 us; skew 500 ppm
I20260709 13:07:18.447803 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:07:18.450662 24113 webserver.cc:533] Webserver started at http://127.23.140.66:32773/ using document root <none> and password file <none>
I20260709 13:07:18.451303 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:07:18.451548 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:07:18.451871 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:07:18.452049 25271 heartbeater.cc:344] Connected to a master server at 127.23.140.126:33305
I20260709 13:07:18.452414 25271 heartbeater.cc:461] Registering TS with master...
I20260709 13:07:18.453083 25271 heartbeater.cc:507] Master 127.23.140.126:33305 requested a full tablet report, sending...
I20260709 13:07:18.453307 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-1-root/instance:
uuid: "2329e428ebbf484fa3ea9452133df567"
format_stamp: "Formatted at 2026-07-09 13:07:18 on dist-test-slave-k73n"
I20260709 13:07:18.455252 25144 ts_manager.cc:194] Registered new tserver with Master: 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:07:18.457651 25144 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39572
I20260709 13:07:18.458794 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 13:07:18.462270 25283 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:18.463089 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260709 13:07:18.463387 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-1-root
uuid: "2329e428ebbf484fa3ea9452133df567"
format_stamp: "Formatted at 2026-07-09 13:07:18 on dist-test-slave-k73n"
I20260709 13:07:18.463663 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:07:18.494397 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:07:18.495559 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:07:18.496999 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:07:18.499125 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:07:18.499351 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:18.499588 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:07:18.499756 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:18.538271 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.66:42315
I20260709 13:07:18.538374 25345 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.66:42315 every 8 connection(s)
I20260709 13:07:18.542929 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:07:18.549784 25349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:18.551812 25350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:18.553117 24113 server_base.cc:1061] running on GCE node
W20260709 13:07:18.553678 25352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:18.555001 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:07:18.555194 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:07:18.555421 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602438555400 us; error 0 us; skew 500 ppm
I20260709 13:07:18.555653 25346 heartbeater.cc:344] Connected to a master server at 127.23.140.126:33305
I20260709 13:07:18.556031 25346 heartbeater.cc:461] Registering TS with master...
I20260709 13:07:18.556102 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:07:18.556949 25346 heartbeater.cc:507] Master 127.23.140.126:33305 requested a full tablet report, sending...
I20260709 13:07:18.558862 24113 webserver.cc:533] Webserver started at http://127.23.140.67:39095/ using document root <none> and password file <none>
I20260709 13:07:18.559569 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:07:18.559660 25144 ts_manager.cc:194] Registered new tserver with Master: 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:18.559832 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:07:18.560298 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:07:18.561590 25144 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39582
I20260709 13:07:18.561698 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-2-root/instance:
uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
format_stamp: "Formatted at 2026-07-09 13:07:18 on dist-test-slave-k73n"
I20260709 13:07:18.566535 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 13:07:18.569720 25357 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:18.570492 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.002s
I20260709 13:07:18.570784 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-2-root
uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
format_stamp: "Formatted at 2026-07-09 13:07:18 on dist-test-slave-k73n"
I20260709 13:07:18.571086 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:07:18.598289 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:07:18.599452 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:07:18.600993 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:07:18.603188 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:07:18.603413 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:18.603654 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:07:18.603824 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:18.641000 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.67:34661
I20260709 13:07:18.641100 25419 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.67:34661 every 8 connection(s)
I20260709 13:07:18.654175 25420 heartbeater.cc:344] Connected to a master server at 127.23.140.126:33305
I20260709 13:07:18.654547 25420 heartbeater.cc:461] Registering TS with master...
I20260709 13:07:18.655208 25420 heartbeater.cc:507] Master 127.23.140.126:33305 requested a full tablet report, sending...
I20260709 13:07:18.657100 25144 ts_manager.cc:194] Registered new tserver with Master: a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:18.657343 24113 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013339879s
I20260709 13:07:18.658612 25144 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39596
I20260709 13:07:18.681655 25144 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39602:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:07:18.688649 25144 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:07:18.879786 25311 tablet_service.cc:1511] Processing CreateTablet for tablet 991bcf9b91a14242837dfb4e18891e91 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:07:18.880383 25310 tablet_service.cc:1511] Processing CreateTablet for tablet eda233ace2a24b379fef320a5f610c7a (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:07:18.881296 25311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 991bcf9b91a14242837dfb4e18891e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.882434 25310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eda233ace2a24b379fef320a5f610c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.882927 25309 tablet_service.cc:1511] Processing CreateTablet for tablet 58ec1834a96f447f8fe74448e8480342 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:07:18.884227 25309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ec1834a96f447f8fe74448e8480342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.889847 25308 tablet_service.cc:1511] Processing CreateTablet for tablet 8fa7e9050d44475381c97025c2b11711 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:07:18.891152 25308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fa7e9050d44475381c97025c2b11711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.891870 25235 tablet_service.cc:1511] Processing CreateTablet for tablet eda233ace2a24b379fef320a5f610c7a (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:07:18.892613 25236 tablet_service.cc:1511] Processing CreateTablet for tablet 991bcf9b91a14242837dfb4e18891e91 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:07:18.893249 25235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eda233ace2a24b379fef320a5f610c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.893872 25236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 991bcf9b91a14242837dfb4e18891e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.895288 25234 tablet_service.cc:1511] Processing CreateTablet for tablet 58ec1834a96f447f8fe74448e8480342 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:07:18.896610 25234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ec1834a96f447f8fe74448e8480342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.898330 25233 tablet_service.cc:1511] Processing CreateTablet for tablet 8fa7e9050d44475381c97025c2b11711 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:07:18.899631 25233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fa7e9050d44475381c97025c2b11711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.912249 25307 tablet_service.cc:1511] Processing CreateTablet for tablet cb389d32141944d4b8f26a17471d50fc (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:07:18.913698 25307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb389d32141944d4b8f26a17471d50fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.914959 25232 tablet_service.cc:1511] Processing CreateTablet for tablet cb389d32141944d4b8f26a17471d50fc (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:07:18.916304 25232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb389d32141944d4b8f26a17471d50fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.920250 25385 tablet_service.cc:1511] Processing CreateTablet for tablet 991bcf9b91a14242837dfb4e18891e91 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:07:18.922240 25384 tablet_service.cc:1511] Processing CreateTablet for tablet eda233ace2a24b379fef320a5f610c7a (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:07:18.924019 25383 tablet_service.cc:1511] Processing CreateTablet for tablet 58ec1834a96f447f8fe74448e8480342 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:07:18.925830 25382 tablet_service.cc:1511] Processing CreateTablet for tablet 8fa7e9050d44475381c97025c2b11711 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:07:18.927659 25381 tablet_service.cc:1511] Processing CreateTablet for tablet cb389d32141944d4b8f26a17471d50fc (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:07:18.928277 25385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 991bcf9b91a14242837dfb4e18891e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.930264 25381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb389d32141944d4b8f26a17471d50fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.937014 25380 tablet_service.cc:1511] Processing CreateTablet for tablet bbeee5054bca4ae28d33636ae50ca11f (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:07:18.938501 25380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbeee5054bca4ae28d33636ae50ca11f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.937316 25235 tablet_service.cc:1511] Processing CreateTablet for tablet bbeee5054bca4ae28d33636ae50ca11f (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:07:18.946503 25235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbeee5054bca4ae28d33636ae50ca11f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.940778 25383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ec1834a96f447f8fe74448e8480342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.950165 25309 tablet_service.cc:1511] Processing CreateTablet for tablet 3cfa8937a5964f5790737950fc3b294e (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:07:18.951460 25309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cfa8937a5964f5790737950fc3b294e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.951982 25436 tablet_bootstrap.cc:492] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:18.953475 25379 tablet_service.cc:1511] Processing CreateTablet for tablet 3cfa8937a5964f5790737950fc3b294e (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:07:18.954720 25379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cfa8937a5964f5790737950fc3b294e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.942081 25231 tablet_service.cc:1511] Processing CreateTablet for tablet 3cfa8937a5964f5790737950fc3b294e (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:07:18.957937 25231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cfa8937a5964f5790737950fc3b294e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.958822 25435 tablet_bootstrap.cc:492] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:18.961477 25384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eda233ace2a24b379fef320a5f610c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.963317 25234 tablet_service.cc:1511] Processing CreateTablet for tablet 3bd9cfccf308499991b366150c2d46ea (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:07:18.964543 25234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bd9cfccf308499991b366150c2d46ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.965080 25382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fa7e9050d44475381c97025c2b11711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.938613 25311 tablet_service.cc:1511] Processing CreateTablet for tablet bbeee5054bca4ae28d33636ae50ca11f (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:07:18.966646 25311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbeee5054bca4ae28d33636ae50ca11f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.966576 25310 tablet_service.cc:1511] Processing CreateTablet for tablet 3bd9cfccf308499991b366150c2d46ea (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:07:18.973431 25310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bd9cfccf308499991b366150c2d46ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.983328 25436 tablet_bootstrap.cc:654] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:18.984858 25378 tablet_service.cc:1511] Processing CreateTablet for tablet 3bd9cfccf308499991b366150c2d46ea (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:07:18.986272 25378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bd9cfccf308499991b366150c2d46ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.986851 25233 tablet_service.cc:1511] Processing CreateTablet for tablet 6afb077d630246d6a315a828b4dcc108 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:07:18.988045 25233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6afb077d630246d6a315a828b4dcc108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.991683 25377 tablet_service.cc:1511] Processing CreateTablet for tablet 6afb077d630246d6a315a828b4dcc108 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:07:18.993024 25377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6afb077d630246d6a315a828b4dcc108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.993551 25232 tablet_service.cc:1511] Processing CreateTablet for tablet 6ff9a7b00f104dd294125870b06a7df4 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:07:18.994750 25232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ff9a7b00f104dd294125870b06a7df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.995714 25307 tablet_service.cc:1511] Processing CreateTablet for tablet 6ff9a7b00f104dd294125870b06a7df4 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:07:18.996932 25307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ff9a7b00f104dd294125870b06a7df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:18.976990 25435 tablet_bootstrap.cc:654] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:18.998757 25309 tablet_service.cc:1511] Processing CreateTablet for tablet 6afb077d630246d6a315a828b4dcc108 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:07:18.999965 25309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6afb077d630246d6a315a828b4dcc108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.023903 25376 tablet_service.cc:1511] Processing CreateTablet for tablet 6ff9a7b00f104dd294125870b06a7df4 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:07:19.029759 25376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ff9a7b00f104dd294125870b06a7df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.038477 25235 tablet_service.cc:1511] Processing CreateTablet for tablet e81523f4f4344a04a1e0b03c8de99290 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:07:19.047497 25235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e81523f4f4344a04a1e0b03c8de99290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.062107 25435 tablet_bootstrap.cc:492] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:19.065959 25435 ts_tablet_manager.cc:1403] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.107s	user 0.020s	sys 0.000s
I20260709 13:07:19.068511 25437 tablet_bootstrap.cc:492] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:19.070871 25436 tablet_bootstrap.cc:492] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:19.071401 25436 ts_tablet_manager.cc:1403] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.120s	user 0.014s	sys 0.006s
I20260709 13:07:19.070994 25310 tablet_service.cc:1511] Processing CreateTablet for tablet e81523f4f4344a04a1e0b03c8de99290 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:07:19.072397 25310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e81523f4f4344a04a1e0b03c8de99290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.075393 25234 tablet_service.cc:1511] Processing CreateTablet for tablet f75b60146e7d4b9b9954b4fb2c414679 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:07:19.076687 25234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f75b60146e7d4b9b9954b4fb2c414679. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.082581 25308 tablet_service.cc:1511] Processing CreateTablet for tablet f75b60146e7d4b9b9954b4fb2c414679 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:07:19.083904 25308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f75b60146e7d4b9b9954b4fb2c414679. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.084812 25435 raft_consensus.cc:359] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } }
I20260709 13:07:19.085605 25435 raft_consensus.cc:385] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.085903 25435 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.088109 25436 raft_consensus.cc:359] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.088874 25436 raft_consensus.cc:385] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.089213 25436 raft_consensus.cc:740] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.090101 25435 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } }
I20260709 13:07:19.091605 25436 consensus_queue.cc:260] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.096670 25233 tablet_service.cc:1511] Processing CreateTablet for tablet a53b1fb6477d486e98bff9549cc487df (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:07:19.097631 25306 tablet_service.cc:1511] Processing CreateTablet for tablet a53b1fb6477d486e98bff9549cc487df (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:07:19.098119 25233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a53b1fb6477d486e98bff9549cc487df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.098903 25306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a53b1fb6477d486e98bff9549cc487df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.083496 25385 tablet_service.cc:1511] Processing CreateTablet for tablet e81523f4f4344a04a1e0b03c8de99290 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:07:19.101800 25385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e81523f4f4344a04a1e0b03c8de99290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.110702 25380 tablet_service.cc:1511] Processing CreateTablet for tablet f75b60146e7d4b9b9954b4fb2c414679 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:07:19.112555 25380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f75b60146e7d4b9b9954b4fb2c414679. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.114629 25436 ts_tablet_manager.cc:1434] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.043s	user 0.006s	sys 0.005s
I20260709 13:07:19.116137 25346 heartbeater.cc:499] Master 127.23.140.126:33305 was elected leader, sending a full tablet report...
I20260709 13:07:19.134222 25232 tablet_service.cc:1511] Processing CreateTablet for tablet 9dac81b348b746fd8ecd3937028554ea (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:07:19.135622 25232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dac81b348b746fd8ecd3937028554ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.137183 25381 tablet_service.cc:1511] Processing CreateTablet for tablet a53b1fb6477d486e98bff9549cc487df (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:07:19.137624 25271 heartbeater.cc:499] Master 127.23.140.126:33305 was elected leader, sending a full tablet report...
I20260709 13:07:19.138469 25381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a53b1fb6477d486e98bff9549cc487df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.142812 25436 tablet_bootstrap.cc:492] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:19.146423 25435 ts_tablet_manager.cc:1434] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.080s	user 0.005s	sys 0.004s
I20260709 13:07:19.147523 25435 tablet_bootstrap.cc:492] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:19.152232 25437 tablet_bootstrap.cc:654] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.159123 25436 tablet_bootstrap.cc:654] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.167402 25306 tablet_service.cc:1511] Processing CreateTablet for tablet 9dac81b348b746fd8ecd3937028554ea (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:07:19.168782 25306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dac81b348b746fd8ecd3937028554ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.172271 25435 tablet_bootstrap.cc:654] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.184497 25231 tablet_service.cc:1511] Processing CreateTablet for tablet b62f4877af9742298b1e149789ca3090 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:07:19.184777 25384 tablet_service.cc:1511] Processing CreateTablet for tablet 9dac81b348b746fd8ecd3937028554ea (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:07:19.185899 25231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b62f4877af9742298b1e149789ca3090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.186275 25384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dac81b348b746fd8ecd3937028554ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.185933 25379 tablet_service.cc:1511] Processing CreateTablet for tablet b62f4877af9742298b1e149789ca3090 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:07:19.186535 25436 tablet_bootstrap.cc:492] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:19.187050 25436 ts_tablet_manager.cc:1403] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.044s	user 0.015s	sys 0.007s
I20260709 13:07:19.187247 25379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b62f4877af9742298b1e149789ca3090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.190065 25436 raft_consensus.cc:359] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.190819 25436 raft_consensus.cc:385] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.191136 25436 raft_consensus.cc:740] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.191185 25311 tablet_service.cc:1511] Processing CreateTablet for tablet b62f4877af9742298b1e149789ca3090 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:07:19.192577 25311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b62f4877af9742298b1e149789ca3090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.200596 25235 tablet_service.cc:1511] Processing CreateTablet for tablet acd2d31837dd470c8bd91f719ffeb461 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:07:19.201653 25308 tablet_service.cc:1511] Processing CreateTablet for tablet acd2d31837dd470c8bd91f719ffeb461 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:07:19.202064 25235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acd2d31837dd470c8bd91f719ffeb461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.202966 25308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acd2d31837dd470c8bd91f719ffeb461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.203768 25376 tablet_service.cc:1511] Processing CreateTablet for tablet acd2d31837dd470c8bd91f719ffeb461 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:07:19.205415 25376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acd2d31837dd470c8bd91f719ffeb461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.226871 25234 tablet_service.cc:1511] Processing CreateTablet for tablet de96ce5d28144cf3894e100bfc4959fa (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:07:19.227188 25307 tablet_service.cc:1511] Processing CreateTablet for tablet de96ce5d28144cf3894e100bfc4959fa (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:07:19.228425 25307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de96ce5d28144cf3894e100bfc4959fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.228231 25236 tablet_service.cc:1511] Processing CreateTablet for tablet 1884b7159da148dbb1c19c424d667792 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:07:19.228274 25234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de96ce5d28144cf3894e100bfc4959fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.230571 25435 tablet_bootstrap.cc:492] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:19.231056 25435 ts_tablet_manager.cc:1403] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.084s	user 0.008s	sys 0.004s
I20260709 13:07:19.237660 25440 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:19.238116 25440 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } }
I20260709 13:07:19.241214 25436 consensus_queue.cc:260] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.243295 25236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1884b7159da148dbb1c19c424d667792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.243752 25379 tablet_service.cc:1511] Processing CreateTablet for tablet de96ce5d28144cf3894e100bfc4959fa (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:07:19.245848 25379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de96ce5d28144cf3894e100bfc4959fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.251695 25436 ts_tablet_manager.cc:1434] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.064s	user 0.007s	sys 0.001s
I20260709 13:07:19.254854 25440 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:19.270910 25437 tablet_bootstrap.cc:492] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:19.271467 25437 ts_tablet_manager.cc:1403] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.203s	user 0.006s	sys 0.035s
I20260709 13:07:19.273697 25376 tablet_service.cc:1511] Processing CreateTablet for tablet 1884b7159da148dbb1c19c424d667792 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:07:19.276235 25231 tablet_service.cc:1511] Processing CreateTablet for tablet 69fee8cad2c84e1492403c23786622a1 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:07:19.273842 25435 raft_consensus.cc:359] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.277400 25435 raft_consensus.cc:385] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.277750 25435 raft_consensus.cc:740] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.277844 25231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69fee8cad2c84e1492403c23786622a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.278592 25435 consensus_queue.cc:260] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.280994 25435 ts_tablet_manager.cc:1434] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.050s	user 0.008s	sys 0.000s
I20260709 13:07:19.274574 25437 raft_consensus.cc:359] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } }
I20260709 13:07:19.283239 25437 raft_consensus.cc:385] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.283718 25437 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.284261 25435 tablet_bootstrap.cc:492] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:19.282363 25307 tablet_service.cc:1511] Processing CreateTablet for tablet 1884b7159da148dbb1c19c424d667792 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:07:19.284613 25437 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } }
I20260709 13:07:19.289832 25380 tablet_service.cc:1511] Processing CreateTablet for tablet 69fee8cad2c84e1492403c23786622a1 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
W20260709 13:07:19.286092 25192 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:07:19.276993 25376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1884b7159da148dbb1c19c424d667792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.290913 25233 tablet_service.cc:1511] Processing CreateTablet for tablet 2dc23779efbf4518a6ada8bde8243a11 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:07:19.292133 25233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dc23779efbf4518a6ada8bde8243a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.292975 25310 tablet_service.cc:1511] Processing CreateTablet for tablet 2dc23779efbf4518a6ada8bde8243a11 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:07:19.294191 25310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dc23779efbf4518a6ada8bde8243a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.294859 25384 tablet_service.cc:1511] Processing CreateTablet for tablet 2dc23779efbf4518a6ada8bde8243a11 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:07:19.285439 25436 tablet_bootstrap.cc:492] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:19.292994 25380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69fee8cad2c84e1492403c23786622a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.297873 25309 tablet_service.cc:1511] Processing CreateTablet for tablet d1bed174e340476591fddc8217b8f644 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:07:19.299120 25309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1bed174e340476591fddc8217b8f644. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.299402 25384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dc23779efbf4518a6ada8bde8243a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.302672 25307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1884b7159da148dbb1c19c424d667792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.304268 25382 tablet_service.cc:1511] Processing CreateTablet for tablet d1bed174e340476591fddc8217b8f644 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:07:19.305536 25382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1bed174e340476591fddc8217b8f644. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.305406 25235 tablet_service.cc:1511] Processing CreateTablet for tablet 6f616cb6823f424a847dceaa5900dfc0 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:07:19.306694 25235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f616cb6823f424a847dceaa5900dfc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.304263 25306 tablet_service.cc:1511] Processing CreateTablet for tablet 69fee8cad2c84e1492403c23786622a1 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:07:19.314420 25232 tablet_service.cc:1511] Processing CreateTablet for tablet d1bed174e340476591fddc8217b8f644 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:07:19.314940 25306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69fee8cad2c84e1492403c23786622a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.315610 25232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1bed174e340476591fddc8217b8f644. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.318558 25441 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:19.319022 25441 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.323102 25441 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:19.347934 25436 tablet_bootstrap.cc:654] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.333004 25435 tablet_bootstrap.cc:654] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.374511 25420 heartbeater.cc:499] Master 127.23.140.126:33305 was elected leader, sending a full tablet report...
I20260709 13:07:19.379034 25379 tablet_service.cc:1511] Processing CreateTablet for tablet 17ef4786898b4ad696baed95b4209788 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:07:19.380383 25379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17ef4786898b4ad696baed95b4209788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.362816 25382 tablet_service.cc:1511] Processing CreateTablet for tablet 6f616cb6823f424a847dceaa5900dfc0 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:07:19.385162 25382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f616cb6823f424a847dceaa5900dfc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.389163 25236 tablet_service.cc:1511] Processing CreateTablet for tablet 3cff7b4cb7184e4c9a69372d5901e19a (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:07:19.385613 25233 tablet_service.cc:1511] Processing CreateTablet for tablet 17ef4786898b4ad696baed95b4209788 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:07:19.390416 25236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cff7b4cb7184e4c9a69372d5901e19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.374663 25437 ts_tablet_manager.cc:1434] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.103s	user 0.007s	sys 0.019s
I20260709 13:07:19.401245 25437 tablet_bootstrap.cc:492] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:19.406894 25233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17ef4786898b4ad696baed95b4209788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.406586 25308 tablet_service.cc:1511] Processing CreateTablet for tablet 17ef4786898b4ad696baed95b4209788 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:07:19.407814 25308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17ef4786898b4ad696baed95b4209788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.409880 25310 tablet_service.cc:1511] Processing CreateTablet for tablet 6f616cb6823f424a847dceaa5900dfc0 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:07:19.411284 25310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f616cb6823f424a847dceaa5900dfc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.423974 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:19.424667 25395 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:19.428040 25311 tablet_service.cc:1511] Processing CreateTablet for tablet e3025c38ad5a4bdfa61cfca91ba3c409 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:07:19.429350 25311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3025c38ad5a4bdfa61cfca91ba3c409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.429617 25384 tablet_service.cc:1511] Processing CreateTablet for tablet e3025c38ad5a4bdfa61cfca91ba3c409 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:07:19.429759 25234 tablet_service.cc:1511] Processing CreateTablet for tablet e3025c38ad5a4bdfa61cfca91ba3c409 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:07:19.430832 25384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3025c38ad5a4bdfa61cfca91ba3c409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.431207 25234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3025c38ad5a4bdfa61cfca91ba3c409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.435093 25212 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:19.436013 25440 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:19.435832 25309 tablet_service.cc:1511] Processing CreateTablet for tablet 3cff7b4cb7184e4c9a69372d5901e19a (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:07:19.436403 25440 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:19.436813 25440 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:19.437204 25309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cff7b4cb7184e4c9a69372d5901e19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.444105 25230 tablet_service.cc:1511] Processing CreateTablet for tablet 2730c370dd9a4a1ea5adfb543e3ba120 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:07:19.444180 25380 tablet_service.cc:1511] Processing CreateTablet for tablet 3cff7b4cb7184e4c9a69372d5901e19a (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:07:19.445713 25230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2730c370dd9a4a1ea5adfb543e3ba120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.447144 25305 tablet_service.cc:1511] Processing CreateTablet for tablet 2730c370dd9a4a1ea5adfb543e3ba120 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:07:19.448608 25305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2730c370dd9a4a1ea5adfb543e3ba120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.450769 25435 tablet_bootstrap.cc:492] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:19.451285 25435 ts_tablet_manager.cc:1403] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.168s	user 0.002s	sys 0.013s
I20260709 13:07:19.456276 25381 tablet_service.cc:1511] Processing CreateTablet for tablet 2730c370dd9a4a1ea5adfb543e3ba120 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:07:19.457845 25381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2730c370dd9a4a1ea5adfb543e3ba120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.458096 25229 tablet_service.cc:1511] Processing CreateTablet for tablet c5481170c5c74798bdfd146068f8b539 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:07:19.459781 25229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5481170c5c74798bdfd146068f8b539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.463902 25304 tablet_service.cc:1511] Processing CreateTablet for tablet c5481170c5c74798bdfd146068f8b539 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:07:19.465435 25304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5481170c5c74798bdfd146068f8b539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.467545 25435 raft_consensus.cc:359] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.468303 25435 raft_consensus.cc:385] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.468616 25435 raft_consensus.cc:740] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.468982 25380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cff7b4cb7184e4c9a69372d5901e19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.452298 25440 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } }
I20260709 13:07:19.486796 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:19.488878 25440 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:19.489454 25440 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:19.489890 25440 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.493160 25440 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:19.498663 25441 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:19.498569 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:19.499171 25441 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:07:19.500335 25209 leader_election.cc:343] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:19.502739 25441 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:19.505896 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:19.505987 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:19.506738 25395 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:19.507761 25285 leader_election.cc:343] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:19.508452 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:19.504192 25379 tablet_service.cc:1511] Processing CreateTablet for tablet c5481170c5c74798bdfd146068f8b539 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:07:19.509492 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:07:19.509657 25209 leader_election.cc:343] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:19.510443 25379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5481170c5c74798bdfd146068f8b539. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:07:19.510818 25209 leader_election.cc:343] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:19.514410 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:07:19.516405 25212 leader_election.cc:343] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:19.516829 25212 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:19.518381 25440 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:19.520721 25436 tablet_bootstrap.cc:492] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:19.521231 25436 ts_tablet_manager.cc:1403] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.236s	user 0.015s	sys 0.009s
I20260709 13:07:19.521479 25235 tablet_service.cc:1511] Processing CreateTablet for tablet bacfad6698774ab8b26566bcecbcb615 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:07:19.522805 25235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bacfad6698774ab8b26566bcecbcb615. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.524080 25436 raft_consensus.cc:359] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } }
I20260709 13:07:19.524762 25436 raft_consensus.cc:385] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.525060 25436 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.525909 25436 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } }
I20260709 13:07:19.530598 25309 tablet_service.cc:1511] Processing CreateTablet for tablet bacfad6698774ab8b26566bcecbcb615 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:07:19.531803 25436 ts_tablet_manager.cc:1434] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 13:07:19.534750 25436 tablet_bootstrap.cc:492] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:19.503170 25435 consensus_queue.cc:260] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.549733 25246 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:19.552865 25285 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:19.553719 25441 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:19.554106 25441 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:19.554433 25441 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:19.556515 25395 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:19.557564 25437 tablet_bootstrap.cc:654] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.557672 25212 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:19.562131 25453 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:19.562665 25453 raft_consensus.cc:697] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:19.563571 25453 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } }
I20260709 13:07:19.568570 25435 ts_tablet_manager.cc:1434] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.117s	user 0.001s	sys 0.009s
I20260709 13:07:19.570155 25435 tablet_bootstrap.cc:492] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:19.573053 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:19.562951 25441 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.577311 25441 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:19.577529 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:19.577960 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:19.586048 25246 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:19.590941 25287 leader_election.cc:343] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:19.591444 25287 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:07:19.592706 25287 leader_election.cc:343] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:19.592667 25308 tablet_service.cc:1511] Processing CreateTablet for tablet 8fd399e2f6384e98ada57177a6010d14 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:07:19.594583 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:19.595721 25230 tablet_service.cc:1511] Processing CreateTablet for tablet 8f209a6539f54de28906492f142efb6b (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:07:19.595891 25231 tablet_service.cc:1511] Processing CreateTablet for tablet 8fd399e2f6384e98ada57177a6010d14 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:07:19.598201 25228 tablet_service.cc:1511] Processing CreateTablet for tablet 4e29e9ab7eda4242964fa76e29c2a221 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
W20260709 13:07:19.598522 25287 leader_election.cc:343] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:19.581037 25309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bacfad6698774ab8b26566bcecbcb615. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.600385 25230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f209a6539f54de28906492f142efb6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.591536 25304 tablet_service.cc:1511] Processing CreateTablet for tablet 9b5c9f65605b4dfca1bbfe0f09629c6d (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:07:19.601642 25382 tablet_service.cc:1511] Processing CreateTablet for tablet 9b5c9f65605b4dfca1bbfe0f09629c6d (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:07:19.603611 25384 tablet_service.cc:1511] Processing CreateTablet for tablet 8fd399e2f6384e98ada57177a6010d14 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:07:19.598788 25380 tablet_service.cc:1511] Processing CreateTablet for tablet 8f209a6539f54de28906492f142efb6b (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:07:19.598555 25311 tablet_service.cc:1511] Processing CreateTablet for tablet 8f209a6539f54de28906492f142efb6b (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:07:19.607308 25231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fd399e2f6384e98ada57177a6010d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.617893 25306 tablet_service.cc:1511] Processing CreateTablet for tablet 4e29e9ab7eda4242964fa76e29c2a221 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:07:19.606421 25441 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:19.621181 25441 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:19.621658 25441 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } }
I20260709 13:07:19.622459 25382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b5c9f65605b4dfca1bbfe0f09629c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.624605 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:19.625437 25395 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 1: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:07:19.629014 25384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fd399e2f6384e98ada57177a6010d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.631026 25380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f209a6539f54de28906492f142efb6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.634315 25441 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:07:19.634860 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:19.636478 25285 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:19.636530 25232 tablet_service.cc:1511] Processing CreateTablet for tablet 9b5c9f65605b4dfca1bbfe0f09629c6d (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:07:19.637881 25232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b5c9f65605b4dfca1bbfe0f09629c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.641744 25381 tablet_service.cc:1511] Processing CreateTablet for tablet bacfad6698774ab8b26566bcecbcb615 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:07:19.642984 25381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bacfad6698774ab8b26566bcecbcb615. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.637789 25441 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:19.645912 25308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fd399e2f6384e98ada57177a6010d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.647431 25228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e29e9ab7eda4242964fa76e29c2a221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.663110 25306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e29e9ab7eda4242964fa76e29c2a221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.669559 25304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b5c9f65605b4dfca1bbfe0f09629c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.672292 25311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f209a6539f54de28906492f142efb6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.676702 25441 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:19.699766 25436 tablet_bootstrap.cc:654] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.700613 25379 tablet_service.cc:1511] Processing CreateTablet for tablet 4e29e9ab7eda4242964fa76e29c2a221 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:07:19.702103 25379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e29e9ab7eda4242964fa76e29c2a221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.703933 25437 tablet_bootstrap.cc:492] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:19.704488 25437 ts_tablet_manager.cc:1403] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.304s	user 0.035s	sys 0.037s
I20260709 13:07:19.707372 25437 raft_consensus.cc:359] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.708076 25437 raft_consensus.cc:385] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.708391 25437 raft_consensus.cc:740] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.709254 25437 consensus_queue.cc:260] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.711503 25437 ts_tablet_manager.cc:1434] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260709 13:07:19.712481 25437 tablet_bootstrap.cc:492] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:19.717056 25231 tablet_service.cc:1511] Processing CreateTablet for tablet 292e92613a634963a5de1f016eb951ea (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:07:19.718425 25231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292e92613a634963a5de1f016eb951ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.718328 25381 tablet_service.cc:1511] Processing CreateTablet for tablet 0e2fe177353b4d6c9de4ccf872ca06ca (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:07:19.699812 25246 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:19.722146 25381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2fe177353b4d6c9de4ccf872ca06ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.722229 25304 tablet_service.cc:1511] Processing CreateTablet for tablet 292e92613a634963a5de1f016eb951ea (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:07:19.723446 25304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292e92613a634963a5de1f016eb951ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.725165 25232 tablet_service.cc:1511] Processing CreateTablet for tablet 7f37bfaeb12446c1a84af172a6f3dde9 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:07:19.717257 25306 tablet_service.cc:1511] Processing CreateTablet for tablet 0e2fe177353b4d6c9de4ccf872ca06ca (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:07:19.720153 25380 tablet_service.cc:1511] Processing CreateTablet for tablet 292e92613a634963a5de1f016eb951ea (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:07:19.730880 25447 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:19.759825 25447 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.761878 25234 tablet_service.cc:1511] Processing CreateTablet for tablet adf2f61a8d304d4eb9c30a77d7de17e4 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:07:19.762431 25447 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:19.731688 25285 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:19.763399 25456 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:19.763871 25456 raft_consensus.cc:697] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:19.764281 25383 tablet_service.cc:1511] Processing CreateTablet for tablet adf2f61a8d304d4eb9c30a77d7de17e4 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:07:19.764675 25456 consensus_queue.cc:237] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.767073 25305 tablet_service.cc:1511] Processing CreateTablet for tablet adf2f61a8d304d4eb9c30a77d7de17e4 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:07:19.768297 25305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adf2f61a8d304d4eb9c30a77d7de17e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.733618 25232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f37bfaeb12446c1a84af172a6f3dde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.735929 25382 tablet_service.cc:1511] Processing CreateTablet for tablet 7f37bfaeb12446c1a84af172a6f3dde9 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:07:19.773989 25307 tablet_service.cc:1511] Processing CreateTablet for tablet 5ead2e92f0614db79c9249687c4246f9 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:07:19.733635 25311 tablet_service.cc:1511] Processing CreateTablet for tablet 7f37bfaeb12446c1a84af172a6f3dde9 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:07:19.737764 25435 tablet_bootstrap.cc:654] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.728788 25142 catalog_manager.cc:5697] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:19.747814 25437 tablet_bootstrap.cc:654] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.748194 25236 tablet_service.cc:1511] Processing CreateTablet for tablet 46ab4fd91d3f4766a7b97848f5244360 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:07:19.752731 25384 tablet_service.cc:1511] Processing CreateTablet for tablet 46ab4fd91d3f4766a7b97848f5244360 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:07:19.758193 25376 tablet_service.cc:1511] Processing CreateTablet for tablet cf0501919f944794bbab0ffae6a334e1 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:07:19.759472 25380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292e92613a634963a5de1f016eb951ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.774184 25310 tablet_service.cc:1511] Processing CreateTablet for tablet 17584a19690d4010851d65378cfcbafa (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:07:19.731474 25228 tablet_service.cc:1511] Processing CreateTablet for tablet 0e2fe177353b4d6c9de4ccf872ca06ca (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:07:19.767949 25308 tablet_service.cc:1511] Processing CreateTablet for tablet cf0501919f944794bbab0ffae6a334e1 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:07:19.790851 25307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ead2e92f0614db79c9249687c4246f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.790524 25233 tablet_service.cc:1511] Processing CreateTablet for tablet 17584a19690d4010851d65378cfcbafa (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:07:19.771770 25230 tablet_service.cc:1511] Processing CreateTablet for tablet cf0501919f944794bbab0ffae6a334e1 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:07:19.787539 25234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adf2f61a8d304d4eb9c30a77d7de17e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.801963 25376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf0501919f944794bbab0ffae6a334e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.813530 25377 tablet_service.cc:1511] Processing CreateTablet for tablet 17584a19690d4010851d65378cfcbafa (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:07:19.814064 25228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2fe177353b4d6c9de4ccf872ca06ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.814872 25236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46ab4fd91d3f4766a7b97848f5244360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.816615 25235 tablet_service.cc:1511] Processing CreateTablet for tablet 5ead2e92f0614db79c9249687c4246f9 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:07:19.817828 25235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ead2e92f0614db79c9249687c4246f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.822556 25233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17584a19690d4010851d65378cfcbafa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.823045 25436 tablet_bootstrap.cc:492] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:19.823504 25436 ts_tablet_manager.cc:1403] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.289s	user 0.025s	sys 0.023s
I20260709 13:07:19.827558 25229 tablet_service.cc:1511] Processing CreateTablet for tablet 1b8652aecc6d4869bb4cad9e0749f3ba (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:07:19.828773 25229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b8652aecc6d4869bb4cad9e0749f3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.829438 25230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf0501919f944794bbab0ffae6a334e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.831739 25227 tablet_service.cc:1511] Processing CreateTablet for tablet 56914a99b2d8417592e1f08979cfadf6 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:07:19.831881 25303 tablet_service.cc:1511] Processing CreateTablet for tablet 1b8652aecc6d4869bb4cad9e0749f3ba (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:07:19.832432 25378 tablet_service.cc:1511] Processing CreateTablet for tablet 5ead2e92f0614db79c9249687c4246f9 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:07:19.838984 25385 tablet_service.cc:1511] Processing CreateTablet for tablet 1b8652aecc6d4869bb4cad9e0749f3ba (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:07:19.841555 25302 tablet_service.cc:1511] Processing CreateTablet for tablet 56914a99b2d8417592e1f08979cfadf6 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:07:19.843967 25382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f37bfaeb12446c1a84af172a6f3dde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.844594 25384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46ab4fd91d3f4766a7b97848f5244360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.845214 25377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17584a19690d4010851d65378cfcbafa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.847321 25378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ead2e92f0614db79c9249687c4246f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.850118 25436 raft_consensus.cc:359] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.833197 25310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17584a19690d4010851d65378cfcbafa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.867290 25435 tablet_bootstrap.cc:492] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:19.867931 25435 ts_tablet_manager.cc:1403] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.298s	user 0.017s	sys 0.004s
I20260709 13:07:19.857712 25436 raft_consensus.cc:385] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.874655 25383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adf2f61a8d304d4eb9c30a77d7de17e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.874894 25227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56914a99b2d8417592e1f08979cfadf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.883400 25308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf0501919f944794bbab0ffae6a334e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.873827 25436 raft_consensus.cc:740] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.896461 25381 tablet_service.cc:1511] Processing CreateTablet for tablet e5cbaabf54304de089abde51df6dc2be (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:07:19.906908 25236 tablet_service.cc:1511] Processing CreateTablet for tablet e5cbaabf54304de089abde51df6dc2be (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:07:19.908282 25236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5cbaabf54304de089abde51df6dc2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.914194 25235 tablet_service.cc:1511] Processing CreateTablet for tablet 2f9c4bd911cc450aa35a16a5d10bf16c (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:07:19.893553 25309 tablet_service.cc:1511] Processing CreateTablet for tablet 46ab4fd91d3f4766a7b97848f5244360 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:07:19.915490 25235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f9c4bd911cc450aa35a16a5d10bf16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.931375 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:19.932955 25358 leader_election.cc:343] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:19.935606 25307 tablet_service.cc:1511] Processing CreateTablet for tablet e5cbaabf54304de089abde51df6dc2be (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:07:19.938591 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:19.938714 25435 raft_consensus.cc:359] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.939471 25435 raft_consensus.cc:385] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.939209 25304 tablet_service.cc:1511] Processing CreateTablet for tablet 2f9c4bd911cc450aa35a16a5d10bf16c (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:07:19.939802 25435 raft_consensus.cc:740] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
W20260709 13:07:19.940315 25359 leader_election.cc:343] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:19.940773 25359 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:19.940660 25435 consensus_queue.cc:260] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.941880 25447 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:19.951439 25437 tablet_bootstrap.cc:492] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:19.952028 25437 ts_tablet_manager.cc:1403] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.240s	user 0.026s	sys 0.015s
I20260709 13:07:19.955231 25437 raft_consensus.cc:359] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.960868 25437 raft_consensus.cc:385] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.961279 25437 raft_consensus.cc:740] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.961804 25379 tablet_service.cc:1511] Processing CreateTablet for tablet 56914a99b2d8417592e1f08979cfadf6 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:07:19.962220 25437 consensus_queue.cc:260] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.966830 25437 ts_tablet_manager.cc:1434] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 13:07:19.967911 25437 tablet_bootstrap.cc:492] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:19.964293 25234 tablet_service.cc:1511] Processing CreateTablet for tablet 98122796f52e40968d1d8ccd7b032b32 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:07:19.945035 25306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2fe177353b4d6c9de4ccf872ca06ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.973589 25303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b8652aecc6d4869bb4cad9e0749f3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.973866 25234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98122796f52e40968d1d8ccd7b032b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.943857 25385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b8652aecc6d4869bb4cad9e0749f3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.978096 25235 tablet_service.cc:1511] Processing CreateTablet for tablet 72001570ee2c41cb88f0e9eac74992c7 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:07:19.983168 25235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72001570ee2c41cb88f0e9eac74992c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.981496 25379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56914a99b2d8417592e1f08979cfadf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.981225 25436 consensus_queue.cc:260] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:19.970785 25435 ts_tablet_manager.cc:1434] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.102s	user 0.008s	sys 0.001s
I20260709 13:07:19.974331 25304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f9c4bd911cc450aa35a16a5d10bf16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.993875 25381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5cbaabf54304de089abde51df6dc2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.998798 25436 ts_tablet_manager.cc:1434] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.175s	user 0.007s	sys 0.002s
I20260709 13:07:19.999478 25380 tablet_service.cc:1511] Processing CreateTablet for tablet 2f9c4bd911cc450aa35a16a5d10bf16c (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:07:20.000102 25436 tablet_bootstrap.cc:492] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:20.000744 25380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f9c4bd911cc450aa35a16a5d10bf16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.001282 25302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56914a99b2d8417592e1f08979cfadf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.003233 25435 tablet_bootstrap.cc:492] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:20.012425 25305 tablet_service.cc:1511] Processing CreateTablet for tablet 72001570ee2c41cb88f0e9eac74992c7 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:07:20.021035 25311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f37bfaeb12446c1a84af172a6f3dde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.029275 25383 tablet_service.cc:1511] Processing CreateTablet for tablet 72001570ee2c41cb88f0e9eac74992c7 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:07:20.029558 25235 tablet_service.cc:1511] Processing CreateTablet for tablet 37bfef8566b143cf94971f26f417678c (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:07:20.031471 25383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72001570ee2c41cb88f0e9eac74992c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.033011 25453 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:20.034505 25235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37bfef8566b143cf94971f26f417678c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.038975 25435 tablet_bootstrap.cc:654] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.043321 25382 tablet_service.cc:1511] Processing CreateTablet for tablet 37bfef8566b143cf94971f26f417678c (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:07:20.046157 25382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37bfef8566b143cf94971f26f417678c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.044840 25229 tablet_service.cc:1511] Processing CreateTablet for tablet 993e3b05a73a4403b04c3b55a5b48abe (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:07:20.073845 25229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 993e3b05a73a4403b04c3b55a5b48abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.044898 25385 tablet_service.cc:1511] Processing CreateTablet for tablet 993e3b05a73a4403b04c3b55a5b48abe (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:07:20.080387 25385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 993e3b05a73a4403b04c3b55a5b48abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.084532 25377 tablet_service.cc:1511] Processing CreateTablet for tablet b6d19448306e4052a13c08a582f4ccfa (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:07:20.086040 25377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d19448306e4052a13c08a582f4ccfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.046213 25231 tablet_service.cc:1511] Processing CreateTablet for tablet b6d19448306e4052a13c08a582f4ccfa (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:07:20.091766 25231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d19448306e4052a13c08a582f4ccfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.091797 25378 tablet_service.cc:1511] Processing CreateTablet for tablet 98122796f52e40968d1d8ccd7b032b32 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:07:20.053247 25309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46ab4fd91d3f4766a7b97848f5244360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.053650 25453 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:20.106333 25307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5cbaabf54304de089abde51df6dc2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.109460 25305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72001570ee2c41cb88f0e9eac74992c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.113204 25308 tablet_service.cc:1511] Processing CreateTablet for tablet 993e3b05a73a4403b04c3b55a5b48abe (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:07:20.114483 25308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 993e3b05a73a4403b04c3b55a5b48abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.058441 25310 tablet_service.cc:1511] Processing CreateTablet for tablet 37bfef8566b143cf94971f26f417678c (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:07:20.115530 25310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37bfef8566b143cf94971f26f417678c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.115931 25437 tablet_bootstrap.cc:654] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.118788 25227 tablet_service.cc:1511] Processing CreateTablet for tablet 9c23d7a710c04baf86ab9ba1a2ddc9cf (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:07:20.120326 25227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c23d7a710c04baf86ab9ba1a2ddc9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.129796 25378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98122796f52e40968d1d8ccd7b032b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.149982 25436 tablet_bootstrap.cc:654] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.157892 25384 tablet_service.cc:1511] Processing CreateTablet for tablet 9c23d7a710c04baf86ab9ba1a2ddc9cf (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:07:20.159211 25384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c23d7a710c04baf86ab9ba1a2ddc9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.165983 25435 tablet_bootstrap.cc:492] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:20.166500 25435 ts_tablet_manager.cc:1403] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.164s	user 0.014s	sys 0.000s
I20260709 13:07:20.167274 25437 tablet_bootstrap.cc:492] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:20.167968 25437 ts_tablet_manager.cc:1403] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.200s	user 0.010s	sys 0.009s
I20260709 13:07:20.170475 25306 tablet_service.cc:1511] Processing CreateTablet for tablet b6d19448306e4052a13c08a582f4ccfa (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:07:20.171729 25306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d19448306e4052a13c08a582f4ccfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.181226 25437 raft_consensus.cc:359] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.182301 25437 raft_consensus.cc:385] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.182749 25437 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.183918 25437 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.186504 25303 tablet_service.cc:1511] Processing CreateTablet for tablet 98122796f52e40968d1d8ccd7b032b32 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:07:20.187763 25303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98122796f52e40968d1d8ccd7b032b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.187813 25456 consensus_queue.cc:1048] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:07:20.191648 25287 consensus_peers.cc:597] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 -> Peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Couldn't send request to peer a2a77ecf2c0941618b5bc79d4c04db61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:20.198856 25383 tablet_service.cc:1511] Processing CreateTablet for tablet d45c0f8b003848989966aa931c6a7c11 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:07:20.214972 25383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d45c0f8b003848989966aa931c6a7c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.215272 25234 tablet_service.cc:1511] Processing CreateTablet for tablet b2cfa29915b34839a60321576f9a67d6 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:07:20.216514 25234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2cfa29915b34839a60321576f9a67d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.224335 25437 ts_tablet_manager.cc:1434] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.056s	user 0.007s	sys 0.000s
I20260709 13:07:20.225353 25437 tablet_bootstrap.cc:492] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:20.205873 25435 raft_consensus.cc:359] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.229701 25435 raft_consensus.cc:385] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.230002 25435 raft_consensus.cc:740] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.230752 25435 consensus_queue.cc:260] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.232843 25435 ts_tablet_manager.cc:1434] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.066s	user 0.005s	sys 0.000s
I20260709 13:07:20.235443 25302 tablet_service.cc:1511] Processing CreateTablet for tablet 9c23d7a710c04baf86ab9ba1a2ddc9cf (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:07:20.236732 25302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c23d7a710c04baf86ab9ba1a2ddc9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.237749 25236 tablet_service.cc:1511] Processing CreateTablet for tablet d45c0f8b003848989966aa931c6a7c11 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:07:20.252374 25311 tablet_service.cc:1511] Processing CreateTablet for tablet d45c0f8b003848989966aa931c6a7c11 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:07:20.253989 25311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d45c0f8b003848989966aa931c6a7c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.262363 25236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d45c0f8b003848989966aa931c6a7c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.281332 25304 tablet_service.cc:1511] Processing CreateTablet for tablet b2cfa29915b34839a60321576f9a67d6 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:07:20.283694 25229 tablet_service.cc:1511] Processing CreateTablet for tablet d4d43163cfdd4a4094e92fe68bc9259a (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:07:20.283368 25143 catalog_manager.cc:5697] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:20.285917 25229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4d43163cfdd4a4094e92fe68bc9259a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.289150 25385 tablet_service.cc:1511] Processing CreateTablet for tablet b2cfa29915b34839a60321576f9a67d6 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:07:20.290462 25385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2cfa29915b34839a60321576f9a67d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.291936 25310 tablet_service.cc:1511] Processing CreateTablet for tablet d4d43163cfdd4a4094e92fe68bc9259a (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:07:20.296463 25381 tablet_service.cc:1511] Processing CreateTablet for tablet d4d43163cfdd4a4094e92fe68bc9259a (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:07:20.297758 25381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4d43163cfdd4a4094e92fe68bc9259a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.298389 25234 tablet_service.cc:1511] Processing CreateTablet for tablet f406a4d41bb9441bba0a83c1d6a56aa3 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:07:20.299455 25234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f406a4d41bb9441bba0a83c1d6a56aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.300417 25382 tablet_service.cc:1511] Processing CreateTablet for tablet f406a4d41bb9441bba0a83c1d6a56aa3 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:07:20.301651 25382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f406a4d41bb9441bba0a83c1d6a56aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.304160 25305 tablet_service.cc:1511] Processing CreateTablet for tablet f406a4d41bb9441bba0a83c1d6a56aa3 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:07:20.305284 25435 tablet_bootstrap.cc:492] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:20.307444 25304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2cfa29915b34839a60321576f9a67d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.309437 25441 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.309891 25441 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.322810 25305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f406a4d41bb9441bba0a83c1d6a56aa3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:07:20.318866 25192 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:07:20.338063 25380 tablet_service.cc:1511] Processing CreateTablet for tablet 94b9d971fbc94215a65a2247f2e2acb1 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:07:20.339401 25380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b9d971fbc94215a65a2247f2e2acb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.349313 25308 tablet_service.cc:1511] Processing CreateTablet for tablet 94b9d971fbc94215a65a2247f2e2acb1 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:07:20.356397 25230 tablet_service.cc:1511] Processing CreateTablet for tablet 94b9d971fbc94215a65a2247f2e2acb1 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:07:20.358930 25230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b9d971fbc94215a65a2247f2e2acb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.361425 25441 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:20.362463 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:20.362920 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:20.370601 25437 tablet_bootstrap.cc:700] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent opening tablet: real 0.106s	user 0.002s	sys 0.000s
I20260709 13:07:20.371402 25437 tablet_bootstrap.cc:654] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
W20260709 13:07:20.373172 25287 leader_election.cc:343] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:20.366271 25246 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:20.372789 25234 tablet_service.cc:1511] Processing CreateTablet for tablet 898e00cfe03c4fc68dc9d6e92a55e996 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:07:20.365597 25310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4d43163cfdd4a4094e92fe68bc9259a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.375254 25234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898e00cfe03c4fc68dc9d6e92a55e996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.395850 25381 tablet_service.cc:1511] Processing CreateTablet for tablet 898e00cfe03c4fc68dc9d6e92a55e996 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:07:20.402360 25381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898e00cfe03c4fc68dc9d6e92a55e996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.377337 25435 tablet_bootstrap.cc:654] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.403165 25227 tablet_service.cc:1511] Processing CreateTablet for tablet 7555c0d2bb294af19f602573931ceb90 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:07:20.404394 25227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7555c0d2bb294af19f602573931ceb90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.380649 25436 tablet_bootstrap.cc:492] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:20.376446 25285 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:20.409900 25456 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:20.410254 25456 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:20.410576 25456 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.411768 25307 tablet_service.cc:1511] Processing CreateTablet for tablet 898e00cfe03c4fc68dc9d6e92a55e996 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:07:20.406332 25453 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.413463 25453 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.413914 25306 tablet_service.cc:1511] Processing CreateTablet for tablet 7555c0d2bb294af19f602573931ceb90 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:07:20.414744 25235 tablet_service.cc:1511] Processing CreateTablet for tablet b7464c8ac67847019342ea6c88895f77 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:07:20.415047 25303 tablet_service.cc:1511] Processing CreateTablet for tablet b7464c8ac67847019342ea6c88895f77 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:07:20.415549 25453 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:20.415831 25235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7464c8ac67847019342ea6c88895f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.419512 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:20.420840 25378 tablet_service.cc:1511] Processing CreateTablet for tablet b7464c8ac67847019342ea6c88895f77 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
W20260709 13:07:20.421420 25209 leader_election.cc:343] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:20.422143 25378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7464c8ac67847019342ea6c88895f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.422282 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:20.422919 25394 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:20.424089 25212 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:20.421283 25308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b9d971fbc94215a65a2247f2e2acb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.425184 25453 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:20.425555 25453 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:20.425895 25453 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.426062 25307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898e00cfe03c4fc68dc9d6e92a55e996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.427114 25306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7555c0d2bb294af19f602573931ceb90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.439498 25377 tablet_service.cc:1511] Processing CreateTablet for tablet 7555c0d2bb294af19f602573931ceb90 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:07:20.429119 25436 ts_tablet_manager.cc:1403] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.429s	user 0.033s	sys 0.017s
I20260709 13:07:20.444957 25303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7464c8ac67847019342ea6c88895f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.447377 25309 tablet_service.cc:1511] Processing CreateTablet for tablet 63c3ef75ef234051b7ae1ea395400f54 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:07:20.448659 25309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63c3ef75ef234051b7ae1ea395400f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.448577 25384 tablet_service.cc:1511] Processing CreateTablet for tablet 63c3ef75ef234051b7ae1ea395400f54 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:07:20.450067 25384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63c3ef75ef234051b7ae1ea395400f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.452665 25377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7555c0d2bb294af19f602573931ceb90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.473935 25453 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.475795 25453 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:20.477093 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:20.477631 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:20.478163 25394 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.483067 25456 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.483791 25394 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:20.485018 25212 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:20.485906 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:20.486483 25246 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.487162 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:07:20.488552 25287 leader_election.cc:343] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:20.486105 25453 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:20.489585 25456 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:20.490305 25453 raft_consensus.cc:697] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:20.491235 25453 consensus_queue.cc:237] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.494370 25246 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:20.495846 25285 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:20.501060 25229 tablet_service.cc:1511] Processing CreateTablet for tablet 63c3ef75ef234051b7ae1ea395400f54 (DEFAULT_TABLE table=test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:07:20.502931 25229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63c3ef75ef234051b7ae1ea395400f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:20.505719 25437 tablet_bootstrap.cc:492] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:20.506282 25437 ts_tablet_manager.cc:1403] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.281s	user 0.009s	sys 0.004s
I20260709 13:07:20.526482 25447 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:07:20.542315 25209 leader_election.cc:343] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:20.532986 25447 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.560397 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:07:20.561461 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 65747.7us
I20260709 13:07:20.561735 25456 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 13:07:20.561977 25359 leader_election.cc:343] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:20.562278 25456 raft_consensus.cc:697] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:20.563110 25456 consensus_queue.cc:237] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.565382 25437 raft_consensus.cc:359] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.566191 25437 raft_consensus.cc:385] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.566500 25437 raft_consensus.cc:740] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.566875 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:20.567534 25435 tablet_bootstrap.cc:492] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
W20260709 13:07:20.568804 25358 leader_election.cc:343] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:20.572250 25435 ts_tablet_manager.cc:1403] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.293s	user 0.022s	sys 0.026s
I20260709 13:07:20.569290 25358 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:20.567324 25437 consensus_queue.cc:260] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.590296 25484 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:20.568310 25447 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:20.599077 25437 ts_tablet_manager.cc:1434] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.092s	user 0.011s	sys 0.020s
I20260709 13:07:20.605252 25437 tablet_bootstrap.cc:492] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:20.605800 25469 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.606261 25469 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.608120 25469 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:20.609349 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:20.611645 25437 tablet_bootstrap.cc:654] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.612237 25143 catalog_manager.cc:5697] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:20.616750 25142 catalog_manager.cc:5697] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
W20260709 13:07:20.620088 25209 leader_election.cc:343] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:20.614193 25435 raft_consensus.cc:359] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.625802 25435 raft_consensus.cc:385] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.626096 25435 raft_consensus.cc:740] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.626703 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:20.626803 25435 consensus_queue.cc:260] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.627424 25394 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:20.632726 25212 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:20.633838 25469 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:20.633584 25436 raft_consensus.cc:359] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.634191 25469 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:20.634313 25436 raft_consensus.cc:385] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.634562 25469 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.634634 25436 raft_consensus.cc:740] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.641099 25469 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.645546 25435 ts_tablet_manager.cc:1434] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.073s	user 0.006s	sys 0.001s
I20260709 13:07:20.647701 25469 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:20.649295 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:20.650736 25437 tablet_bootstrap.cc:492] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
W20260709 13:07:20.651091 25209 leader_election.cc:343] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:20.651233 25437 ts_tablet_manager.cc:1403] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.046s	user 0.018s	sys 0.010s
I20260709 13:07:20.652279 25435 tablet_bootstrap.cc:492] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:20.652185 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:20.652904 25394 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.654247 25437 raft_consensus.cc:359] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.654937 25437 raft_consensus.cc:385] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.655224 25437 raft_consensus.cc:740] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.655941 25437 consensus_queue.cc:260] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.657344 25436 consensus_queue.cc:260] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.658185 25437 ts_tablet_manager.cc:1434] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:07:20.659196 25437 tablet_bootstrap.cc:492] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:20.659569 25436 ts_tablet_manager.cc:1434] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.216s	user 0.002s	sys 0.006s
I20260709 13:07:20.661844 25394 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:20.663317 25212 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:20.664572 25469 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:20.665056 25469 raft_consensus.cc:697] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:20.666253 25437 tablet_bootstrap.cc:654] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.665966 25469 consensus_queue.cc:237] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.668255 25447 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.668735 25447 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.671195 25447 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:20.673542 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:07:20.675022 25359 leader_election.cc:343] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:20.679139 25436 tablet_bootstrap.cc:492] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:20.679211 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:20.683137 25358 leader_election.cc:343] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:20.683656 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:20.685361 25447 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:20.689621 25436 tablet_bootstrap.cc:654] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.692744 25435 tablet_bootstrap.cc:654] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.697754 25143 catalog_manager.cc:5697] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:20.709410 25435 tablet_bootstrap.cc:492] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:20.709332 25437 tablet_bootstrap.cc:492] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:20.712157 25435 ts_tablet_manager.cc:1403] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.060s	user 0.011s	sys 0.002s
I20260709 13:07:20.712273 25437 ts_tablet_manager.cc:1403] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.053s	user 0.028s	sys 0.019s
I20260709 13:07:20.713285 25436 tablet_bootstrap.cc:492] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:20.713778 25436 ts_tablet_manager.cc:1403] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.035s	user 0.024s	sys 0.004s
I20260709 13:07:20.714859 25435 raft_consensus.cc:359] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.715404 25435 raft_consensus.cc:385] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.715693 25435 raft_consensus.cc:740] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.715487 25437 raft_consensus.cc:359] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.716315 25437 raft_consensus.cc:385] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.716370 25435 consensus_queue.cc:260] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.716785 25437 raft_consensus.cc:740] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.716760 25436 raft_consensus.cc:359] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.718007 25436 raft_consensus.cc:385] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.717864 25437 consensus_queue.cc:260] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.718348 25436 raft_consensus.cc:740] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.718511 25435 ts_tablet_manager.cc:1434] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:07:20.719169 25436 consensus_queue.cc:260] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.720705 25435 tablet_bootstrap.cc:492] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:20.721820 25436 ts_tablet_manager.cc:1434] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:07:20.725415 25437 ts_tablet_manager.cc:1434] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.004s
I20260709 13:07:20.725874 25436 tablet_bootstrap.cc:492] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:20.726444 25437 tablet_bootstrap.cc:492] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:20.727535 25435 tablet_bootstrap.cc:654] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.729959 25447 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.730453 25447 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.732760 25447 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:20.734241 25483 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.734321 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:20.734745 25483 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.733085 25436 tablet_bootstrap.cc:654] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.736809 25483 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:20.739559 25437 tablet_bootstrap.cc:654] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.743415 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:20.734764 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:20.745308 25358 leader_election.cc:343] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:20.747346 25358 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:20.747999 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:20.748780 25246 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:20.750402 25359 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:20.752671 25447 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:20.754212 25447 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:20.754711 25447 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:20.755187 25447 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.758816 25435 tablet_bootstrap.cc:492] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:20.759487 25435 ts_tablet_manager.cc:1403] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.039s	user 0.014s	sys 0.004s
I20260709 13:07:20.763438 25435 raft_consensus.cc:359] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.765576 25435 raft_consensus.cc:385] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.765898 25435 raft_consensus.cc:740] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.766625 25435 consensus_queue.cc:260] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.767812 25437 tablet_bootstrap.cc:492] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:20.768486 25437 ts_tablet_manager.cc:1403] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.042s	user 0.017s	sys 0.015s
I20260709 13:07:20.768262 25447 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.768769 25435 ts_tablet_manager.cc:1434] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.000s
I20260709 13:07:20.769735 25435 tablet_bootstrap.cc:492] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:20.771153 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:20.771353 25437 raft_consensus.cc:359] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.771759 25246 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.772002 25437 raft_consensus.cc:385] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.772365 25437 raft_consensus.cc:740] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.773010 25437 consensus_queue.cc:260] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.774916 25437 ts_tablet_manager.cc:1434] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 13:07:20.775660 25435 tablet_bootstrap.cc:654] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.775707 25437 tablet_bootstrap.cc:492] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:20.782444 25436 tablet_bootstrap.cc:492] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:20.782593 25456 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.782927 25436 ts_tablet_manager.cc:1403] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.057s	user 0.026s	sys 0.011s
I20260709 13:07:20.783083 25456 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.785346 25456 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:20.785825 25436 raft_consensus.cc:359] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.786586 25436 raft_consensus.cc:385] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.786878 25436 raft_consensus.cc:740] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.787559 25436 consensus_queue.cc:260] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.788347 25437 tablet_bootstrap.cc:654] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.788410 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:20.789768 25436 ts_tablet_manager.cc:1434] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:07:20.790695 25436 tablet_bootstrap.cc:492] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:20.796154 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:20.796984 25394 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 1: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:07:20.798456 25287 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:20.798911 25436 tablet_bootstrap.cc:654] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.799152 25246 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:20.800470 25359 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:20.802832 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:20.803478 25320 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.804193 25483 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:20.804466 25435 tablet_bootstrap.cc:492] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:20.804741 25483 raft_consensus.cc:697] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:20.805047 25435 ts_tablet_manager.cc:1403] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.035s	user 0.010s	sys 0.009s
I20260709 13:07:20.805718 25483 consensus_queue.cc:237] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.808388 25435 raft_consensus.cc:359] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.809285 25435 raft_consensus.cc:385] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.809609 25435 raft_consensus.cc:740] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.810573 25435 consensus_queue.cc:260] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.811556 25456 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.813297 25435 ts_tablet_manager.cc:1434] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:07:20.814904 25320 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:20.816258 25447 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:20.818167 25435 tablet_bootstrap.cc:492] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:20.824982 25435 tablet_bootstrap.cc:654] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.843696 25141 catalog_manager.cc:5697] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:20.844805 25437 tablet_bootstrap.cc:492] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:20.845320 25437 ts_tablet_manager.cc:1403] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.070s	user 0.016s	sys 0.027s
I20260709 13:07:20.845803 25435 tablet_bootstrap.cc:492] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:20.846437 25435 ts_tablet_manager.cc:1403] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.028s	user 0.020s	sys 0.004s
I20260709 13:07:20.848044 25437 raft_consensus.cc:359] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.848752 25437 raft_consensus.cc:385] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.849038 25437 raft_consensus.cc:740] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.849764 25435 raft_consensus.cc:359] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.849799 25437 consensus_queue.cc:260] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.850567 25435 raft_consensus.cc:385] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.850972 25435 raft_consensus.cc:740] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.851891 25456 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:20.851646 25435 consensus_queue.cc:260] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.852349 25437 ts_tablet_manager.cc:1434] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:07:20.853437 25437 tablet_bootstrap.cc:492] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:20.854002 25435 ts_tablet_manager.cc:1434] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:07:20.856231 25436 tablet_bootstrap.cc:492] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:20.857568 25436 ts_tablet_manager.cc:1403] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.067s	user 0.026s	sys 0.022s
I20260709 13:07:20.859354 25437 tablet_bootstrap.cc:654] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.862108 25436 raft_consensus.cc:359] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.862912 25436 raft_consensus.cc:385] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.863225 25436 raft_consensus.cc:740] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.864068 25436 consensus_queue.cc:260] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.865688 25435 tablet_bootstrap.cc:492] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:20.865967 25437 tablet_bootstrap.cc:492] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:20.866393 25437 ts_tablet_manager.cc:1403] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.013s	user 0.011s	sys 0.000s
I20260709 13:07:20.867801 25436 ts_tablet_manager.cc:1434] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20260709 13:07:20.869029 25437 raft_consensus.cc:359] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.869796 25437 raft_consensus.cc:385] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.870071 25437 raft_consensus.cc:740] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.871198 25436 tablet_bootstrap.cc:492] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:20.870879 25437 consensus_queue.cc:260] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.872104 25435 tablet_bootstrap.cc:654] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.875568 25469 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.876243 25469 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.878851 25437 ts_tablet_manager.cc:1434] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 13:07:20.879777 25437 tablet_bootstrap.cc:492] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:20.880635 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:20.881572 25394 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:20.883850 25212 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:20.885311 25437 tablet_bootstrap.cc:654] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.885087 25453 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:20.885674 25453 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:20.886162 25453 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.889938 25435 tablet_bootstrap.cc:492] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:20.890273 25437 tablet_bootstrap.cc:492] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:20.890477 25435 ts_tablet_manager.cc:1403] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.025s	user 0.005s	sys 0.018s
I20260709 13:07:20.890038 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:20.890713 25437 ts_tablet_manager.cc:1403] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.005s
I20260709 13:07:20.891098 25320 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:20.876904 25436 tablet_bootstrap.cc:654] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.893486 25435 raft_consensus.cc:359] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.894230 25435 raft_consensus.cc:385] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.894526 25435 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.890559 25469 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:20.893761 25437 raft_consensus.cc:359] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.895236 25435 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.895769 25437 raft_consensus.cc:385] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.896242 25437 raft_consensus.cc:740] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.896886 25437 consensus_queue.cc:260] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.897360 25435 ts_tablet_manager.cc:1434] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 13:07:20.898798 25435 tablet_bootstrap.cc:492] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:20.898940 25437 ts_tablet_manager.cc:1434] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 13:07:20.899911 25437 tablet_bootstrap.cc:492] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:20.904778 25435 tablet_bootstrap.cc:654] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.905454 25437 tablet_bootstrap.cc:654] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.916324 25453 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.920189 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:20.920058 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:20.920861 25394 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.921060 25320 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.925277 25453 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:20.925683 25436 tablet_bootstrap.cc:492] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:20.926102 25436 ts_tablet_manager.cc:1403] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.055s	user 0.005s	sys 0.009s
I20260709 13:07:20.928383 25394 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:20.929106 25436 raft_consensus.cc:359] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.913002 25469 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260709 13:07:20.929754 25436 raft_consensus.cc:385] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.930028 25436 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.931164 25212 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:20.932003 25469 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:20.932189 25320 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:20.932518 25469 raft_consensus.cc:697] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:20.937798 25436 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.940932 25436 ts_tablet_manager.cc:1434] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.000s
I20260709 13:07:20.942253 25436 tablet_bootstrap.cc:492] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:20.950472 25436 tablet_bootstrap.cc:654] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.952600 25456 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.953245 25456 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.958315 25456 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:20.958671 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:20.959095 25456 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.959609 25246 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:20.961458 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:20.961697 25285 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:20.962169 25394 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:20.962913 25478 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:20.963439 25478 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:20.963887 25478 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.959671 25456 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.970844 25453 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.971101 25456 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:20.970963 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:20.971328 25453 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.972126 25246 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:20.973150 25469 consensus_queue.cc:237] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.973641 25453 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:20.974196 25285 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:20.971028 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:20.981792 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:20.982091 25394 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:20.982390 25319 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:20.983651 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:20.984025 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:07:20.986981 25212 leader_election.cc:343] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:20.988133 25436 tablet_bootstrap.cc:492] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:20.988706 25436 ts_tablet_manager.cc:1403] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.047s	user 0.013s	sys 0.003s
I20260709 13:07:20.987766 25478 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:20.988533 25456 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:20.998018 25478 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:20.997208 25437 tablet_bootstrap.cc:492] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:20.998402 25435 tablet_bootstrap.cc:492] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:20.995301 25456 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:20.999023 25435 ts_tablet_manager.cc:1403] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.100s	user 0.055s	sys 0.031s
I20260709 13:07:20.998963 25437 ts_tablet_manager.cc:1403] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.099s	user 0.035s	sys 0.014s
I20260709 13:07:20.999400 25456 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.999886 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:21.001266 25395 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.001538 25436 raft_consensus.cc:359] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.003618 25436 raft_consensus.cc:385] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.004034 25436 raft_consensus.cc:740] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.010800 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:21.011544 25503 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:21.011654 25246 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.013113 25503 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:21.015118 25503 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.053272 25394 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.053737 25395 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:21.058066 25287 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:21.061532 25246 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:21.082306 25453 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:21.084308 25478 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:21.085644 25395 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.088229 25508 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:21.088775 25508 raft_consensus.cc:697] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:21.089691 25508 consensus_queue.cc:237] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.085522 25435 raft_consensus.cc:359] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.086624 25436 consensus_queue.cc:260] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.093375 25435 raft_consensus.cc:385] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.093690 25435 raft_consensus.cc:740] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.094373 25435 consensus_queue.cc:260] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.094969 25436 ts_tablet_manager.cc:1434] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.106s	user 0.034s	sys 0.019s
I20260709 13:07:21.096117 25436 tablet_bootstrap.cc:492] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:21.086134 25143 catalog_manager.cc:5697] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:21.098984 25435 ts_tablet_manager.cc:1434] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.100s	user 0.007s	sys 0.000s
I20260709 13:07:21.095939 25453 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.100338 25437 raft_consensus.cc:359] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.100971 25437 raft_consensus.cc:385] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.101279 25437 raft_consensus.cc:740] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.103663 25435 tablet_bootstrap.cc:492] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:21.101915 25437 consensus_queue.cc:260] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.103114 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:21.105389 25453 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:21.106611 25392 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:21.129664 25437 ts_tablet_manager.cc:1434] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.130s	user 0.009s	sys 0.019s
I20260709 13:07:21.131374 25437 tablet_bootstrap.cc:492] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:21.130213 25483 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:21.133628 25483 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.113809 25510 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20260709 13:07:21.136595 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:21.135329 25456 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.137609 25246 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:21.135864 25483 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:21.147359 25359 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:21.129027 25212 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:21.150604 25447 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:07:21.157634 25503 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.152407 25447 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:21.158478 25447 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.158735 25437 tablet_bootstrap.cc:654] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.159668 25503 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:21.158952 25514 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:21.160152 25514 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:21.162882 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:21.163489 25319 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:21.164513 25436 tablet_bootstrap.cc:654] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.164367 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:21.164919 25393 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.153208 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:21.160593 25514 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.169929 25447 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:07:21.155333 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 47339us
I20260709 13:07:21.174364 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:21.174909 25246 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.176132 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:07:21.173043 25209 consensus_peers.cc:597] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:21.165604 25456 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:21.177767 25395 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.184260 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:21.185006 25320 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.187778 25246 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:07:21.188784 25395 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:21.192493 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:21.194999 25483 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:21.194731 25359 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:21.195492 25483 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.200158 25393 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:21.203958 25484 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:21.203590 25513 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:21.201246 25320 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:21.206990 25483 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:21.206851 25503 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:21.205691 25287 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:21.208006 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:07:21.190611 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 177700us
I20260709 13:07:21.210256 25478 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:21.201632 25447 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:07:21.211928 25359 leader_election.cc:343] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:21.212513 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:21.214740 25514 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.217839 25435 tablet_bootstrap.cc:654] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.221268 25478 raft_consensus.cc:697] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:21.228081 25478 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.225793 25513 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:21.231861 25513 raft_consensus.cc:697] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:21.232694 25513 consensus_queue.cc:237] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.234555 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:21.229763 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:21.234611 25437 tablet_bootstrap.cc:492] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:21.235740 25437 ts_tablet_manager.cc:1403] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.105s	user 0.020s	sys 0.026s
I20260709 13:07:21.236975 25514 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:21.235251 25246 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.245620 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:21.245934 25245 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 13:07:21.246181 25319 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.240927 25484 raft_consensus.cc:697] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:07:21.250999 25358 leader_election.cc:343] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:21.252266 25358 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:21.254891 25447 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:21.255888 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:21.255331 25141 catalog_manager.cc:5697] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:21.256414 25392 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.272833 25436 tablet_bootstrap.cc:492] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:21.273317 25436 ts_tablet_manager.cc:1403] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.178s	user 0.011s	sys 0.005s
I20260709 13:07:21.277081 25436 raft_consensus.cc:359] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.278092 25436 raft_consensus.cc:385] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.278455 25436 raft_consensus.cc:740] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.282733 25436 consensus_queue.cc:260] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.285595 25436 ts_tablet_manager.cc:1434] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.002s
I20260709 13:07:21.289160 25436 tablet_bootstrap.cc:492] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:21.272073 25478 consensus_queue.cc:1059] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260709 13:07:21.287978 25484 consensus_queue.cc:237] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.296741 25141 catalog_manager.cc:5697] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:21.299710 25508 raft_consensus.cc:3055] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:21.299592 25437 raft_consensus.cc:359] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.300103 25508 raft_consensus.cc:740] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:21.300457 25437 raft_consensus.cc:385] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.300737 25437 raft_consensus.cc:740] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.300923 25508 consensus_queue.cc:260] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.301455 25437 consensus_queue.cc:260] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.302023 25508 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.302328 25447 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:21.308027 25143 catalog_manager.cc:5697] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:21.309056 25393 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 13:07:21.309422 25437 ts_tablet_manager.cc:1434] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.073s	user 0.007s	sys 0.000s
I20260709 13:07:21.304075 25319 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:21.314044 25246 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:21.314566 25435 tablet_bootstrap.cc:492] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:21.315915 25435 ts_tablet_manager.cc:1403] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.216s	user 0.018s	sys 0.016s
I20260709 13:07:21.315379 25209 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:21.317502 25513 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:21.320571 25435 raft_consensus.cc:359] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.321280 25435 raft_consensus.cc:385] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.321569 25435 raft_consensus.cc:740] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.319176 25513 raft_consensus.cc:697] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:21.322228 25435 consensus_queue.cc:260] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.325879 25513 consensus_queue.cc:237] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.326627 25437 tablet_bootstrap.cc:492] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:21.316793 25392 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:21.330605 25435 ts_tablet_manager.cc:1434] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.014s	user 0.003s	sys 0.004s
I20260709 13:07:21.333406 25435 tablet_bootstrap.cc:492] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:21.343282 25435 tablet_bootstrap.cc:654] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.343811 25483 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:21.345386 25436 tablet_bootstrap.cc:654] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.346403 25437 tablet_bootstrap.cc:654] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.349874 25142 catalog_manager.cc:5697] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 1 to 2, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 2 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:21.370267 25513 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:21.370784 25513 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.372936 25513 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:07:21.374117 25192 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:07:21.374349 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:21.376174 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:21.391111 25209 leader_election.cc:343] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:21.390117 25142 catalog_manager.cc:5697] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
W20260709 13:07:21.392174 25212 leader_election.cc:343] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:21.392614 25212 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:21.393268 25513 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:21.395442 25437 tablet_bootstrap.cc:492] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:21.395882 25437 ts_tablet_manager.cc:1403] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.069s	user 0.015s	sys 0.027s
I20260709 13:07:21.396837 25436 tablet_bootstrap.cc:492] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:21.397392 25436 ts_tablet_manager.cc:1403] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.109s	user 0.020s	sys 0.001s
I20260709 13:07:21.398353 25437 raft_consensus.cc:359] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.398981 25437 raft_consensus.cc:385] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.399227 25437 raft_consensus.cc:740] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.403658 25437 consensus_queue.cc:260] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.409857 25437 ts_tablet_manager.cc:1434] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.001s
I20260709 13:07:21.410789 25437 tablet_bootstrap.cc:492] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:21.416631 25437 tablet_bootstrap.cc:654] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.419610 25436 raft_consensus.cc:359] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.420547 25436 raft_consensus.cc:385] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.420894 25436 raft_consensus.cc:740] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.421890 25436 consensus_queue.cc:260] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.425271 25435 tablet_bootstrap.cc:492] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:21.425722 25435 ts_tablet_manager.cc:1403] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.093s	user 0.022s	sys 0.048s
I20260709 13:07:21.426975 25436 ts_tablet_manager.cc:1434] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.000s
I20260709 13:07:21.428243 25436 tablet_bootstrap.cc:492] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:21.428261 25435 raft_consensus.cc:359] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.429147 25435 raft_consensus.cc:385] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.429430 25435 raft_consensus.cc:740] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.430054 25435 consensus_queue.cc:260] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.432281 25435 ts_tablet_manager.cc:1434] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:07:21.433169 25435 tablet_bootstrap.cc:492] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:21.435169 25437 tablet_bootstrap.cc:492] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:21.435665 25437 ts_tablet_manager.cc:1403] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.025s	user 0.014s	sys 0.010s
I20260709 13:07:21.436857 25436 tablet_bootstrap.cc:654] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.438562 25435 tablet_bootstrap.cc:654] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.438342 25437 raft_consensus.cc:359] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.439136 25437 raft_consensus.cc:385] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.439565 25437 raft_consensus.cc:740] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.440683 25437 consensus_queue.cc:260] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.449275 25435 tablet_bootstrap.cc:492] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:21.453105 25435 ts_tablet_manager.cc:1403] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.017s	user 0.008s	sys 0.007s
I20260709 13:07:21.456068 25437 ts_tablet_manager.cc:1434] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.000s
I20260709 13:07:21.455968 25435 raft_consensus.cc:359] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.456664 25435 raft_consensus.cc:385] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.456943 25435 raft_consensus.cc:740] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.457780 25435 consensus_queue.cc:260] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.459939 25435 ts_tablet_manager.cc:1434] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:07:21.463552 25435 tablet_bootstrap.cc:492] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:21.464937 25437 tablet_bootstrap.cc:492] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:21.470090 25435 tablet_bootstrap.cc:654] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.474370 25437 tablet_bootstrap.cc:654] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.475427 25435 tablet_bootstrap.cc:492] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:21.475862 25435 ts_tablet_manager.cc:1403] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.013s	user 0.005s	sys 0.003s
I20260709 13:07:21.477525 25436 tablet_bootstrap.cc:492] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:21.478361 25436 ts_tablet_manager.cc:1403] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.050s	user 0.023s	sys 0.022s
I20260709 13:07:21.478487 25435 raft_consensus.cc:359] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.479171 25435 raft_consensus.cc:385] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.479427 25435 raft_consensus.cc:740] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.480340 25435 consensus_queue.cc:260] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.480571 25436 raft_consensus.cc:359] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.481246 25436 raft_consensus.cc:385] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.481510 25436 raft_consensus.cc:740] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.482302 25435 ts_tablet_manager.cc:1434] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 13:07:21.482213 25436 consensus_queue.cc:260] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.483198 25435 tablet_bootstrap.cc:492] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:21.483561 25513 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:21.484233 25436 ts_tablet_manager.cc:1434] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:07:21.486332 25437 tablet_bootstrap.cc:492] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:21.486761 25437 ts_tablet_manager.cc:1403] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.022s	user 0.012s	sys 0.000s
I20260709 13:07:21.489243 25435 tablet_bootstrap.cc:654] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.489341 25437 raft_consensus.cc:359] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.490072 25437 raft_consensus.cc:385] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.491624 25437 raft_consensus.cc:740] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.492367 25437 consensus_queue.cc:260] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.494645 25437 ts_tablet_manager.cc:1434] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 13:07:21.491394 25436 tablet_bootstrap.cc:492] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:21.501811 25437 tablet_bootstrap.cc:492] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:21.508625 25437 tablet_bootstrap.cc:654] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.510107 25436 tablet_bootstrap.cc:654] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.518685 25437 tablet_bootstrap.cc:492] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:21.519981 25437 ts_tablet_manager.cc:1403] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.018s	user 0.008s	sys 0.004s
I20260709 13:07:21.521497 25435 tablet_bootstrap.cc:492] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:21.521914 25435 ts_tablet_manager.cc:1403] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.039s	user 0.016s	sys 0.017s
I20260709 13:07:21.522549 25437 raft_consensus.cc:359] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.523190 25437 raft_consensus.cc:385] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.523448 25437 raft_consensus.cc:740] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.524279 25437 consensus_queue.cc:260] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.526549 25437 ts_tablet_manager.cc:1434] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:07:21.527426 25437 tablet_bootstrap.cc:492] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:21.524441 25435 raft_consensus.cc:359] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.529287 25435 raft_consensus.cc:385] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.529601 25435 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.530192 25435 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.532140 25435 ts_tablet_manager.cc:1434] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.000s
I20260709 13:07:21.532969 25435 tablet_bootstrap.cc:492] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:21.540330 25513 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:21.541208 25437 tablet_bootstrap.cc:654] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.540956 25513 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.542258 25435 tablet_bootstrap.cc:654] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.543623 25513 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:21.544283 25513 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:21.551612 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:21.563163 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:21.563870 25392 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:21.568766 25212 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:21.572296 25513 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:21.572768 25513 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:07:21.573352 25209 leader_election.cc:343] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:21.578336 25513 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.599442 25508 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:21.607911 25435 tablet_bootstrap.cc:492] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:21.608543 25435 ts_tablet_manager.cc:1403] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.076s	user 0.008s	sys 0.024s
I20260709 13:07:21.609416 25437 tablet_bootstrap.cc:492] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:21.609908 25437 ts_tablet_manager.cc:1403] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.083s	user 0.011s	sys 0.053s
I20260709 13:07:21.610512 25513 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.612547 25508 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:21.613051 25508 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.613977 25436 tablet_bootstrap.cc:492] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:21.614238 25320 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.613210 25437 raft_consensus.cc:359] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.614802 25437 raft_consensus.cc:385] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.615516 25437 raft_consensus.cc:740] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.615976 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:21.615259 25436 ts_tablet_manager.cc:1403] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.124s	user 0.018s	sys 0.062s
I20260709 13:07:21.616600 25245 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:21.619967 25435 raft_consensus.cc:359] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.620772 25508 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:21.620651 25285 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:21.616261 25437 consensus_queue.cc:260] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.622445 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:21.622922 25437 ts_tablet_manager.cc:1434] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.000s
I20260709 13:07:21.623029 25392 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:21.623847 25437 tablet_bootstrap.cc:492] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:21.624758 25518 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:21.625263 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:21.625349 25518 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:21.626013 25518 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.620626 25435 raft_consensus.cc:385] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.626895 25435 raft_consensus.cc:740] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.627887 25435 consensus_queue.cc:260] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.630213 25435 ts_tablet_manager.cc:1434] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.021s	user 0.003s	sys 0.005s
I20260709 13:07:21.631634 25436 raft_consensus.cc:359] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.632256 25436 raft_consensus.cc:385] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.632496 25436 raft_consensus.cc:740] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.633093 25436 consensus_queue.cc:260] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.630928 25513 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:07:21.630744 25209 leader_election.cc:343] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:21.636027 25436 ts_tablet_manager.cc:1434] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.000s
I20260709 13:07:21.636878 25436 tablet_bootstrap.cc:492] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:21.638768 25508 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:21.639206 25508 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.642191 25435 tablet_bootstrap.cc:492] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:21.641850 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:21.644625 25395 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.639670 25518 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.649420 25513 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:21.651672 25513 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.653402 25516 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:21.653599 25513 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:21.654908 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:21.657424 25209 leader_election.cc:343] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:21.657475 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:21.658099 25245 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:21.659210 25285 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:21.659442 25508 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:21.659765 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:21.660018 25508 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:21.660475 25508 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:21.660533 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:21.661216 25394 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:21.660791 25508 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.664634 25518 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:21.668370 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:21.668926 25394 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:21.670217 25212 leader_election.cc:343] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:21.670688 25212 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:21.669572 25395 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:21.666285 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:21.671411 25513 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:21.671837 25245 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.673586 25212 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:21.674381 25513 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:21.674858 25513 raft_consensus.cc:697] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:21.678864 25514 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:21.679333 25514 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.681344 25514 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:21.681715 25394 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:21.681921 25436 tablet_bootstrap.cc:654] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.675688 25513 consensus_queue.cc:237] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.685510 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:21.682845 25287 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:21.687414 25395 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:21.687547 25518 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:21.688220 25516 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:21.688124 25518 raft_consensus.cc:697] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:21.692373 25212 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:21.692493 25518 consensus_queue.cc:237] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.693269 25514 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:21.693663 25514 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:21.694008 25514 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.695608 25437 tablet_bootstrap.cc:654] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.695673 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:21.697275 25209 leader_election.cc:343] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:21.698698 25245 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:21.703642 25484 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:21.703390 25514 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.703752 25508 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.708513 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:21.708714 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:21.712738 25395 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:21.713100 25209 leader_election.cc:343] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:21.713801 25503 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:21.714252 25503 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.714701 25435 tablet_bootstrap.cc:654] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:21.716212 25503 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:21.709184 25514 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:21.704154 25484 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.719292 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:21.719494 25484 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:21.719923 25394 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:21.721480 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:21.722114 25469 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:21.722846 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:21.723531 25212 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:21.724002 25395 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:21.727842 25514 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:21.728374 25514 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:21.728816 25514 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.730804 25212 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:21.732196 25503 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:21.733700 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:21.751875 25516 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:21.757280 25508 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:21.763094 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:21.764818 25244 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:21.757568 25358 leader_election.cc:343] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:21.769773 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:21.770540 25395 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.774370 25469 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:21.782047 25395 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:21.783588 25287 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:21.784789 25508 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:21.785271 25508 raft_consensus.cc:697] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:21.786033 25508 consensus_queue.cc:237] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.793100 25514 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.794584 25245 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 1: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:07:21.798491 25244 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:21.798699 25359 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:21.800182 25484 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:21.801973 25514 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:07:21.803567 25209 leader_election.cc:343] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:21.804214 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 66785.9us
I20260709 13:07:21.812563 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:21.814980 25320 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260709 13:07:21.815541 25320 pending_rounds.cc:85] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 0
I20260709 13:07:21.815918 25320 pending_rounds.cc:107] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 13:07:21.816290 25320 raft_consensus.cc:2889] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:07:21.818109 25483 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:21.854089 25508 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:21.840106 25503 raft_consensus.cc:697] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:21.854789 25508 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:07:21.847291 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 43935.8us
I20260709 13:07:21.877290 25246 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:07:21.893441 25508 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:21.896919 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:07:21.898391 25285 leader_election.cc:343] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:21.904903 25483 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 13:07:21.917989 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:21.921270 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:21.921890 25393 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:21.922796 25483 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:21.923239 25483 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.924307 25143 catalog_manager.cc:5697] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:21.926868 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:07:21.929141 25359 leader_election.cc:343] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:21.934458 25484 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:21.925117 25483 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:21.912864 25503 consensus_queue.cc:237] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.965376 25392 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:21.967729 25212 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:21.968576 25514 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:21.969336 25553 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:21.969839 25553 raft_consensus.cc:697] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:07:21.970145 25209 leader_election.cc:343] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:21.970686 25553 consensus_queue.cc:237] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.971385 25469 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:07:21.973384 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 62464.5us
I20260709 13:07:21.981791 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:21.980856 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:21.995304 25435 tablet_bootstrap.cc:492] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:21.995664 25435 ts_tablet_manager.cc:1403] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.365s	user 0.024s	sys 0.009s
I20260709 13:07:21.998023 25435 raft_consensus.cc:359] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:21.998710 25435 raft_consensus.cc:385] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:21.998994 25435 raft_consensus.cc:740] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:21.999486 25435 consensus_queue.cc:260] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.004611 25436 tablet_bootstrap.cc:492] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:22.005097 25436 ts_tablet_manager.cc:1403] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.368s	user 0.012s	sys 0.015s
I20260709 13:07:22.007395 25435 ts_tablet_manager.cc:1434] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 13:07:22.008343 25435 tablet_bootstrap.cc:492] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:22.011749 25437 tablet_bootstrap.cc:492] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:22.012178 25437 ts_tablet_manager.cc:1403] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.389s	user 0.035s	sys 0.056s
I20260709 13:07:22.014478 25435 tablet_bootstrap.cc:654] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.018024 25437 raft_consensus.cc:359] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.019097 25437 raft_consensus.cc:385] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.019531 25437 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.018610 25508 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:22.024114 25508 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:22.024475 25508 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.025072 25437 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.027858 25436 raft_consensus.cc:359] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.028383 25436 raft_consensus.cc:385] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.028597 25436 raft_consensus.cc:740] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.029062 25436 consensus_queue.cc:260] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:07:22.040926 25358 leader_election.cc:343] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:22.041532 25358 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:22.053771 25483 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:22.055994 25553 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.056583 25553 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.059552 25508 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.060149 25553 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.073871 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:07:22.075743 25212 leader_election.cc:343] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:22.136770 25437 ts_tablet_manager.cc:1434] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.124s	user 0.007s	sys 0.003s
I20260709 13:07:22.101236 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:22.142427 25437 tablet_bootstrap.cc:492] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
W20260709 13:07:22.120970 25209 consensus_peers.cc:597] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:22.137985 25435 tablet_bootstrap.cc:492] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:22.145594 25435 ts_tablet_manager.cc:1403] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.137s	user 0.028s	sys 0.019s
I20260709 13:07:22.147343 25508 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.147279 25555 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:22.150888 25436 ts_tablet_manager.cc:1434] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.145s	user 0.017s	sys 0.002s
I20260709 13:07:22.151777 25436 tablet_bootstrap.cc:492] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:22.152551 25435 raft_consensus.cc:359] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.153409 25435 raft_consensus.cc:385] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.153725 25435 raft_consensus.cc:740] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.154462 25435 consensus_queue.cc:260] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.155025 25555 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:22.157495 25435 ts_tablet_manager.cc:1434] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.001s
I20260709 13:07:22.158943 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:22.162834 25246 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:22.163529 25209 leader_election.cc:343] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:22.164026 25209 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:22.163714 25435 tablet_bootstrap.cc:492] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:22.164923 25503 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:22.170075 25436 tablet_bootstrap.cc:654] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.177563 25435 tablet_bootstrap.cc:654] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.175876 25141 catalog_manager.cc:5697] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:22.183230 25437 tablet_bootstrap.cc:654] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.191135 25246 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:22.193979 25285 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:22.194783 25555 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:22.195828 25555 raft_consensus.cc:697] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:22.196645 25555 consensus_queue.cc:237] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.204872 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:22.207693 25395 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.212100 25436 tablet_bootstrap.cc:492] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:22.212571 25436 ts_tablet_manager.cc:1403] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.061s	user 0.015s	sys 0.016s
I20260709 13:07:22.214237 25395 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:22.235659 25436 raft_consensus.cc:359] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.236340 25436 raft_consensus.cc:385] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.236586 25436 raft_consensus.cc:740] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.236897 25437 tablet_bootstrap.cc:492] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:22.237340 25437 ts_tablet_manager.cc:1403] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.095s	user 0.017s	sys 0.000s
I20260709 13:07:22.237193 25436 consensus_queue.cc:260] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.241150 25518 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.241662 25518 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.242645 25436 ts_tablet_manager.cc:1434] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.028s	user 0.002s	sys 0.003s
I20260709 13:07:22.243645 25436 tablet_bootstrap.cc:492] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:22.242967 25143 catalog_manager.cc:5697] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:22.248524 25143 catalog_manager.cc:5697] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:22.250402 25435 tablet_bootstrap.cc:492] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:22.250967 25435 ts_tablet_manager.cc:1403] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.087s	user 0.016s	sys 0.001s
I20260709 13:07:22.256639 25437 raft_consensus.cc:359] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.259766 25437 raft_consensus.cc:385] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.260160 25437 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.260918 25437 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.261054 25435 raft_consensus.cc:359] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.261758 25435 raft_consensus.cc:385] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.262151 25435 raft_consensus.cc:740] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.262795 25437 ts_tablet_manager.cc:1434] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.000s
I20260709 13:07:22.263545 25437 tablet_bootstrap.cc:492] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:22.264931 25436 tablet_bootstrap.cc:654] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.267416 25437 tablet_bootstrap.cc:654] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.262758 25435 consensus_queue.cc:260] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.270061 25435 ts_tablet_manager.cc:1434] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.005s
I20260709 13:07:22.270910 25435 tablet_bootstrap.cc:492] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:22.273991 25510 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:22.275590 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:22.275799 25518 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.277388 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:22.278115 25395 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 1: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:07:22.279147 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:22.280167 25510 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.280300 25553 consensus_queue.cc:1048] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:22.290086 25435 tablet_bootstrap.cc:654] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.290241 25510 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:22.296578 25436 tablet_bootstrap.cc:492] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:22.297017 25436 ts_tablet_manager.cc:1403] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.054s	user 0.005s	sys 0.009s
I20260709 13:07:22.299654 25436 raft_consensus.cc:359] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.300289 25436 raft_consensus.cc:385] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.300554 25436 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.302839 25518 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:22.303005 25436 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.304958 25436 ts_tablet_manager.cc:1434] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.000s
I20260709 13:07:22.302650 25142 catalog_manager.cc:5697] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:22.307119 25142 catalog_manager.cc:5697] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:22.308909 25436 tablet_bootstrap.cc:492] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:22.319154 25578 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.319691 25578 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.321787 25578 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:22.322997 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:22.323658 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:22.324263 25359 leader_election.cc:343] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:22.325049 25358 leader_election.cc:343] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:22.325484 25358 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:22.326072 25578 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:22.330241 25437 tablet_bootstrap.cc:492] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:22.330680 25437 ts_tablet_manager.cc:1403] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.067s	user 0.014s	sys 0.019s
I20260709 13:07:22.331365 25435 tablet_bootstrap.cc:492] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:22.331797 25435 ts_tablet_manager.cc:1403] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.061s	user 0.009s	sys 0.000s
I20260709 13:07:22.332957 25436 tablet_bootstrap.cc:654] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.333171 25437 raft_consensus.cc:359] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.333827 25437 raft_consensus.cc:385] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.334152 25437 raft_consensus.cc:740] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.335088 25437 consensus_queue.cc:260] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.336686 25437 ts_tablet_manager.cc:1434] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.001s
I20260709 13:07:22.337523 25437 tablet_bootstrap.cc:492] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:22.338493 25435 raft_consensus.cc:359] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.339150 25435 raft_consensus.cc:385] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.339421 25435 raft_consensus.cc:740] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.345865 25510 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.346338 25510 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.349618 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:22.349850 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:22.350526 25392 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 1: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:07:22.351562 25287 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:22.352905 25518 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.353677 25510 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.356055 25437 tablet_bootstrap.cc:654] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.358302 25578 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.358839 25578 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.363945 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:22.364527 25244 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:22.365818 25359 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:22.365886 25514 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:22.366753 25576 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:22.367105 25576 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:22.367447 25576 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.367142 25578 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:22.369460 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:22.370854 25358 leader_election.cc:343] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:22.371191 25518 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:22.340305 25435 consensus_queue.cc:260] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.371901 25514 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.372354 25514 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.376304 25514 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.377287 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:22.378104 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:22.378683 25392 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
W20260709 13:07:22.379170 25209 leader_election.cc:343] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:22.379781 25212 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:22.380559 25553 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:22.380883 25553 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:22.381223 25553 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.378043 25435 ts_tablet_manager.cc:1434] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.046s	user 0.007s	sys 0.000s
I20260709 13:07:22.381850 25437 tablet_bootstrap.cc:492] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:22.382311 25437 ts_tablet_manager.cc:1403] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.045s	user 0.015s	sys 0.000s
I20260709 13:07:22.382768 25435 tablet_bootstrap.cc:492] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:22.381522 25576 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.387065 25576 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:22.387279 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:22.387813 25244 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.384827 25437 raft_consensus.cc:359] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.389703 25437 raft_consensus.cc:385] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.389977 25437 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.390623 25437 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.393443 25437 ts_tablet_manager.cc:1434] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 13:07:22.393996 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:22.394310 25437 tablet_bootstrap.cc:492] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
W20260709 13:07:22.395345 25358 leader_election.cc:343] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:22.400506 25437 tablet_bootstrap.cc:654] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.401294 25244 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:22.402511 25510 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.403396 25510 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.405455 25510 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.407670 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:22.408253 25246 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
W20260709 13:07:22.406401 25209 consensus_peers.cc:597] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:22.408380 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:22.409798 25394 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:22.409827 25285 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:22.410867 25510 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:22.411216 25510 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:22.411533 25510 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.413004 25359 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:22.414402 25576 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:22.414999 25576 raft_consensus.cc:697] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:22.416010 25576 consensus_queue.cc:237] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.402014 25553 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.420869 25514 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:22.423401 25553 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.424633 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:22.425660 25394 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.424880 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:22.432178 25553 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.432737 25553 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.434952 25394 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
W20260709 13:07:22.435287 25209 leader_election.cc:343] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:22.435487 25553 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.435983 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:22.436651 25212 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:22.438023 25437 tablet_bootstrap.cc:492] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
W20260709 13:07:22.438067 25212 leader_election.cc:343] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:22.438573 25437 ts_tablet_manager.cc:1403] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.044s	user 0.005s	sys 0.022s
I20260709 13:07:22.439327 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:22.440714 25209 leader_election.cc:343] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:22.440603 25142 catalog_manager.cc:5697] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:22.441138 25209 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:22.441100 25437 raft_consensus.cc:359] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.441960 25553 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:22.450103 25552 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:22.450299 25553 raft_consensus.cc:697] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:07:22.450917 25192 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:07:22.451169 25553 consensus_queue.cc:237] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.442590 25510 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.454345 25435 tablet_bootstrap.cc:654] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.463708 25437 raft_consensus.cc:385] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.464051 25437 raft_consensus.cc:740] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.456796 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:22.454584 25510 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.456075 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:22.464753 25244 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.465095 25392 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.468875 25503 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:22.454362 25436 tablet_bootstrap.cc:492] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:22.472929 25436 ts_tablet_manager.cc:1403] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.164s	user 0.035s	sys 0.028s
I20260709 13:07:22.476845 25392 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:22.478395 25287 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:22.479815 25244 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:22.480139 25510 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:22.480954 25510 raft_consensus.cc:697] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:22.474061 25437 consensus_queue.cc:260] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.487906 25510 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.490243 25437 ts_tablet_manager.cc:1434] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.051s	user 0.003s	sys 0.006s
I20260709 13:07:22.491153 25437 tablet_bootstrap.cc:492] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:22.495570 25436 raft_consensus.cc:359] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.496277 25436 raft_consensus.cc:385] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.496569 25436 raft_consensus.cc:740] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.497228 25436 consensus_queue.cc:260] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.504765 25436 ts_tablet_manager.cc:1434] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.031s	user 0.006s	sys 0.000s
I20260709 13:07:22.505887 25142 catalog_manager.cc:5697] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:22.509368 25553 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.509835 25553 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.513388 25141 catalog_manager.cc:5697] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:22.515671 25436 tablet_bootstrap.cc:492] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:22.512988 25553 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.519676 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:22.520042 25435 tablet_bootstrap.cc:492] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:22.520704 25435 ts_tablet_manager.cc:1403] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.138s	user 0.019s	sys 0.010s
W20260709 13:07:22.521986 25209 leader_election.cc:343] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:22.521117 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:22.523401 25435 raft_consensus.cc:359] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.524012 25435 raft_consensus.cc:385] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.524293 25435 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
W20260709 13:07:22.524678 25212 leader_election.cc:343] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:22.525091 25435 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.525488 25212 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:22.527247 25514 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:22.527565 25435 ts_tablet_manager.cc:1434] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.005s
I20260709 13:07:22.528424 25435 tablet_bootstrap.cc:492] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:22.530408 25437 tablet_bootstrap.cc:654] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.538414 25435 tablet_bootstrap.cc:654] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.543493 25436 tablet_bootstrap.cc:654] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.554751 25437 tablet_bootstrap.cc:492] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:22.555207 25437 ts_tablet_manager.cc:1403] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.064s	user 0.009s	sys 0.004s
I20260709 13:07:22.555768 25435 tablet_bootstrap.cc:492] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:22.556320 25435 ts_tablet_manager.cc:1403] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.028s	user 0.013s	sys 0.010s
I20260709 13:07:22.557854 25437 raft_consensus.cc:359] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.558557 25437 raft_consensus.cc:385] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.558809 25437 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.558918 25435 raft_consensus.cc:359] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.559578 25435 raft_consensus.cc:385] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.559463 25437 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.559837 25435 raft_consensus.cc:740] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.560670 25435 consensus_queue.cc:260] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.561623 25437 ts_tablet_manager.cc:1434] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.001s
I20260709 13:07:22.562649 25435 ts_tablet_manager.cc:1434] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 13:07:22.563591 25435 tablet_bootstrap.cc:492] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:22.568610 25436 tablet_bootstrap.cc:492] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:22.569304 25436 ts_tablet_manager.cc:1403] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.064s	user 0.010s	sys 0.005s
I20260709 13:07:22.569370 25435 tablet_bootstrap.cc:654] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.572274 25436 raft_consensus.cc:359] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.572933 25436 raft_consensus.cc:385] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.573213 25436 raft_consensus.cc:740] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.573918 25436 consensus_queue.cc:260] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.578629 25436 ts_tablet_manager.cc:1434] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 13:07:22.579497 25436 tablet_bootstrap.cc:492] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:22.585340 25437 tablet_bootstrap.cc:492] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:22.585834 25436 tablet_bootstrap.cc:654] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.590827 25435 tablet_bootstrap.cc:492] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:22.591264 25435 ts_tablet_manager.cc:1403] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.028s	user 0.013s	sys 0.012s
I20260709 13:07:22.593717 25435 raft_consensus.cc:359] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.594372 25435 raft_consensus.cc:385] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.594667 25435 raft_consensus.cc:740] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.595293 25435 consensus_queue.cc:260] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.593372 25437 tablet_bootstrap.cc:654] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.602685 25435 ts_tablet_manager.cc:1434] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.001s
I20260709 13:07:22.603658 25435 tablet_bootstrap.cc:492] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:22.609467 25435 tablet_bootstrap.cc:654] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.614732 25436 tablet_bootstrap.cc:492] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:22.615191 25436 ts_tablet_manager.cc:1403] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.036s	user 0.013s	sys 0.017s
I20260709 13:07:22.615564 25510 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:22.617218 25435 tablet_bootstrap.cc:492] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:22.617636 25435 ts_tablet_manager.cc:1403] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.014s	user 0.012s	sys 0.000s
I20260709 13:07:22.617707 25436 raft_consensus.cc:359] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.618398 25436 raft_consensus.cc:385] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.618656 25436 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.619227 25436 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.620586 25435 raft_consensus.cc:359] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.621279 25435 raft_consensus.cc:385] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.621534 25435 raft_consensus.cc:740] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.622126 25435 consensus_queue.cc:260] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.622576 25437 tablet_bootstrap.cc:492] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:22.622984 25437 ts_tablet_manager.cc:1403] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.038s	user 0.005s	sys 0.011s
I20260709 13:07:22.624140 25435 ts_tablet_manager.cc:1434] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 13:07:22.625015 25435 tablet_bootstrap.cc:492] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:22.625797 25437 raft_consensus.cc:359] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.626410 25437 raft_consensus.cc:385] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.626660 25437 raft_consensus.cc:740] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.627316 25437 consensus_queue.cc:260] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.628724 25510 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:22.630898 25435 tablet_bootstrap.cc:654] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.632507 25436 ts_tablet_manager.cc:1434] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.001s
I20260709 13:07:22.633433 25436 tablet_bootstrap.cc:492] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:22.640734 25436 tablet_bootstrap.cc:654] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.644184 25437 ts_tablet_manager.cc:1434] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.001s
I20260709 13:07:22.645100 25437 tablet_bootstrap.cc:492] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:22.652793 25437 tablet_bootstrap.cc:654] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.655701 25435 tablet_bootstrap.cc:492] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:22.656185 25435 ts_tablet_manager.cc:1403] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.031s	user 0.002s	sys 0.023s
I20260709 13:07:22.658612 25435 raft_consensus.cc:359] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.659276 25435 raft_consensus.cc:385] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.659538 25435 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.660132 25435 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.662264 25435 ts_tablet_manager.cc:1434] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.002s
I20260709 13:07:22.663323 25435 tablet_bootstrap.cc:492] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:22.667695 25437 tablet_bootstrap.cc:492] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:22.668159 25437 ts_tablet_manager.cc:1403] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.023s	user 0.006s	sys 0.010s
I20260709 13:07:22.669322 25435 tablet_bootstrap.cc:654] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.670181 25553 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.670639 25553 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.673812 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:22.672062 25437 raft_consensus.cc:359] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.674492 25321 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:22.675664 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:22.676234 25394 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:22.676302 25437 raft_consensus.cc:385] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.676769 25437 raft_consensus.cc:740] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.677575 25212 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:22.677631 25437 consensus_queue.cc:260] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.678472 25469 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:22.678866 25469 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:22.679183 25469 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.682139 25435 tablet_bootstrap.cc:492] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:22.682571 25435 ts_tablet_manager.cc:1403] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.019s	user 0.013s	sys 0.004s
I20260709 13:07:22.683092 25437 ts_tablet_manager.cc:1434] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.000s
I20260709 13:07:22.682139 25436 tablet_bootstrap.cc:492] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:22.683804 25436 ts_tablet_manager.cc:1403] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.051s	user 0.011s	sys 0.007s
I20260709 13:07:22.684763 25553 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.684983 25435 raft_consensus.cc:359] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.685684 25435 raft_consensus.cc:385] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.685950 25435 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.686129 25436 raft_consensus.cc:359] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.686776 25436 raft_consensus.cc:385] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.686571 25435 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.687032 25436 raft_consensus.cc:740] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.688050 25437 tablet_bootstrap.cc:492] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:22.688741 25435 ts_tablet_manager.cc:1434] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:07:22.689584 25435 tablet_bootstrap.cc:492] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:22.692263 25469 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.693944 25437 tablet_bootstrap.cc:654] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.694885 25435 tablet_bootstrap.cc:654] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.694981 25469 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.695187 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:22.695719 25436 consensus_queue.cc:260] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.697160 25321 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.698484 25436 ts_tablet_manager.cc:1434] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 13:07:22.699347 25436 tablet_bootstrap.cc:492] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:22.701581 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:22.702135 25394 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.706118 25576 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.707310 25576 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.709277 25576 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:22.709647 25321 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:22.710016 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:22.710680 25244 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:22.710881 25209 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:22.710897 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:22.716174 25358 leader_election.cc:343] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:22.710239 25394 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:22.717700 25359 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:22.718737 25469 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:22.719372 25469 raft_consensus.cc:697] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:22.720144 25469 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.721000 25436 tablet_bootstrap.cc:654] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.718884 25576 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:22.722713 25576 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:22.723043 25576 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.729820 25576 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.734398 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:22.734947 25244 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.736421 25436 tablet_bootstrap.cc:492] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:22.738477 25436 ts_tablet_manager.cc:1403] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.039s	user 0.014s	sys 0.000s
I20260709 13:07:22.741600 25244 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:22.742780 25359 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:22.743634 25569 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:22.744246 25569 raft_consensus.cc:697] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:22.745144 25569 consensus_queue.cc:237] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.745584 25436 raft_consensus.cc:359] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.746934 25436 raft_consensus.cc:385] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.747617 25436 raft_consensus.cc:740] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.748679 25143 catalog_manager.cc:5697] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:22.749271 25436 consensus_queue.cc:260] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.754120 25437 tablet_bootstrap.cc:492] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:22.754532 25437 ts_tablet_manager.cc:1403] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.067s	user 0.024s	sys 0.016s
I20260709 13:07:22.757253 25435 tablet_bootstrap.cc:492] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:22.757748 25435 ts_tablet_manager.cc:1403] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.068s	user 0.008s	sys 0.014s
I20260709 13:07:22.757921 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:07:22.759378 25358 leader_election.cc:343] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:22.760107 25576 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:22.762569 25435 raft_consensus.cc:359] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.763412 25436 ts_tablet_manager.cc:1434] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.024s	user 0.007s	sys 0.000s
I20260709 13:07:22.763836 25435 raft_consensus.cc:385] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.764195 25435 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.764644 25437 raft_consensus.cc:359] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.764958 25435 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.765789 25437 raft_consensus.cc:385] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.766086 25437 raft_consensus.cc:740] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.767690 25436 tablet_bootstrap.cc:492] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:22.770380 25435 ts_tablet_manager.cc:1434] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.012s	user 0.000s	sys 0.006s
I20260709 13:07:22.771255 25435 tablet_bootstrap.cc:492] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:22.773698 25437 consensus_queue.cc:260] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.776002 25437 ts_tablet_manager.cc:1434] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.001s
I20260709 13:07:22.776855 25437 tablet_bootstrap.cc:492] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:22.778695 25435 tablet_bootstrap.cc:654] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.779582 25141 catalog_manager.cc:5697] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:22.782524 25437 tablet_bootstrap.cc:654] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.784883 25569 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.785497 25569 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.788120 25569 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:22.789000 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:22.789022 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:22.791112 25358 leader_election.cc:343] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:22.792052 25359 leader_election.cc:343] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:22.792500 25359 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:22.793195 25569 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:22.799150 25437 tablet_bootstrap.cc:492] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:22.799597 25437 ts_tablet_manager.cc:1403] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.023s	user 0.014s	sys 0.001s
I20260709 13:07:22.800417 25555 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.800874 25555 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.802340 25437 raft_consensus.cc:359] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.802846 25555 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.802968 25437 raft_consensus.cc:385] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.803418 25437 raft_consensus.cc:740] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.804229 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:22.804167 25437 consensus_queue.cc:260] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.804994 25394 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 1: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:07:22.806815 25437 ts_tablet_manager.cc:1434] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 13:07:22.807453 25436 tablet_bootstrap.cc:654] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.807893 25437 tablet_bootstrap.cc:492] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:22.810278 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:22.811602 25285 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:22.812373 25516 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.814322 25437 tablet_bootstrap.cc:654] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.817657 25436 tablet_bootstrap.cc:492] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:22.818081 25436 ts_tablet_manager.cc:1403] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.051s	user 0.009s	sys 0.001s
I20260709 13:07:22.818645 25435 tablet_bootstrap.cc:492] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:22.819046 25516 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:22.819245 25435 ts_tablet_manager.cc:1403] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.048s	user 0.018s	sys 0.014s
I20260709 13:07:22.820888 25436 raft_consensus.cc:359] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.821893 25436 raft_consensus.cc:385] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.822167 25436 raft_consensus.cc:740] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.822921 25436 consensus_queue.cc:260] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.823987 25321 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.824136 25435 raft_consensus.cc:359] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.824929 25435 raft_consensus.cc:385] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.825045 25436 ts_tablet_manager.cc:1434] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:07:22.825286 25435 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.826083 25435 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.826507 25436 tablet_bootstrap.cc:492] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:22.833045 25436 tablet_bootstrap.cc:654] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.835201 25553 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.835654 25553 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.836179 25516 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.836745 25516 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.837623 25553 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.839242 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:22.839346 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:22.839879 25245 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:22.839963 25320 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:22.840375 25569 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:22.841586 25209 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:22.841665 25285 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:22.842598 25555 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:07:22.842932 25469 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:22.842945 25555 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:22.843678 25555 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:22.845492 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:22.846056 25394 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:22.843297 25469 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:22.849694 25469 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.849511 25569 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:22.849944 25553 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:22.851065 25437 tablet_bootstrap.cc:492] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:22.851960 25437 ts_tablet_manager.cc:1403] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.044s	user 0.017s	sys 0.000s
I20260709 13:07:22.852726 25553 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.853179 25553 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.855064 25437 raft_consensus.cc:359] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.855690 25437 raft_consensus.cc:385] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.855756 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:22.856012 25437 raft_consensus.cc:740] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.856437 25555 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:07:22.857141 25209 leader_election.cc:343] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:22.863520 25576 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.863976 25576 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.866704 25576 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:22.867084 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:22.867527 25516 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.867772 25319 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:22.868712 25321 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 13:07:22.869374 25553 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.857936 25435 ts_tablet_manager.cc:1434] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.037s	user 0.004s	sys 0.003s
I20260709 13:07:22.856840 25437 consensus_queue.cc:260] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.869818 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:22.870388 25246 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:22.870524 25435 tablet_bootstrap.cc:492] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:22.871416 25437 ts_tablet_manager.cc:1434] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.000s
I20260709 13:07:22.871356 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:22.872330 25437 tablet_bootstrap.cc:492] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:22.873456 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:22.873416 25359 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:22.861032 25469 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.875730 25569 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:22.879948 25569 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:22.880383 25569 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.881482 25319 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.884549 25436 tablet_bootstrap.cc:492] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:22.885004 25436 ts_tablet_manager.cc:1403] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.059s	user 0.013s	sys 0.001s
I20260709 13:07:22.885040 25469 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.887607 25437 tablet_bootstrap.cc:654] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.887822 25436 raft_consensus.cc:359] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.888484 25436 raft_consensus.cc:385] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.888756 25436 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.889202 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:22.889396 25436 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.877547 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:22.877447 25435 tablet_bootstrap.cc:654] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.877810 25555 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.890308 25394 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:22.879406 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:22.889767 25393 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.894306 25436 ts_tablet_manager.cc:1434] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 13:07:22.895859 25436 tablet_bootstrap.cc:492] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:22.877691 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:22.897054 25321 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.901332 25436 tablet_bootstrap.cc:654] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:22.904134 25555 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:22.905754 25287 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:22.907997 25469 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:22.910084 25555 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:07:22.915419 25518 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:22.923627 25212 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:22.925539 25553 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:22.926062 25553 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:22.926548 25553 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:22.931144 25634 consensus_queue.cc:1059] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:22.934231 25576 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.934682 25576 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.941816 25393 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:22.966282 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:22.967971 25245 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:22.969245 25359 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:22.969753 25514 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:22.970388 25514 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.968712 25321 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:22.976452 25514 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:22.982518 25435 tablet_bootstrap.cc:492] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:22.982991 25435 ts_tablet_manager.cc:1403] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.113s	user 0.012s	sys 0.032s
I20260709 13:07:22.966672 25576 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:22.987624 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:22.988758 25209 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:22.991040 25569 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.992516 25552 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:22.992241 25435 raft_consensus.cc:359] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:22.994901 25435 raft_consensus.cc:385] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:22.995136 25435 raft_consensus.cc:740] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:22.976869 25632 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:22.997776 25632 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:23.007436 25435 consensus_queue.cc:260] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.008915 25632 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:23.010641 25209 leader_election.cc:343] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.011492 25516 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:07:23.012207 25437 tablet_bootstrap.cc:492] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:23.012009 25516 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.012641 25437 ts_tablet_manager.cc:1403] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.141s	user 0.032s	sys 0.015s
I20260709 13:07:23.013756 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:23.015558 25358 leader_election.cc:343] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.015889 25437 raft_consensus.cc:359] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.020706 25437 raft_consensus.cc:385] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.020952 25437 raft_consensus.cc:740] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.029774 25437 consensus_queue.cc:260] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.039767 25437 ts_tablet_manager.cc:1434] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.027s	user 0.008s	sys 0.000s
I20260709 13:07:23.040967 25437 tablet_bootstrap.cc:492] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:23.046919 25435 ts_tablet_manager.cc:1434] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.064s	user 0.004s	sys 0.003s
I20260709 13:07:23.047659 25435 tablet_bootstrap.cc:492] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:23.009567 25553 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.020411 25552 raft_consensus.cc:697] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:23.026141 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:23.060060 25436 tablet_bootstrap.cc:492] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:23.060539 25436 ts_tablet_manager.cc:1403] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.165s	user 0.029s	sys 0.071s
I20260709 13:07:23.055241 25569 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:23.062980 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:23.063630 25244 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:23.064303 25634 raft_consensus.cc:3055] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:23.064674 25634 raft_consensus.cc:740] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:23.065516 25634 consensus_queue.cc:260] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.066540 25634 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:23.062685 25632 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.084306 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:23.091912 25469 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:23.092407 25469 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.095461 25516 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:23.077188 25638 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:23.107733 25244 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:23.108626 25638 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.096863 25632 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:23.120883 25436 raft_consensus.cc:359] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.122167 25436 raft_consensus.cc:385] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.122417 25436 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.123180 25436 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.126536 25436 ts_tablet_manager.cc:1434] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.066s	user 0.005s	sys 0.001s
I20260709 13:07:23.126852 25469 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:23.127609 25436 tablet_bootstrap.cc:492] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:23.075086 25552 consensus_queue.cc:237] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.139999 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:23.142318 25320 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:23.143369 25437 tablet_bootstrap.cc:654] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.133719 25553 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:23.148240 25359 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:23.149147 25632 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:23.149607 25632 raft_consensus.cc:697] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:23.152078 25632 consensus_queue.cc:237] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.156145 25435 tablet_bootstrap.cc:654] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.160854 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:23.161428 25245 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:23.164467 25320 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:23.167783 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:07:23.168054 25212 leader_election.cc:343] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.169566 25212 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:23.170863 25245 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:23.172847 25578 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:23.173324 25578 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.174538 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:23.175191 25578 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:23.177273 25639 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:23.175802 25638 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:07:23.179051 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 116414us
I20260709 13:07:23.172508 25359 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:23.193562 25632 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:23.194236 25632 raft_consensus.cc:697] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:23.197834 25632 consensus_queue.cc:237] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.201557 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:23.201979 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:23.202160 25395 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:23.202459 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:23.203029 25392 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:23.208923 25143 catalog_manager.cc:5697] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:23.210044 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:23.213038 25393 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:23.221434 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:23.222718 25245 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:23.223579 25287 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:23.221539 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:23.226238 25436 tablet_bootstrap.cc:654] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.226104 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:23.227864 25640 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:07:23.228235 25318 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:23.227306 25469 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:23.230161 25469 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.233669 25395 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:23.236586 25469 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:23.227444 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:23.237965 25319 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:23.244302 25143 catalog_manager.cc:5697] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:23.246189 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:23.248044 25359 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:07:23.250999 25358 leader_election.cc:343] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:23.251878 25358 leader_election.cc:343] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.254187 25632 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:23.255334 25212 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:23.255743 25632 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:23.256153 25632 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:23.256618 25212 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:23.258872 25319 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:23.262840 25643 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:23.263254 25643 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:23.263607 25643 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:23.264847 25437 tablet_bootstrap.cc:492] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:23.265681 25437 ts_tablet_manager.cc:1403] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.225s	user 0.017s	sys 0.015s
W20260709 13:07:23.268622 25358 consensus_peers.cc:597] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:23.268360 25632 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.260129 25634 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:23.270288 25634 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.272354 25634 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:23.262181 25514 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:23.273197 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:23.273604 25435 tablet_bootstrap.cc:492] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:23.274107 25435 ts_tablet_manager.cc:1403] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.227s	user 0.006s	sys 0.011s
I20260709 13:07:23.273555 25514 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:07:23.274495 25359 leader_election.cc:343] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.269235 25212 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:23.274853 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:23.275139 25514 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:23.279991 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:23.280269 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:07:23.280902 25209 leader_election.cc:343] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:23.281571 25358 leader_election.cc:343] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.282006 25358 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:07:23.282594 25209 leader_election.cc:343] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.282701 25638 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:07:23.283062 25358 leader_election.cc:343] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.283599 25639 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:23.284097 25639 raft_consensus.cc:697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:23.283970 25514 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.285573 25634 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:23.275374 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:23.286129 25639 consensus_queue.cc:237] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.292789 25245 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:23.292591 25643 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.296623 25643 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:23.287660 25435 raft_consensus.cc:359] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:07:23.289995 25212 leader_election.cc:343] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.300546 25435 raft_consensus.cc:385] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.302649 25435 raft_consensus.cc:740] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.303442 25435 consensus_queue.cc:260] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.307713 25514 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:23.276492 25632 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:23.305790 25212 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:23.313104 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:23.310276 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:23.313665 25392 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:23.313800 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:23.314395 25395 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:23.317306 25469 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:23.320313 25392 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:23.321419 25212 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:23.322144 25395 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:23.322194 25469 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:23.322882 25469 raft_consensus.cc:697] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:23.323407 25212 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:23.324393 25638 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:23.324844 25638 raft_consensus.cc:697] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:23.325043 25245 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:23.326011 25143 catalog_manager.cc:5697] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:23.325676 25638 consensus_queue.cc:237] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.327318 25435 ts_tablet_manager.cc:1434] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.053s	user 0.006s	sys 0.001s
I20260709 13:07:23.328181 25435 tablet_bootstrap.cc:492] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:23.328529 25437 raft_consensus.cc:359] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.329190 25437 raft_consensus.cc:385] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.329448 25437 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.329978 25437 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.331277 25359 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:23.323665 25469 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.333628 25634 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:23.337852 25634 raft_consensus.cc:697] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:23.338376 25437 ts_tablet_manager.cc:1434] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.072s	user 0.005s	sys 0.000s
I20260709 13:07:23.338606 25634 consensus_queue.cc:237] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.339246 25437 tablet_bootstrap.cc:492] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:23.339856 25436 tablet_bootstrap.cc:492] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:23.340245 25436 ts_tablet_manager.cc:1403] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.213s	user 0.013s	sys 0.011s
W20260709 13:07:23.385447 25209 leader_election.cc:343] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.386452 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:23.388020 25319 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:23.395587 25141 catalog_manager.cc:5697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:23.405674 25578 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:23.406766 25437 tablet_bootstrap.cc:654] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.427273 25435 tablet_bootstrap.cc:654] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.438431 25436 raft_consensus.cc:359] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.439253 25436 raft_consensus.cc:385] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.440287 25643 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:23.441716 25319 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:23.445072 25436 raft_consensus.cc:740] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.446389 25436 consensus_queue.cc:260] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.450767 25436 ts_tablet_manager.cc:1434] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.110s	user 0.011s	sys 0.000s
I20260709 13:07:23.471107 25640 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:23.471709 25640 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.477105 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:23.478217 25245 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 1: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:07:23.487221 25436 tablet_bootstrap.cc:492] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:23.490836 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:23.489377 25640 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:23.495482 25287 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:23.496887 25640 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:23.518564 25435 tablet_bootstrap.cc:492] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:23.519309 25435 ts_tablet_manager.cc:1403] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.191s	user 0.015s	sys 0.002s
I20260709 13:07:23.524322 25436 tablet_bootstrap.cc:654] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.524084 25435 raft_consensus.cc:359] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.537341 25435 raft_consensus.cc:385] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.537788 25435 raft_consensus.cc:740] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.537288 25142 catalog_manager.cc:5697] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:23.538678 25435 consensus_queue.cc:260] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.540443 25435 ts_tablet_manager.cc:1434] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.001s
I20260709 13:07:23.544445 25435 tablet_bootstrap.cc:492] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:23.547797 25469 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:23.548298 25469 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.550467 25642 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:23.550933 25642 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.551204 25469 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:23.551905 25469 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:23.553076 25642 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:23.554423 25436 tablet_bootstrap.cc:492] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:23.554924 25436 ts_tablet_manager.cc:1403] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.083s	user 0.012s	sys 0.003s
I20260709 13:07:23.556475 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:23.557207 25390 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:23.536306 25437 tablet_bootstrap.cc:492] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:23.557624 25436 raft_consensus.cc:359] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.558337 25437 ts_tablet_manager.cc:1403] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.219s	user 0.014s	sys 0.014s
I20260709 13:07:23.558507 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:23.560487 25212 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:23.561976 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:23.562618 25643 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:23.562532 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:23.563061 25643 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:23.563423 25643 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:23.564386 25209 leader_election.cc:343] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.558450 25436 raft_consensus.cc:385] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.567374 25436 raft_consensus.cc:740] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
W20260709 13:07:23.567713 25212 leader_election.cc:343] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:23.565922 25209 leader_election.cc:343] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.568396 25212 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:23.568181 25436 consensus_queue.cc:260] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.567643 25578 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:23.569516 25553 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:23.530797 25640 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:23.571352 25436 ts_tablet_manager.cc:1434] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.016s	user 0.009s	sys 0.000s
I20260709 13:07:23.572573 25436 tablet_bootstrap.cc:492] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
W20260709 13:07:23.573915 25192 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:07:23.566471 25437 raft_consensus.cc:359] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.575739 25437 raft_consensus.cc:385] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.576061 25437 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.575814 25643 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.577657 25553 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:23.577775 25437 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.583973 25435 tablet_bootstrap.cc:654] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.595052 25437 ts_tablet_manager.cc:1434] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.036s	user 0.006s	sys 0.002s
I20260709 13:07:23.596513 25437 tablet_bootstrap.cc:492] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:23.602391 25143 catalog_manager.cc:5697] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:23.604552 25436 tablet_bootstrap.cc:654] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.604115 25143 catalog_manager.cc:5697] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:23.614437 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:23.614852 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:23.615453 25395 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:23.626163 25209 leader_election.cc:343] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.630894 25437 tablet_bootstrap.cc:654] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.632031 25643 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:23.644587 25435 tablet_bootstrap.cc:492] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:23.645094 25435 ts_tablet_manager.cc:1403] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.101s	user 0.026s	sys 0.014s
I20260709 13:07:23.648401 25395 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:23.661664 25436 tablet_bootstrap.cc:492] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:23.662163 25436 ts_tablet_manager.cc:1403] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.090s	user 0.022s	sys 0.001s
I20260709 13:07:23.666095 25319 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:23.666456 25436 raft_consensus.cc:359] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.667172 25436 raft_consensus.cc:385] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.667428 25436 raft_consensus.cc:740] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.668161 25436 consensus_queue.cc:260] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.669739 25436 ts_tablet_manager.cc:1434] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.001s
I20260709 13:07:23.672487 25436 tablet_bootstrap.cc:492] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:23.671860 25435 raft_consensus.cc:359] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.673470 25435 raft_consensus.cc:385] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.673745 25435 raft_consensus.cc:740] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.674377 25634 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:23.674345 25435 consensus_queue.cc:260] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.673251 25212 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:23.679824 25642 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:23.680301 25642 raft_consensus.cc:697] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:23.681471 25435 ts_tablet_manager.cc:1434] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.036s	user 0.006s	sys 0.002s
I20260709 13:07:23.681412 25642 consensus_queue.cc:237] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.682680 25435 tablet_bootstrap.cc:492] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:23.688725 25436 tablet_bootstrap.cc:654] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.695535 25435 tablet_bootstrap.cc:654] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.699716 25553 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:23.711144 25437 tablet_bootstrap.cc:492] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:23.711699 25437 ts_tablet_manager.cc:1403] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.116s	user 0.037s	sys 0.021s
I20260709 13:07:23.713012 25435 tablet_bootstrap.cc:492] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:23.713439 25435 ts_tablet_manager.cc:1403] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.031s	user 0.011s	sys 0.000s
I20260709 13:07:23.714421 25437 raft_consensus.cc:359] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.715080 25437 raft_consensus.cc:385] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.715350 25437 raft_consensus.cc:740] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.715382 25435 raft_consensus.cc:359] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.716025 25435 raft_consensus.cc:385] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.716284 25435 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.716059 25437 consensus_queue.cc:260] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.716915 25435 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.718180 25437 ts_tablet_manager.cc:1434] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:07:23.718690 25435 ts_tablet_manager.cc:1434] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:07:23.719161 25437 tablet_bootstrap.cc:492] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:23.719513 25435 tablet_bootstrap.cc:492] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:23.725849 25437 tablet_bootstrap.cc:654] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.725904 25435 tablet_bootstrap.cc:654] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.736896 25578 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:23.737833 25435 tablet_bootstrap.cc:492] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:23.744189 25435 ts_tablet_manager.cc:1403] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.025s	user 0.018s	sys 0.000s
I20260709 13:07:23.747416 25435 raft_consensus.cc:359] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.748276 25435 raft_consensus.cc:385] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.748594 25435 raft_consensus.cc:740] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.749293 25435 consensus_queue.cc:260] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.749686 25469 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:23.751403 25435 ts_tablet_manager.cc:1434] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.001s
I20260709 13:07:23.752499 25435 tablet_bootstrap.cc:492] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:23.755115 25436 tablet_bootstrap.cc:492] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:23.755697 25436 ts_tablet_manager.cc:1403] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.085s	user 0.006s	sys 0.006s
I20260709 13:07:23.759111 25435 tablet_bootstrap.cc:654] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.778905 25141 catalog_manager.cc:5697] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:23.786527 25634 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:23.792426 25436 raft_consensus.cc:359] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.793093 25436 raft_consensus.cc:385] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.793411 25436 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.795370 25436 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.797973 25436 ts_tablet_manager.cc:1434] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.042s	user 0.004s	sys 0.004s
I20260709 13:07:23.798970 25436 tablet_bootstrap.cc:492] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:23.807993 25436 tablet_bootstrap.cc:654] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
W20260709 13:07:23.810519 25358 consensus_peers.cc:597] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:23.811367 25437 tablet_bootstrap.cc:492] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:23.811851 25437 ts_tablet_manager.cc:1403] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.093s	user 0.042s	sys 0.033s
I20260709 13:07:23.814661 25435 tablet_bootstrap.cc:492] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:23.814437 25437 raft_consensus.cc:359] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.815093 25437 raft_consensus.cc:385] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.815100 25435 ts_tablet_manager.cc:1403] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.063s	user 0.019s	sys 0.033s
I20260709 13:07:23.815505 25437 raft_consensus.cc:740] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.816252 25437 consensus_queue.cc:260] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.818550 25437 ts_tablet_manager.cc:1434] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.001s
I20260709 13:07:23.818807 25435 raft_consensus.cc:359] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.820065 25435 raft_consensus.cc:385] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.820359 25435 raft_consensus.cc:740] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.822458 25437 tablet_bootstrap.cc:492] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:23.820969 25435 consensus_queue.cc:260] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.825069 25578 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:23.826356 25553 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:23.828959 25437 tablet_bootstrap.cc:654] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.829241 25469 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:23.830220 25643 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:23.833541 25576 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:23.834013 25576 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.836268 25576 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:23.841627 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:23.843559 25435 ts_tablet_manager.cc:1434] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.028s	user 0.006s	sys 0.000s
I20260709 13:07:23.844664 25435 tablet_bootstrap.cc:492] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:23.844787 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:23.845614 25246 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:23.849577 25553 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:23.849857 25359 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:23.850077 25553 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.850942 25578 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:23.851420 25578 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:23.851830 25578 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:23.856112 25436 tablet_bootstrap.cc:492] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:23.856570 25436 ts_tablet_manager.cc:1403] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.058s	user 0.009s	sys 0.003s
I20260709 13:07:23.858661 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:23.860383 25209 leader_election.cc:343] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.866546 25578 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:07:23.869833 25358 leader_election.cc:343] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.872707 25578 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:23.875466 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:23.877110 25246 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:23.877554 25209 consensus_peers.cc:597] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:23.884814 25436 raft_consensus.cc:359] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.885509 25436 raft_consensus.cc:385] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.885787 25436 raft_consensus.cc:740] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.886410 25436 consensus_queue.cc:260] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.876683 25437 tablet_bootstrap.cc:492] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:23.887753 25437 ts_tablet_manager.cc:1403] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.068s	user 0.009s	sys 0.009s
I20260709 13:07:23.881747 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:23.888538 25436 ts_tablet_manager.cc:1434] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.032s	user 0.005s	sys 0.000s
I20260709 13:07:23.889794 25553 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:23.894455 25246 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:23.896108 25436 tablet_bootstrap.cc:492] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:23.902696 25435 tablet_bootstrap.cc:654] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.901093 25359 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:23.904096 25437 raft_consensus.cc:359] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.904886 25437 raft_consensus.cc:385] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.905248 25437 raft_consensus.cc:740] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.905954 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:23.905949 25437 consensus_queue.cc:260] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.908623 25576 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:23.909168 25643 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:23.909587 25576 raft_consensus.cc:697] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:23.909772 25643 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.910619 25576 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.911777 25643 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:07:23.913241 25212 leader_election.cc:343] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.913684 25212 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:23.912619 25437 ts_tablet_manager.cc:1434] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.024s	user 0.003s	sys 0.002s
I20260709 13:07:23.912535 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:23.914677 25553 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:07:23.915694 25209 leader_election.cc:343] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.916417 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:07:23.916641 25358 leader_election.cc:343] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.915715 25437 tablet_bootstrap.cc:492] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
W20260709 13:07:23.918548 25212 leader_election.cc:343] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.918998 25212 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:23.919970 25643 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:23.930685 25437 tablet_bootstrap.cc:654] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.939757 25436 tablet_bootstrap.cc:654] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.948479 25514 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:23.949193 25435 tablet_bootstrap.cc:492] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:23.948972 25514 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.950073 25435 ts_tablet_manager.cc:1403] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.106s	user 0.023s	sys 0.013s
I20260709 13:07:23.951627 25514 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:23.951261 25141 catalog_manager.cc:5697] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:23.953219 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:23.953256 25435 raft_consensus.cc:359] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.954064 25390 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:23.954116 25435 raft_consensus.cc:385] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.954571 25435 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.955044 25436 tablet_bootstrap.cc:492] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:23.955475 25436 ts_tablet_manager.cc:1403] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.060s	user 0.010s	sys 0.005s
I20260709 13:07:23.955297 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:23.956075 25514 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:23.955288 25435 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.956490 25514 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:23.958199 25435 ts_tablet_manager.cc:1434] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:07:23.959408 25435 tablet_bootstrap.cc:492] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:23.959839 25437 tablet_bootstrap.cc:492] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:23.960259 25437 ts_tablet_manager.cc:1403] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.045s	user 0.001s	sys 0.012s
I20260709 13:07:23.960964 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:23.961169 25514 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:23.962376 25209 leader_election.cc:343] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.962816 25437 raft_consensus.cc:359] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.963526 25437 raft_consensus.cc:385] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.963796 25437 raft_consensus.cc:740] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.964462 25437 consensus_queue.cc:260] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.957871 25436 raft_consensus.cc:359] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.965402 25436 raft_consensus.cc:385] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.965675 25436 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.966012 25435 tablet_bootstrap.cc:654] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.966641 25436 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.969311 25437 ts_tablet_manager.cc:1434] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 13:07:23.970101 25514 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.970587 25437 tablet_bootstrap.cc:492] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:23.973425 25436 ts_tablet_manager.cc:1434] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.001s
I20260709 13:07:23.977764 25436 tablet_bootstrap.cc:492] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:23.978907 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:07:23.980353 25209 leader_election.cc:343] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.982020 25514 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:23.982522 25553 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:23.983006 25553 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.983628 25436 tablet_bootstrap.cc:654] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.985034 25553 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:23.987035 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:23.987942 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:23.988474 25390 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:23.989094 25437 tablet_bootstrap.cc:654] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:23.991297 25435 tablet_bootstrap.cc:492] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:23.991757 25435 ts_tablet_manager.cc:1403] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.033s	user 0.010s	sys 0.008s
I20260709 13:07:23.993530 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:23.994488 25435 raft_consensus.cc:359] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.994956 25436 tablet_bootstrap.cc:492] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:23.995131 25435 raft_consensus.cc:385] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.995397 25436 ts_tablet_manager.cc:1403] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.018s	user 0.009s	sys 0.000s
I20260709 13:07:23.995385 25435 raft_consensus.cc:740] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.996131 25435 consensus_queue.cc:260] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.996414 25390 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
W20260709 13:07:23.988488 25209 leader_election.cc:343] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:23.997208 25212 leader_election.cc:343] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:23.998160 25435 ts_tablet_manager.cc:1434] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 13:07:23.997946 25436 raft_consensus.cc:359] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:23.998566 25436 raft_consensus.cc:385] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:23.998805 25436 raft_consensus.cc:740] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:23.999507 25435 tablet_bootstrap.cc:492] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:24.000428 25553 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.000982 25553 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.003005 25553 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:24.003459 25437 tablet_bootstrap.cc:492] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:24.003881 25437 ts_tablet_manager.cc:1403] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.034s	user 0.005s	sys 0.007s
I20260709 13:07:24.005477 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:24.006419 25436 consensus_queue.cc:260] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:07:24.010216 25209 leader_election.cc:343] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.008726 25436 ts_tablet_manager.cc:1434] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.000s
I20260709 13:07:24.011346 25436 tablet_bootstrap.cc:492] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:23.999298 25212 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:24.009969 25437 raft_consensus.cc:359] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.013365 25437 raft_consensus.cc:385] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.013484 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:24.013681 25437 raft_consensus.cc:740] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.014479 25553 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:24.014756 25643 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.014745 25437 consensus_queue.cc:260] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.015336 25643 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:07:24.016309 25212 leader_election.cc:343] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.016741 25212 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:24.017027 25437 ts_tablet_manager.cc:1434] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.001s
I20260709 13:07:24.017275 25435 tablet_bootstrap.cc:654] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.018025 25437 tablet_bootstrap.cc:492] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:24.017941 25643 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:24.018569 25643 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:24.020371 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:07:24.022639 25212 leader_election.cc:343] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.024004 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:24.024173 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:24.024765 25643 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:24.025233 25643 raft_consensus.cc:697] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:07:24.025840 25209 leader_election.cc:343] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.026029 25643 consensus_queue.cc:237] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.030026 25435 tablet_bootstrap.cc:492] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:24.030447 25435 ts_tablet_manager.cc:1403] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.031s	user 0.013s	sys 0.005s
I20260709 13:07:24.030874 25436 tablet_bootstrap.cc:654] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.032708 25209 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:24.026582 25437 tablet_bootstrap.cc:654] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.033303 25514 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:24.033035 25435 raft_consensus.cc:359] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.033737 25435 raft_consensus.cc:385] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.034013 25435 raft_consensus.cc:740] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.034754 25435 consensus_queue.cc:260] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.038760 25435 ts_tablet_manager.cc:1434] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.003s
I20260709 13:07:24.043197 25435 tablet_bootstrap.cc:492] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:24.043670 25576 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.044148 25576 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.045364 25144 catalog_manager.cc:5697] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:24.048195 25576 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:24.049458 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:24.053433 25437 tablet_bootstrap.cc:492] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:24.053735 25435 tablet_bootstrap.cc:654] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.053788 25436 tablet_bootstrap.cc:492] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:24.053879 25437 ts_tablet_manager.cc:1403] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.036s	user 0.014s	sys 0.003s
I20260709 13:07:24.054365 25436 ts_tablet_manager.cc:1403] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.043s	user 0.015s	sys 0.000s
I20260709 13:07:24.050071 25246 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:24.058578 25359 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:24.059434 25576 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:24.059830 25576 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:24.060170 25576 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.060761 25436 raft_consensus.cc:359] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.061638 25436 raft_consensus.cc:385] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.061946 25436 raft_consensus.cc:740] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.062003 25437 raft_consensus.cc:359] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.062668 25437 raft_consensus.cc:385] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.063040 25437 raft_consensus.cc:740] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.065524 25437 consensus_queue.cc:260] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.068195 25437 ts_tablet_manager.cc:1434] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.000s
I20260709 13:07:24.062619 25436 consensus_queue.cc:260] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.068327 25640 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.068828 25640 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.070817 25436 ts_tablet_manager.cc:1434] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20260709 13:07:24.072324 25436 tablet_bootstrap.cc:492] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:24.072710 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:24.074060 25437 tablet_bootstrap.cc:492] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:24.077052 25640 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:24.078069 25555 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.083510 25555 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.078518 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:24.084401 25319 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:24.081759 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:24.087363 25287 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:24.089000 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:24.092599 25555 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:24.094390 25555 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:24.087886 25576 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.095368 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:24.098035 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:24.098619 25246 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.098757 25287 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:24.105786 25437 tablet_bootstrap.cc:654] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.109462 25246 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:24.110731 25359 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:24.095911 25436 tablet_bootstrap.cc:654] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.112210 25576 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:24.113014 25634 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:24.113744 25640 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:24.113847 25634 raft_consensus.cc:697] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:24.114961 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:24.114718 25634 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.115476 25319 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.123090 25319 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:24.140890 25435 tablet_bootstrap.cc:492] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:24.140890 25437 tablet_bootstrap.cc:492] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:24.141412 25437 ts_tablet_manager.cc:1403] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.068s	user 0.016s	sys 0.012s
I20260709 13:07:24.141412 25435 ts_tablet_manager.cc:1403] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.098s	user 0.009s	sys 0.041s
I20260709 13:07:24.144141 25435 raft_consensus.cc:359] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.144268 25437 raft_consensus.cc:359] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.144814 25435 raft_consensus.cc:385] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.144405 25144 catalog_manager.cc:5697] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:24.145246 25435 raft_consensus.cc:740] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.146104 25435 consensus_queue.cc:260] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.147655 25435 ts_tablet_manager.cc:1434] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.004s
I20260709 13:07:24.148813 25435 tablet_bootstrap.cc:492] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:24.144842 25437 raft_consensus.cc:385] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.153153 25437 raft_consensus.cc:740] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.154153 25697 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.154141 25437 consensus_queue.cc:260] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.154874 25697 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.157215 25697 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:24.158002 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:24.158675 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:24.159190 25390 raft_consensus.cc:2393] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 1: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:07:24.152864 25634 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.159641 25634 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.160318 25287 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:24.161314 25697 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.164618 25437 ts_tablet_manager.cc:1434] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.000s
I20260709 13:07:24.165588 25437 tablet_bootstrap.cc:492] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:24.166563 25697 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:24.168808 25435 tablet_bootstrap.cc:654] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.171690 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:24.172173 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:24.173074 25359 leader_election.cc:343] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:24.173801 25358 leader_election.cc:343] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.174229 25358 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:24.175083 25578 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:24.176195 25634 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:24.178953 25437 tablet_bootstrap.cc:654] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.179517 25435 tablet_bootstrap.cc:492] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:24.179955 25435 ts_tablet_manager.cc:1403] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.031s	user 0.008s	sys 0.008s
I20260709 13:07:24.181353 25436 tablet_bootstrap.cc:492] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:24.181805 25436 ts_tablet_manager.cc:1403] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.110s	user 0.012s	sys 0.007s
I20260709 13:07:24.182622 25435 raft_consensus.cc:359] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.183269 25435 raft_consensus.cc:385] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.183518 25435 raft_consensus.cc:740] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.183667 25437 tablet_bootstrap.cc:492] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:24.184072 25437 ts_tablet_manager.cc:1403] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 13:07:24.184283 25436 raft_consensus.cc:359] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.184209 25435 consensus_queue.cc:260] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.186452 25435 ts_tablet_manager.cc:1434] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 13:07:24.187693 25435 tablet_bootstrap.cc:492] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:24.184942 25436 raft_consensus.cc:385] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.188973 25437 raft_consensus.cc:359] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.189620 25437 raft_consensus.cc:385] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.189353 25436 raft_consensus.cc:740] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.189993 25437 raft_consensus.cc:740] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.190495 25436 consensus_queue.cc:260] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.190788 25437 consensus_queue.cc:260] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.193007 25437 ts_tablet_manager.cc:1434] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 13:07:24.193960 25437 tablet_bootstrap.cc:492] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:24.195464 25436 ts_tablet_manager.cc:1434] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 13:07:24.197048 25436 tablet_bootstrap.cc:492] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:24.203081 25634 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.203678 25634 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.204788 25436 tablet_bootstrap.cc:654] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.205817 25634 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:24.207372 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:24.207998 25246 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:24.208452 25435 tablet_bootstrap.cc:654] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.209504 25359 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:24.210103 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:24.211388 25319 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:24.215526 25634 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:24.216006 25634 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:24.216403 25634 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.220487 25435 tablet_bootstrap.cc:492] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:24.220921 25435 ts_tablet_manager.cc:1403] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.033s	user 0.007s	sys 0.003s
I20260709 13:07:24.223150 25435 raft_consensus.cc:359] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.223618 25435 raft_consensus.cc:385] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.223878 25435 raft_consensus.cc:740] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.223996 25578 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.224454 25578 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.224603 25435 consensus_queue.cc:260] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.226455 25578 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:24.230216 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:24.230845 25246 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:24.231045 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:24.231642 25319 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:24.232198 25359 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:24.233021 25576 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:24.233378 25576 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:24.233685 25576 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.235255 25569 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.235812 25435 ts_tablet_manager.cc:1434] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.015s	user 0.004s	sys 0.000s
I20260709 13:07:24.235713 25569 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.238235 25569 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:24.240025 25576 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.242187 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:24.242480 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:24.243088 25246 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
W20260709 13:07:24.243804 25358 leader_election.cc:343] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.236747 25435 tablet_bootstrap.cc:492] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:24.246311 25359 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:24.248742 25578 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:24.250108 25578 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:24.250458 25578 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.252561 25437 tablet_bootstrap.cc:654] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.256619 25634 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.258774 25578 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.260423 25578 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:24.260481 25634 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:24.261456 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:24.262477 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:24.263036 25321 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.263469 25514 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.264282 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:24.264948 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:24.264885 25246 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.265594 25244 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:24.266330 25358 leader_election.cc:343] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.265249 25514 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.271540 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:24.271785 25321 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:24.264808 25576 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:24.272275 25245 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.272912 25358 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:24.273600 25246 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:24.273862 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:24.274372 25319 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.275123 25514 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:24.276182 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:24.276818 25390 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:24.277873 25212 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:24.278264 25435 tablet_bootstrap.cc:654] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.278620 25514 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:24.278970 25514 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:24.279278 25576 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:24.279745 25576 raft_consensus.cc:697] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:24.280452 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:24.280545 25576 consensus_queue.cc:237] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.282451 25437 tablet_bootstrap.cc:492] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:24.282959 25437 ts_tablet_manager.cc:1403] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.089s	user 0.013s	sys 0.004s
I20260709 13:07:24.279297 25514 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:24.283742 25209 leader_election.cc:343] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.285399 25639 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:24.288409 25697 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.289024 25697 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.290144 25514 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.294621 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:24.295181 25392 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.295455 25319 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:24.296594 25358 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:24.297345 25634 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:24.297802 25634 raft_consensus.cc:697] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:24.298273 25697 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:24.298343 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:24.299103 25246 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 1: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:07:24.298611 25634 consensus_queue.cc:237] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.300535 25244 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:24.306731 25643 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:24.310688 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:24.310811 25359 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
W20260709 13:07:24.312376 25209 leader_election.cc:343] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.312539 25578 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:24.312712 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:24.313422 25578 raft_consensus.cc:697] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:24.314069 25287 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:24.314513 25578 consensus_queue.cc:237] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.315697 25697 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.318955 25437 raft_consensus.cc:359] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.319850 25437 raft_consensus.cc:385] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.320178 25437 raft_consensus.cc:740] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.316994 25245 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:24.323725 25436 tablet_bootstrap.cc:492] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:24.324487 25436 ts_tablet_manager.cc:1403] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.128s	user 0.025s	sys 0.016s
I20260709 13:07:24.329875 25392 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:24.320979 25437 consensus_queue.cc:260] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.329883 25436 raft_consensus.cc:359] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.332882 25436 raft_consensus.cc:385] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.333173 25436 raft_consensus.cc:740] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.333684 25212 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:24.334105 25514 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:24.334416 25436 consensus_queue.cc:260] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.333695 25437 ts_tablet_manager.cc:1434] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.050s	user 0.005s	sys 0.002s
I20260709 13:07:24.339340 25436 ts_tablet_manager.cc:1434] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 13:07:24.339658 25437 tablet_bootstrap.cc:492] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:24.340283 25436 tablet_bootstrap.cc:492] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:24.338680 25435 tablet_bootstrap.cc:492] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:24.344470 25435 ts_tablet_manager.cc:1403] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.108s	user 0.014s	sys 0.026s
I20260709 13:07:24.343235 25141 catalog_manager.cc:5697] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:24.346344 25697 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:24.350296 25435 raft_consensus.cc:359] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.351192 25435 raft_consensus.cc:385] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.351521 25435 raft_consensus.cc:740] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.352262 25435 consensus_queue.cc:260] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.354351 25435 ts_tablet_manager.cc:1434] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.003s
I20260709 13:07:24.359361 25634 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:24.361159 25435 tablet_bootstrap.cc:492] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:24.334645 25514 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:24.373448 25634 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:24.373734 25514 raft_consensus.cc:697] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:24.374564 25514 consensus_queue.cc:237] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.429411 25576 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.430200 25576 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.431771 25319 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.432497 25576 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:24.416218 25141 catalog_manager.cc:5697] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:24.435408 25141 catalog_manager.cc:5697] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:24.436787 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:24.439458 25358 leader_election.cc:343] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.442574 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:24.443199 25245 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:24.444787 25437 tablet_bootstrap.cc:654] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.445704 25359 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:24.449394 25435 tablet_bootstrap.cc:654] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.448179 25578 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:24.448074 25576 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.455533 25578 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:24.459611 25634 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.459849 25578 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.460234 25634 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.464449 25634 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:24.456727 25576 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.468351 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:24.469002 25244 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:24.470340 25359 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:24.472993 25576 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:24.473589 25576 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:24.473953 25576 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:24.475142 25576 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.476673 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:24.477406 25245 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:24.478989 25359 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:24.482240 25634 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:24.482591 25634 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:24.482928 25634 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.486204 25436 tablet_bootstrap.cc:654] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.486821 25144 catalog_manager.cc:5697] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
W20260709 13:07:24.487705 25209 consensus_peers.cc:597] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:24.496963 25317 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.496335 25639 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:07:24.501607 25639 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.505038 25698 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.513213 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:24.515182 25698 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.511368 25639 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:24.516477 25318 raft_consensus.cc:2393] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 2: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:07:24.516352 25634 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.512993 25578 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.518781 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:24.520526 25244 raft_consensus.cc:2410] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260709 13:07:24.521567 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 58666.9us
I20260709 13:07:24.532852 25634 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:24.520156 25698 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:24.536926 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:24.542315 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:24.542800 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:24.542400 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:24.542841 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:24.543521 25395 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:07:24.544414 25578 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:24.546211 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:24.546783 25244 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.541190 25435 tablet_bootstrap.cc:492] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:24.548538 25435 ts_tablet_manager.cc:1403] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.193s	user 0.014s	sys 0.013s
I20260709 13:07:24.550721 25437 tablet_bootstrap.cc:492] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:24.551134 25437 ts_tablet_manager.cc:1403] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.212s	user 0.007s	sys 0.007s
I20260709 13:07:24.552677 25287 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:24.554327 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:24.556531 25245 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.560214 25640 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:07:24.562718 25576 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.566941 25436 tablet_bootstrap.cc:492] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:24.567376 25436 ts_tablet_manager.cc:1403] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.227s	user 0.017s	sys 0.003s
I20260709 13:07:24.567951 25435 raft_consensus.cc:359] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.568601 25435 raft_consensus.cc:385] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.568842 25435 raft_consensus.cc:740] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.569484 25435 consensus_queue.cc:260] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.569787 25436 raft_consensus.cc:359] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.570369 25436 raft_consensus.cc:385] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.570623 25436 raft_consensus.cc:740] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.571007 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:24.571218 25436 consensus_queue.cc:260] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.571542 25246 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.564893 25576 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:24.574353 25212 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:24.575577 25638 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:07:24.575935 25638 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:07:24.576332 25638 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:24.576020 25437 raft_consensus.cc:359] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.576741 25437 raft_consensus.cc:385] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.577003 25437 raft_consensus.cc:740] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
W20260709 13:07:24.577464 25358 leader_election.cc:343] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:24.578414 25358 leader_election.cc:343] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.578637 25435 ts_tablet_manager.cc:1434] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.029s	user 0.005s	sys 0.000s
W20260709 13:07:24.579214 25358 leader_election.cc:343] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.579610 25435 tablet_bootstrap.cc:492] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:24.577665 25437 consensus_queue.cc:260] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.581632 25436 ts_tablet_manager.cc:1434] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.000s
I20260709 13:07:24.589416 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:24.591543 25436 tablet_bootstrap.cc:492] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:24.595028 25639 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:07:24.597587 25358 leader_election.cc:343] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.598902 25437 ts_tablet_manager.cc:1434] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.048s	user 0.006s	sys 0.000s
I20260709 13:07:24.602407 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:24.604274 25246 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:24.605502 25359 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:24.609163 25437 tablet_bootstrap.cc:492] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:24.609817 25723 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:24.608299 25244 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:24.610217 25245 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:24.610306 25723 raft_consensus.cc:697] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:24.599510 25638 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:07:24.612385 25358 leader_election.cc:343] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.618191 25359 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:24.619009 25634 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:24.619408 25514 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.619491 25634 raft_consensus.cc:697] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:24.619481 25359 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:24.619977 25514 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.620595 25634 consensus_queue.cc:237] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.621496 25723 consensus_queue.cc:237] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.622852 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:24.624307 25209 leader_election.cc:343] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.625010 25717 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:24.625838 25717 raft_consensus.cc:697] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:24.626487 25514 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:24.626713 25717 consensus_queue.cc:237] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.635212 25723 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:24.636269 25638 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:24.644886 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:24.646072 25317 raft_consensus.cc:2393] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 2: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:07:24.649470 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:24.650394 25392 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:24.651804 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:24.653316 25723 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:24.655349 25212 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:24.656175 25639 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:24.656497 25639 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:24.658614 25436 tablet_bootstrap.cc:654] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.656876 25639 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.687811 25144 catalog_manager.cc:5697] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:24.695822 25144 catalog_manager.cc:5697] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:24.699569 25435 tablet_bootstrap.cc:654] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.699921 25317 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.697075 25144 catalog_manager.cc:5697] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:24.703498 25717 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.717533 25717 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.727061 25717 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:24.697340 25639 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.715389 25634 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260709 13:07:24.734635 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:24.734935 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:24.736253 25359 leader_election.cc:343] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:24.736984 25358 leader_election.cc:343] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.737387 25358 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:24.740674 25723 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:24.745904 25435 tablet_bootstrap.cc:492] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:24.746343 25435 ts_tablet_manager.cc:1403] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.167s	user 0.015s	sys 0.007s
I20260709 13:07:24.748930 25437 tablet_bootstrap.cc:654] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.750849 25436 tablet_bootstrap.cc:492] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:24.751293 25436 ts_tablet_manager.cc:1403] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.160s	user 0.023s	sys 0.013s
I20260709 13:07:24.758463 25435 raft_consensus.cc:359] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.759130 25435 raft_consensus.cc:385] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.759359 25435 raft_consensus.cc:740] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.760037 25435 consensus_queue.cc:260] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.761004 25634 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:24.763547 25394 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:07:24.771660 25212 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:24.772208 25436 raft_consensus.cc:359] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.772894 25436 raft_consensus.cc:385] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.773183 25436 raft_consensus.cc:740] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.773876 25514 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:24.773789 25436 consensus_queue.cc:260] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.774539 25514 raft_consensus.cc:697] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 2 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:24.775323 25514 consensus_queue.cc:237] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.776121 25436 ts_tablet_manager.cc:1434] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.001s
I20260709 13:07:24.777107 25436 tablet_bootstrap.cc:492] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:24.765398 25435 ts_tablet_manager.cc:1434] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 13:07:24.779923 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:07:24.781260 25209 leader_election.cc:343] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.781396 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:24.781972 25394 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:24.779364 25639 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:24.794003 25639 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.794518 25639 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.795228 25436 tablet_bootstrap.cc:654] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.798205 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:24.784024 25634 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:24.801952 25435 tablet_bootstrap.cc:492] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
W20260709 13:07:24.802911 25209 leader_election.cc:343] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:24.803617 25727 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 13:07:24.811160 25638 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.811640 25638 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.815701 25638 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:24.822376 25437 tablet_bootstrap.cc:492] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:24.822952 25437 ts_tablet_manager.cc:1403] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.214s	user 0.016s	sys 0.002s
I20260709 13:07:24.825755 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:24.826479 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:07:24.827216 25209 leader_election.cc:343] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.818969 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:24.828173 25435 tablet_bootstrap.cc:654] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
W20260709 13:07:24.827981 25212 leader_election.cc:343] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.828655 25212 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:24.829164 25639 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:07:24.831197 25212 leader_election.cc:343] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.831327 25639 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:24.831568 25212 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:24.832424 25639 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:24.836966 25437 raft_consensus.cc:359] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.836923 25723 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 13:07:24.839214 25437 raft_consensus.cc:385] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.839618 25437 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.840396 25437 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.846998 25437 ts_tablet_manager.cc:1434] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.000s
I20260709 13:07:24.854117 25437 tablet_bootstrap.cc:492] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:24.860360 25576 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:24.862336 25436 tablet_bootstrap.cc:492] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:24.862812 25436 ts_tablet_manager.cc:1403] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.086s	user 0.021s	sys 0.030s
I20260709 13:07:24.892503 25642 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.904918 25437 tablet_bootstrap.cc:654] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:24.913964 25436 raft_consensus.cc:359] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.917752 25436 raft_consensus.cc:385] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.918133 25436 raft_consensus.cc:740] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.919013 25436 consensus_queue.cc:260] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.916599 25698 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:07:24.925231 25192 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:07:24.924971 25698 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.908742 25642 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.928714 25642 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:24.942840 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:24.955757 25394 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:24.962065 25435 tablet_bootstrap.cc:492] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:24.962764 25435 ts_tablet_manager.cc:1403] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.173s	user 0.022s	sys 0.060s
I20260709 13:07:24.966856 25436 ts_tablet_manager.cc:1434] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.104s	user 0.012s	sys 0.006s
I20260709 13:07:24.972067 25747 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:24.972560 25747 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.974344 25747 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:07:24.975085 25212 leader_election.cc:343] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:24.975941 25212 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:24.976840 25642 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:24.980580 25436 tablet_bootstrap.cc:492] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:24.983662 25435 raft_consensus.cc:359] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.984398 25435 raft_consensus.cc:385] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:24.984663 25435 raft_consensus.cc:740] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:24.978175 25642 raft_consensus.cc:697] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:24.990167 25642 consensus_queue.cc:237] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:24.995873 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:25.079607 25437 tablet_bootstrap.cc:492] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:25.101666 25437 ts_tablet_manager.cc:1403] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.250s	user 0.041s	sys 0.025s
I20260709 13:07:25.084616 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:25.101979 25698 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:25.099428 25632 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 13:07:24.997401 25318 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:25.104540 25437 raft_consensus.cc:359] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.105209 25437 raft_consensus.cc:385] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.105471 25437 raft_consensus.cc:740] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.106066 25437 consensus_queue.cc:260] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.075618 25435 consensus_queue.cc:260] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.096710 25436 tablet_bootstrap.cc:654] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.110991 25437 ts_tablet_manager.cc:1434] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.001s
I20260709 13:07:25.111949 25437 tablet_bootstrap.cc:492] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:25.116976 25317 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:25.118875 25437 tablet_bootstrap.cc:654] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.122686 25436 tablet_bootstrap.cc:492] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:25.123049 25437 tablet_bootstrap.cc:492] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:25.123109 25436 ts_tablet_manager.cc:1403] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.143s	user 0.007s	sys 0.007s
I20260709 13:07:25.123466 25437 ts_tablet_manager.cc:1403] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.012s	user 0.010s	sys 0.001s
I20260709 13:07:25.125065 25436 raft_consensus.cc:359] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.125677 25436 raft_consensus.cc:385] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.125963 25436 raft_consensus.cc:740] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.125886 25437 raft_consensus.cc:359] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:07:25.126294 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 84555.3us
I20260709 13:07:25.126564 25437 raft_consensus.cc:385] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.126843 25437 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.126678 25436 consensus_queue.cc:260] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.127179 25514 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:25.127446 25437 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.128706 25436 ts_tablet_manager.cc:1434] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20260709 13:07:25.128932 25437 ts_tablet_manager.cc:1434] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20260709 13:07:25.129577 25436 tablet_bootstrap.cc:492] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:25.129801 25437 tablet_bootstrap.cc:492] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:25.130095 25435 ts_tablet_manager.cc:1434] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.167s	user 0.009s	sys 0.027s
I20260709 13:07:25.131368 25435 tablet_bootstrap.cc:492] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
W20260709 13:07:25.131735 25209 leader_election.cc:343] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:25.132117 25209 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:25.133723 25643 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:25.135712 25437 tablet_bootstrap.cc:654] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.137511 25723 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:25.138360 25436 tablet_bootstrap.cc:654] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.141507 25435 tablet_bootstrap.cc:654] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.147229 25318 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:25.153780 25436 tablet_bootstrap.cc:492] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:25.154268 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:25.156890 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:25.160224 25436 ts_tablet_manager.cc:1403] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.031s	user 0.013s	sys 0.005s
I20260709 13:07:25.161811 25632 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:25.162998 25436 raft_consensus.cc:359] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.163657 25436 raft_consensus.cc:385] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.163923 25436 raft_consensus.cc:740] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.164482 25436 consensus_queue.cc:260] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.169515 25436 ts_tablet_manager.cc:1434] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 13:07:25.169397 25144 catalog_manager.cc:5697] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 1 to 2, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 2 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:25.179905 25741 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:25.191708 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:25.193704 25436 tablet_bootstrap.cc:492] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:25.259774 25698 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:25.260609 25437 tablet_bootstrap.cc:492] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:25.261195 25437 ts_tablet_manager.cc:1403] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.132s	user 0.040s	sys 0.020s
I20260709 13:07:25.260550 25698 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.265223 25698 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:25.265941 25437 raft_consensus.cc:359] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.266819 25437 raft_consensus.cc:385] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.267165 25437 raft_consensus.cc:740] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.269007 25437 consensus_queue.cc:260] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.270562 25435 tablet_bootstrap.cc:492] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:25.270867 25437 ts_tablet_manager.cc:1434] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 13:07:25.271149 25435 ts_tablet_manager.cc:1403] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.140s	user 0.015s	sys 0.005s
I20260709 13:07:25.272130 25437 tablet_bootstrap.cc:492] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:25.286309 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:25.287227 25244 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:25.290124 25435 raft_consensus.cc:359] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.290512 25319 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:25.288959 25287 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:25.292177 25287 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:25.292779 25698 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:25.291186 25435 raft_consensus.cc:385] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.293280 25285 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:25.294584 25435 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.295662 25435 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.297851 25436 tablet_bootstrap.cc:654] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.298678 25698 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:25.298806 25746 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:07:25.299114 25698 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:25.299463 25698 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:25.301563 25435 ts_tablet_manager.cc:1434] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.030s	user 0.009s	sys 0.001s
I20260709 13:07:25.310750 25632 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:25.311239 25632 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.322451 25642 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:07:25.332741 25358 consensus_peers.cc:597] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:25.335036 25435 tablet_bootstrap.cc:492] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:25.332154 25634 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 13:07:25.377173 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:25.378160 25245 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:25.381477 25359 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:25.383463 25634 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:25.383898 25634 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:25.384312 25634 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:25.385201 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 74119.6us
I20260709 13:07:25.386150 25632 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:25.392608 25435 tablet_bootstrap.cc:654] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.392817 25576 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:25.399533 25394 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:25.399415 25634 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.402288 25634 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:25.403234 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:25.403803 25245 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:25.407253 25698 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.408977 25437 tablet_bootstrap.cc:654] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.410040 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:25.410600 25244 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:25.411849 25761 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:25.404222 25632 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:25.416523 25632 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.421463 25632 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:25.423703 25244 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:25.424881 25285 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:25.425248 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:25.426002 25241 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:25.426254 25746 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:25.426743 25746 raft_consensus.cc:697] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:25.427706 25746 consensus_queue.cc:237] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.427932 25359 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:25.429684 25576 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:25.430153 25576 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:25.430804 25576 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:25.432403 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 121387us
I20260709 13:07:25.430235 25245 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:25.433305 25632 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:25.433923 25632 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.433943 25359 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:25.435380 25764 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:25.315089 25143 catalog_manager.cc:5697] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:25.437027 25764 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:25.437882 25764 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.443665 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:25.444288 25241 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:25.448536 25359 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:07:25.445026 25358 consensus_peers.cc:597] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:25.449421 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:25.450089 25723 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:25.450089 25395 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:25.450634 25723 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:25.451009 25723 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:25.458139 25723 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.463523 25435 tablet_bootstrap.cc:492] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:25.463999 25435 ts_tablet_manager.cc:1403] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.129s	user 0.010s	sys 0.012s
I20260709 13:07:25.460224 25723 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:25.472034 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:25.473315 25245 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:25.474913 25436 tablet_bootstrap.cc:492] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:25.475348 25436 ts_tablet_manager.cc:1403] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.282s	user 0.019s	sys 0.000s
I20260709 13:07:25.469329 25632 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:25.479204 25746 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:25.482664 25435 raft_consensus.cc:359] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.484954 25435 raft_consensus.cc:385] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.485351 25435 raft_consensus.cc:740] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.488421 25435 consensus_queue.cc:260] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.491322 25435 ts_tablet_manager.cc:1434] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.003s
I20260709 13:07:25.492240 25576 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.492654 25435 tablet_bootstrap.cc:492] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:25.494274 25576 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:25.495180 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:25.495736 25241 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:25.479729 25746 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.504840 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:25.506464 25245 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:25.488960 25698 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:25.509537 25437 tablet_bootstrap.cc:492] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:25.509991 25437 ts_tablet_manager.cc:1403] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.238s	user 0.043s	sys 0.041s
I20260709 13:07:25.510561 25359 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:25.510489 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:25.511610 25632 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:25.512141 25632 raft_consensus.cc:697] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:25.512750 25437 raft_consensus.cc:359] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.519217 25437 raft_consensus.cc:385] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.512988 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:25.519529 25437 raft_consensus.cc:740] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.521683 25632 consensus_queue.cc:237] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.520357 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:25.527141 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:25.527413 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:25.528692 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:25.515022 25436 raft_consensus.cc:359] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.509822 25241 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:25.517799 25746 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:25.524875 25142 catalog_manager.cc:5697] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:25.532573 25320 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:25.536105 25435 tablet_bootstrap.cc:654] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.530138 25436 raft_consensus.cc:385] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.530834 25761 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:25.537251 25436 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.538233 25436 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.541419 25437 consensus_queue.cc:260] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.542758 25761 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.545526 25359 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:25.550529 25436 ts_tablet_manager.cc:1434] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.075s	user 0.005s	sys 0.002s
I20260709 13:07:25.551537 25436 tablet_bootstrap.cc:492] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:25.548457 25764 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:25.553460 25764 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:25.554275 25764 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.557996 25437 ts_tablet_manager.cc:1434] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.048s	user 0.007s	sys 0.000s
I20260709 13:07:25.547714 25317 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:25.583048 25437 tablet_bootstrap.cc:492] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:25.583165 25761 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:25.583698 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:25.616659 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 49432.8us
I20260709 13:07:25.621012 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:25.622128 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:25.622812 25394 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:25.621675 25395 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
W20260709 13:07:25.621393 25358 leader_election.cc:343] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:25.627751 25436 tablet_bootstrap.cc:654] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
W20260709 13:07:25.627959 25358 leader_election.cc:343] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:25.629884 25358 leader_election.cc:343] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:25.630836 25358 leader_election.cc:343] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:25.632267 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:25.632941 25395 raft_consensus.cc:2393] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 1: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:07:25.632697 25212 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:07:25.634531 25209 leader_election.cc:343] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:25.639140 25394 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:25.640509 25745 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:25.640905 25745 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:25.643939 25762 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 13:07:25.648746 25745 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:25.654241 25287 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:25.657269 25698 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:25.671856 25144 catalog_manager.cc:5697] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:25.683422 25437 tablet_bootstrap.cc:654] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.685467 25761 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:25.689118 25723 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:25.689612 25723 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.691548 25723 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:25.691728 25435 tablet_bootstrap.cc:492] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:25.692163 25435 ts_tablet_manager.cc:1403] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.200s	user 0.055s	sys 0.049s
I20260709 13:07:25.722090 25638 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:25.724697 25638 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.725548 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:25.726156 25241 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:25.726809 25638 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:25.728051 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:25.728530 25435 raft_consensus.cc:359] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.729211 25435 raft_consensus.cc:385] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.729466 25435 raft_consensus.cc:740] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
W20260709 13:07:25.730039 25209 leader_election.cc:343] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:25.731160 25359 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:25.730103 25435 consensus_queue.cc:260] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.731145 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:25.733790 25723 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 13:07:25.735443 25212 leader_election.cc:343] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:25.735594 25723 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:25.736335 25723 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:25.736215 25212 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:25.737505 25435 ts_tablet_manager.cc:1434] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.045s	user 0.006s	sys 0.002s
I20260709 13:07:25.733193 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:25.740826 25358 leader_election.cc:343] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:25.743602 25435 tablet_bootstrap.cc:492] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:25.748665 25142 catalog_manager.cc:5697] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:25.749249 25763 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:25.750923 25142 catalog_manager.cc:5697] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:25.754734 25436 tablet_bootstrap.cc:492] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:25.751497 25723 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.755190 25436 ts_tablet_manager.cc:1403] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.204s	user 0.017s	sys 0.008s
I20260709 13:07:25.770002 25436 raft_consensus.cc:359] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.771049 25436 raft_consensus.cc:385] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.771188 25437 tablet_bootstrap.cc:492] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:25.771706 25437 ts_tablet_manager.cc:1403] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.189s	user 0.015s	sys 0.004s
I20260709 13:07:25.771322 25436 raft_consensus.cc:740] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.772969 25436 consensus_queue.cc:260] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.775230 25436 ts_tablet_manager.cc:1434] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.000s
I20260709 13:07:25.776093 25436 tablet_bootstrap.cc:492] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:25.776166 25437 raft_consensus.cc:359] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.776849 25437 raft_consensus.cc:385] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.777112 25437 raft_consensus.cc:740] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.759081 25745 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.788555 25745 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:25.790724 25437 consensus_queue.cc:260] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.792513 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:25.793310 25243 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:25.795403 25436 tablet_bootstrap.cc:654] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.799600 25437 ts_tablet_manager.cc:1434] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.028s	user 0.008s	sys 0.004s
I20260709 13:07:25.800619 25437 tablet_bootstrap.cc:492] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:25.800693 25723 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:25.800122 25435 tablet_bootstrap.cc:654] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.808415 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:25.809289 25390 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:25.812594 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:25.814493 25436 tablet_bootstrap.cc:492] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:25.814944 25436 ts_tablet_manager.cc:1403] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.039s	user 0.011s	sys 0.002s
I20260709 13:07:25.817399 25436 raft_consensus.cc:359] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:07:25.817698 25358 leader_election.cc:343] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:25.819511 25243 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:25.820031 25435 tablet_bootstrap.cc:492] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:25.820468 25435 ts_tablet_manager.cc:1403] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.077s	user 0.011s	sys 0.004s
I20260709 13:07:25.817812 25390 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:25.820513 25359 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:25.822496 25723 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:25.822932 25723 raft_consensus.cc:697] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:25.822969 25435 raft_consensus.cc:359] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.823649 25435 raft_consensus.cc:385] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.823925 25435 raft_consensus.cc:740] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.823800 25723 consensus_queue.cc:237] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.824532 25435 consensus_queue.cc:260] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.826574 25435 ts_tablet_manager.cc:1434] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.003s
I20260709 13:07:25.827647 25435 tablet_bootstrap.cc:492] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:25.827979 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:25.828997 25212 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:25.829700 25745 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:25.830296 25745 raft_consensus.cc:697] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:25.818058 25436 raft_consensus.cc:385] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 13:07:25.830618 25209 leader_election.cc:343] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:25.830374 25358 consensus_peers.cc:597] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:25.830737 25436 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.832145 25436 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.838414 25436 ts_tablet_manager.cc:1434] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.001s
I20260709 13:07:25.839315 25435 tablet_bootstrap.cc:654] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.839363 25436 tablet_bootstrap.cc:492] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:25.833197 25745 consensus_queue.cc:237] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.859385 25436 tablet_bootstrap.cc:654] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.858996 25320 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:25.869251 25437 tablet_bootstrap.cc:654] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.871011 25435 tablet_bootstrap.cc:492] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:25.871479 25435 ts_tablet_manager.cc:1403] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.044s	user 0.005s	sys 0.013s
I20260709 13:07:25.874045 25435 raft_consensus.cc:359] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.874848 25435 raft_consensus.cc:385] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.875093 25435 raft_consensus.cc:740] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.875656 25435 consensus_queue.cc:260] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:07:25.876497 25272 tablet.cc:2406] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 13:07:25.877036 25435 ts_tablet_manager.cc:1434] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:07:25.877911 25435 tablet_bootstrap.cc:492] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:25.883327 25435 tablet_bootstrap.cc:654] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.894726 25435 tablet_bootstrap.cc:492] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:25.895181 25435 ts_tablet_manager.cc:1403] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.017s	user 0.010s	sys 0.005s
I20260709 13:07:25.896889 25437 tablet_bootstrap.cc:492] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:25.897385 25437 ts_tablet_manager.cc:1403] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.097s	user 0.012s	sys 0.000s
I20260709 13:07:25.899258 25435 raft_consensus.cc:359] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.899817 25435 raft_consensus.cc:385] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.900035 25435 raft_consensus.cc:740] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.900017 25437 raft_consensus.cc:359] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.900749 25437 raft_consensus.cc:385] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.901077 25437 raft_consensus.cc:740] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.900784 25435 consensus_queue.cc:260] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.900563 25142 catalog_manager.cc:5697] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:25.901726 25437 consensus_queue.cc:260] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.902966 25436 tablet_bootstrap.cc:492] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:25.903169 25435 ts_tablet_manager.cc:1434] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:07:25.903365 25436 ts_tablet_manager.cc:1403] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.064s	user 0.007s	sys 0.004s
I20260709 13:07:25.903609 25437 ts_tablet_manager.cc:1434] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 13:07:25.904109 25435 tablet_bootstrap.cc:492] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:25.904505 25437 tablet_bootstrap.cc:492] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:25.906260 25436 raft_consensus.cc:359] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.906885 25436 raft_consensus.cc:385] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.907143 25436 raft_consensus.cc:740] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.907822 25436 consensus_queue.cc:260] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.910738 25435 tablet_bootstrap.cc:654] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.914326 25436 ts_tablet_manager.cc:1434] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.005s
I20260709 13:07:25.917029 25436 tablet_bootstrap.cc:492] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:25.919034 25437 tablet_bootstrap.cc:654] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.924825 25436 tablet_bootstrap.cc:654] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.927409 25745 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:25.928309 25745 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.918704 25143 catalog_manager.cc:5697] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:25.936115 25745 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:25.938886 25634 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:25.945318 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:25.946870 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:25.949589 25209 leader_election.cc:343] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:25.951006 25212 leader_election.cc:343] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:25.951400 25212 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:25.952692 25639 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:25.953440 25697 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:07:25.953943 25697 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.962154 25437 tablet_bootstrap.cc:492] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:25.962594 25437 ts_tablet_manager.cc:1403] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.058s	user 0.011s	sys 0.000s
I20260709 13:07:25.967098 25437 raft_consensus.cc:359] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.967576 25437 raft_consensus.cc:385] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.967803 25437 raft_consensus.cc:740] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.968376 25437 consensus_queue.cc:260] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.970381 25437 ts_tablet_manager.cc:1434] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 13:07:25.971289 25437 tablet_bootstrap.cc:492] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:25.971454 25435 tablet_bootstrap.cc:492] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:25.972430 25435 ts_tablet_manager.cc:1403] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.069s	user 0.027s	sys 0.014s
I20260709 13:07:25.972815 25697 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:25.973554 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:25.974918 25435 raft_consensus.cc:359] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.975535 25435 raft_consensus.cc:385] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.975605 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:25.975849 25435 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.976495 25435 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.978639 25435 ts_tablet_manager.cc:1434] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:07:25.980885 25435 tablet_bootstrap.cc:492] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:25.981360 25634 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:25.981948 25287 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:25.983215 25698 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:07:25.983007 25747 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 13:07:25.990321 25435 tablet_bootstrap.cc:654] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:25.993099 25436 tablet_bootstrap.cc:492] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:25.993549 25436 ts_tablet_manager.cc:1403] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.077s	user 0.004s	sys 0.009s
I20260709 13:07:25.995996 25436 raft_consensus.cc:359] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:25.996595 25436 raft_consensus.cc:385] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:25.996851 25436 raft_consensus.cc:740] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:25.997960 25318 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:25.998003 25436 consensus_queue.cc:260] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.001586 25634 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:26.002027 25634 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.003536 25436 ts_tablet_manager.cc:1434] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.002s
I20260709 13:07:26.004089 25634 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:26.004447 25436 tablet_bootstrap.cc:492] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:26.005044 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:26.005669 25241 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:26.006438 25435 tablet_bootstrap.cc:492] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:26.005044 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:26.006862 25435 ts_tablet_manager.cc:1403] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.026s	user 0.014s	sys 0.001s
W20260709 13:07:26.007601 25358 leader_election.cc:343] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.009241 25359 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:26.009521 25435 raft_consensus.cc:359] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.010144 25435 raft_consensus.cc:385] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:26.010504 25436 tablet_bootstrap.cc:654] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:26.010370 25634 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:26.010599 25745 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:26.010949 25634 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:26.011430 25634 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:26.011224 25745 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.013439 25745 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:26.010715 25763 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:26.014822 25437 tablet_bootstrap.cc:654] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:26.014634 25763 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.018748 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:26.020116 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:26.020876 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:07:26.021349 25212 leader_election.cc:343] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.021468 25395 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:26.021896 25763 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:26.022749 25212 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:07:26.023793 25209 leader_election.cc:343] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.025197 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:26.026350 25436 tablet_bootstrap.cc:492] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:26.026027 25639 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:26.025739 25576 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:26.026750 25639 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:26.027094 25639 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:26.028332 25209 leader_election.cc:343] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.028707 25436 ts_tablet_manager.cc:1403] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.024s	user 0.012s	sys 0.001s
I20260709 13:07:26.028748 25209 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:26.031414 25745 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:26.031323 25436 raft_consensus.cc:359] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.031997 25436 raft_consensus.cc:385] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:26.032297 25436 raft_consensus.cc:740] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:26.034534 25437 tablet_bootstrap.cc:492] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:26.035020 25437 ts_tablet_manager.cc:1403] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.064s	user 0.011s	sys 0.000s
I20260709 13:07:26.036319 25634 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.037495 25437 raft_consensus.cc:359] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.038128 25437 raft_consensus.cc:385] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:26.038403 25437 raft_consensus.cc:740] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:26.039014 25437 consensus_queue.cc:260] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.039358 25639 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.041039 25576 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:26.041507 25436 consensus_queue.cc:260] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.010402 25435 raft_consensus.cc:740] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:26.042827 25435 consensus_queue.cc:260] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.043424 25436 ts_tablet_manager.cc:1434] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.000s
I20260709 13:07:26.044399 25436 tablet_bootstrap.cc:492] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:26.044456 25639 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:26.044689 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:26.040977 25437 ts_tablet_manager.cc:1434] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 13:07:26.045919 25437 tablet_bootstrap.cc:492] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
W20260709 13:07:26.046298 25358 leader_election.cc:343] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.047175 25435 ts_tablet_manager.cc:1434] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.040s	user 0.006s	sys 0.001s
I20260709 13:07:26.047192 25634 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:26.049510 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:26.050112 25241 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:26.048182 25435 tablet_bootstrap.cc:492] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:26.048679 25747 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:26.051795 25437 tablet_bootstrap.cc:654] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:26.043452 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:26.056032 25395 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:26.058360 25747 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:26.059122 25436 tablet_bootstrap.cc:654] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:26.058966 25747 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.062772 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:26.064447 25698 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:26.064898 25698 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.067832 25698 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:26.068298 25747 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:26.069432 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:26.079538 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:26.081462 25287 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:26.086028 25436 tablet_bootstrap.cc:492] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:26.085961 25395 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:26.086553 25436 ts_tablet_manager.cc:1403] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.042s	user 0.009s	sys 0.009s
I20260709 13:07:26.087551 25212 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:26.088306 25763 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:26.088784 25763 raft_consensus.cc:697] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:26.089411 25435 tablet_bootstrap.cc:654] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:26.089655 25763 consensus_queue.cc:237] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.080961 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:26.092761 25747 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:07:26.107532 25209 leader_election.cc:343] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.099409 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:26.110441 25436 raft_consensus.cc:359] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.114631 25436 raft_consensus.cc:385] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:26.114957 25436 raft_consensus.cc:740] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:26.125617 25319 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:26.126121 25436 consensus_queue.cc:260] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.123657 25241 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:26.138656 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:26.144371 25436 ts_tablet_manager.cc:1434] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.058s	user 0.003s	sys 0.005s
I20260709 13:07:26.140851 25698 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:26.146292 25698 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:26.146693 25698 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.148564 25698 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:26.149886 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:26.150434 25395 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:26.152529 25287 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:26.155723 25698 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:26.157027 25698 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:26.157435 25698 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:26.190028 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:26.191867 25246 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
W20260709 13:07:26.227883 25192 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:07:26.228729 25436 tablet_bootstrap.cc:492] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:26.236472 25437 tablet_bootstrap.cc:492] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:26.228281 25764 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260709 13:07:26.273003 25746 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:26.278689 25746 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.279874 25437 ts_tablet_manager.cc:1403] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.234s	user 0.032s	sys 0.021s
I20260709 13:07:26.321189 25436 tablet_bootstrap.cc:654] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:26.325139 25437 raft_consensus.cc:359] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.325932 25437 raft_consensus.cc:385] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:26.326206 25437 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:26.326884 25437 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.328521 25437 ts_tablet_manager.cc:1434] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.048s	user 0.003s	sys 0.000s
I20260709 13:07:26.337383 25359 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:26.339506 25634 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:26.340327 25634 raft_consensus.cc:697] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:26.340739 25437 tablet_bootstrap.cc:492] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:26.341323 25634 consensus_queue.cc:237] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.348151 25437 tablet_bootstrap.cc:654] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:26.354036 25435 tablet_bootstrap.cc:492] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:26.354478 25435 ts_tablet_manager.cc:1403] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.307s	user 0.045s	sys 0.034s
W20260709 13:07:26.322283 25802 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.170s	user 0.002s	sys 0.012s
W20260709 13:07:26.350796 25803 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.145s	user 0.000s	sys 0.009s
W20260709 13:07:26.359328 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 52469.2us
I20260709 13:07:26.359884 25698 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.441502 25435 raft_consensus.cc:359] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.434223 25746 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:26.443500 25435 raft_consensus.cc:385] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:26.448153 25435 raft_consensus.cc:740] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:26.465401 25435 consensus_queue.cc:260] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.464478 25818 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:26.481719 25435 ts_tablet_manager.cc:1434] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.127s	user 0.008s	sys 0.000s
I20260709 13:07:26.478305 25698 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:26.488000 25819 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:26.501727 25435 tablet_bootstrap.cc:492] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:26.478657 25818 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.514048 25436 tablet_bootstrap.cc:492] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:26.514668 25436 ts_tablet_manager.cc:1403] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.308s	user 0.017s	sys 0.008s
I20260709 13:07:26.521799 25437 tablet_bootstrap.cc:492] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:26.522063 25317 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:26.522465 25437 ts_tablet_manager.cc:1403] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.182s	user 0.004s	sys 0.017s
I20260709 13:07:26.512287 25763 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:26.525713 25746 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:26.534366 25437 raft_consensus.cc:359] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.534859 25437 raft_consensus.cc:385] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:26.535485 25437 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:26.503925 25819 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.535811 25746 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.544652 25437 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.548804 25436 raft_consensus.cc:359] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.533915 25763 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.549408 25436 raft_consensus.cc:385] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:26.557849 25436 raft_consensus.cc:740] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:26.558512 25436 consensus_queue.cc:260] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.559037 25763 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:26.550414 25818 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:26.560451 25746 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:26.560552 25436 ts_tablet_manager.cc:1434] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.046s	user 0.005s	sys 0.000s
I20260709 13:07:26.562422 25436 tablet_bootstrap.cc:492] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:26.563040 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:26.565511 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:26.566030 25244 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:26.572379 25244 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:26.582167 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:26.583509 25321 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:26.587855 25435 tablet_bootstrap.cc:654] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:26.589661 25817 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:26.586133 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:26.590132 25817 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.596673 25437 ts_tablet_manager.cc:1434] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.074s	user 0.009s	sys 0.000s
I20260709 13:07:26.598315 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:26.599020 25392 raft_consensus.cc:2393] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 1: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:07:26.599021 25209 leader_election.cc:343] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:26.600440 25358 consensus_peers.cc:597] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:26.613770 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:26.614459 25387 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:26.608922 25816 log.cc:927] Time spent T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 13:07:26.620168 25437 tablet_bootstrap.cc:492] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
W20260709 13:07:26.622087 25212 leader_election.cc:343] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.622313 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:26.622457 25212 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:26.621039 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:26.625406 25815 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:26.627910 25819 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:26.623658 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:26.629935 25316 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:26.635125 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:07:26.645396 25209 leader_election.cc:343] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.623677 25817 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:26.636471 25643 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260709 13:07:26.652277 25285 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:26.653343 25747 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:26.653851 25747 raft_consensus.cc:697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:26.654646 25747 consensus_queue.cc:237] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.637813 25828 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.045s
I20260709 13:07:26.657850 25287 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:26.658004 25811 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 13:07:26.658597 25697 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:26.661175 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:26.661979 25241 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:26.665307 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:26.665987 25359 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:26.666152 25763 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:26.667228 25634 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:26.667578 25634 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:07:26.667495 25359 leader_election.cc:343] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.667937 25634 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:26.666647 25763 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.669902 25763 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:07:26.673945 25209 consensus_peers.cc:597] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:26.666664 25576 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 13:07:26.678992 25387 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:26.681396 25436 tablet_bootstrap.cc:654] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
W20260709 13:07:26.681559 25212 leader_election.cc:343] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.682504 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:26.684063 25634 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.684762 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:26.686344 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:26.687376 25437 tablet_bootstrap.cc:654] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:26.695298 25634 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:26.687384 25285 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:26.690420 25143 catalog_manager.cc:5697] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
W20260709 13:07:26.701116 25358 leader_election.cc:343] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:26.702450 25358 leader_election.cc:343] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.702790 25358 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:26.704051 25634 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:26.693557 25639 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:26.714985 25697 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:26.722278 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:26.722895 25243 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:26.738763 25698 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:26.739274 25698 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.741375 25698 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:26.743568 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:07:26.748617 25358 leader_election.cc:343] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.755352 25435 tablet_bootstrap.cc:492] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:26.755928 25435 ts_tablet_manager.cc:1403] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.262s	user 0.014s	sys 0.012s
I20260709 13:07:26.745877 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:26.760164 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:26.760715 25242 raft_consensus.cc:2393] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 1: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:07:26.762205 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:07:26.764822 25212 leader_election.cc:343] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.765683 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:26.765937 25285 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:26.767174 25747 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:26.769265 25243 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
W20260709 13:07:26.770643 25209 leader_election.cc:343] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.772034 25209 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:26.773622 25824 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:26.774273 25359 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:26.775486 25576 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:26.775941 25576 raft_consensus.cc:697] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:26.776760 25576 consensus_queue.cc:237] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.787247 25819 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:26.787679 25819 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.789609 25819 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:26.797832 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:26.799638 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:26.801432 25436 tablet_bootstrap.cc:492] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:26.801884 25436 ts_tablet_manager.cc:1403] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.240s	user 0.015s	sys 0.009s
W20260709 13:07:26.803628 25359 leader_election.cc:343] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.804646 25436 raft_consensus.cc:359] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:07:26.804886 25358 leader_election.cc:343] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.805190 25436 raft_consensus.cc:385] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:26.805469 25358 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:26.805981 25436 raft_consensus.cc:740] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:26.807510 25819 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:26.809923 25435 raft_consensus.cc:359] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.813068 25435 raft_consensus.cc:385] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:26.817253 25436 consensus_queue.cc:260] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.813401 25435 raft_consensus.cc:740] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:26.824221 25436 ts_tablet_manager.cc:1434] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.000s
I20260709 13:07:26.825908 25747 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:26.827490 25436 tablet_bootstrap.cc:492] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:26.830765 25435 consensus_queue.cc:260] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.831815 25437 tablet_bootstrap.cc:492] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:26.832288 25437 ts_tablet_manager.cc:1403] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.227s	user 0.022s	sys 0.007s
I20260709 13:07:26.833019 25435 ts_tablet_manager.cc:1434] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.077s	user 0.006s	sys 0.000s
I20260709 13:07:26.833935 25435 tablet_bootstrap.cc:492] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:26.834766 25437 raft_consensus.cc:359] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.835404 25437 raft_consensus.cc:385] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:26.835654 25437 raft_consensus.cc:740] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:26.836328 25437 consensus_queue.cc:260] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.837764 25437 ts_tablet_manager.cc:1434] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.005s	user 0.007s	sys 0.000s
I20260709 13:07:26.838548 25437 tablet_bootstrap.cc:492] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:26.843590 25435 tablet_bootstrap.cc:654] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:26.843762 25437 tablet_bootstrap.cc:654] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:26.853245 25436 tablet_bootstrap.cc:654] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:26.854919 25828 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:26.855746 25828 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.857858 25828 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:26.859555 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:26.860633 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:07:26.860915 25209 leader_election.cc:343] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.857612 25819 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:26.859136 25144 catalog_manager.cc:5697] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:26.874598 25435 tablet_bootstrap.cc:492] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:26.875583 25435 ts_tablet_manager.cc:1403] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.042s	user 0.010s	sys 0.007s
W20260709 13:07:26.876024 25212 leader_election.cc:343] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.876458 25212 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:26.877513 25828 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:26.878407 25435 raft_consensus.cc:359] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.879076 25435 raft_consensus.cc:385] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:26.879328 25435 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:26.879936 25435 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.881919 25435 ts_tablet_manager.cc:1434] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:07:26.882746 25435 tablet_bootstrap.cc:492] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: Bootstrap starting.
I20260709 13:07:26.887102 25437 tablet_bootstrap.cc:492] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:26.887598 25437 ts_tablet_manager.cc:1403] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.049s	user 0.010s	sys 0.005s
I20260709 13:07:26.888144 25436 tablet_bootstrap.cc:492] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:26.888587 25436 ts_tablet_manager.cc:1403] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.064s	user 0.012s	sys 0.006s
I20260709 13:07:26.890344 25435 tablet_bootstrap.cc:654] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:26.890424 25437 raft_consensus.cc:359] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.887755 25143 catalog_manager.cc:5697] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:26.891240 25436 raft_consensus.cc:359] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.891875 25436 raft_consensus.cc:385] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:26.892124 25436 raft_consensus.cc:740] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:26.892751 25436 consensus_queue.cc:260] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.894817 25436 ts_tablet_manager.cc:1434] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:07:26.895668 25436 tablet_bootstrap.cc:492] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:26.896127 25698 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:26.896565 25698 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.898463 25698 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:26.899132 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:26.899885 25242 raft_consensus.cc:2393] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 1: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:07:26.904884 25435 tablet_bootstrap.cc:492] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: No bootstrap required, opened a new log
I20260709 13:07:26.904297 25142 catalog_manager.cc:5697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:26.905331 25435 ts_tablet_manager.cc:1403] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: Time spent bootstrapping tablet: real 0.023s	user 0.012s	sys 0.005s
I20260709 13:07:26.905843 25436 tablet_bootstrap.cc:654] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:26.907840 25435 raft_consensus.cc:359] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.908491 25435 raft_consensus.cc:385] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:26.909222 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:26.910697 25287 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:26.911367 25698 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:26.908733 25435 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Initialized, Role: FOLLOWER
I20260709 13:07:26.912431 25435 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.914511 25435 ts_tablet_manager.cc:1434] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 13:07:26.918249 25698 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:26.922971 25436 tablet_bootstrap.cc:492] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:26.923431 25436 ts_tablet_manager.cc:1403] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.028s	user 0.005s	sys 0.011s
I20260709 13:07:26.925457 25437 raft_consensus.cc:385] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:26.925801 25437 raft_consensus.cc:740] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:26.926440 25437 consensus_queue.cc:260] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.927642 25576 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:26.928680 25437 ts_tablet_manager.cc:1434] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.041s	user 0.003s	sys 0.000s
I20260709 13:07:26.929592 25437 tablet_bootstrap.cc:492] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:26.931845 25436 raft_consensus.cc:359] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.932511 25436 raft_consensus.cc:385] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:26.932777 25436 raft_consensus.cc:740] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:26.934399 25436 consensus_queue.cc:260] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.936468 25436 ts_tablet_manager.cc:1434] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.001s
I20260709 13:07:26.937316 25436 tablet_bootstrap.cc:492] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:26.936995 25437 tablet_bootstrap.cc:654] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:26.939599 25828 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:26.940167 25828 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.942469 25828 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:26.943368 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:07:26.945111 25209 leader_election.cc:343] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:26.945595 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:26.946219 25391 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:26.947640 25436 tablet_bootstrap.cc:654] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:26.947860 25212 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:26.949164 25828 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:26.950366 25828 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:26.950721 25828 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:26.968153 25828 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:26.978592 25321 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.011322 25741 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:27.011965 25741 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.016108 25741 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:27.021176 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:27.048285 25391 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 0.
I20260709 13:07:27.049479 25212 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:27.049772 25828 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:27.035434 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:27.051241 25828 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:27.051890 25828 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:27.052413 25828 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.037638 25634 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:27.058187 25634 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.059319 25437 tablet_bootstrap.cc:492] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:27.059798 25437 ts_tablet_manager.cc:1403] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.130s	user 0.061s	sys 0.063s
I20260709 13:07:27.059940 25698 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:27.060631 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:27.061323 25242 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:27.061879 25828 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.063050 25437 raft_consensus.cc:359] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.063712 25437 raft_consensus.cc:385] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:27.063891 25828 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:27.063992 25437 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:27.064050 25436 tablet_bootstrap.cc:492] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:27.064739 25436 ts_tablet_manager.cc:1403] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.128s	user 0.026s	sys 0.019s
I20260709 13:07:27.064810 25437 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.065377 25359 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:27.067073 25437 ts_tablet_manager.cc:1434] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.003s
I20260709 13:07:27.068578 25810 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:27.068929 25810 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:27.069308 25810 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.070225 25436 raft_consensus.cc:359] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.070992 25436 raft_consensus.cc:385] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:27.071303 25436 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:27.072396 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:27.069815 25634 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:27.071962 25436 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.074239 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:27.075337 25436 ts_tablet_manager.cc:1434] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 13:07:27.076488 25436 tablet_bootstrap.cc:492] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:27.077528 25810 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.080292 25810 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:27.082410 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:27.082978 25241 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.083705 25437 tablet_bootstrap.cc:492] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:27.083642 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:27.084753 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:27.085316 25390 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.089569 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:27.090106 25393 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.094192 25241 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
W20260709 13:07:27.101904 25209 leader_election.cc:343] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:27.104307 25209 leader_election.cc:343] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:27.105377 25209 leader_election.cc:343] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:27.106531 25359 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:27.108832 25817 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:27.109302 25817 raft_consensus.cc:697] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:27.110097 25817 consensus_queue.cc:237] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.118603 25393 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
W20260709 13:07:27.112110 25358 leader_election.cc:343] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:27.133101 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:27.136785 25212 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:27.142849 25761 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
W20260709 13:07:27.145565 25358 leader_election.cc:343] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:27.143221 25390 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:07:27.148373 25212 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:27.149600 25828 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:27.150050 25828 raft_consensus.cc:697] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:27.150664 25436 tablet_bootstrap.cc:654] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:27.150872 25828 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.144650 25761 raft_consensus.cc:697] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:27.156787 25761 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.163393 25437 tablet_bootstrap.cc:654] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:27.170311 25855 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:27.165169 25144 catalog_manager.cc:5697] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:27.194347 25810 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:27.194834 25810 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.198535 25856 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:27.199142 25856 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.203205 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:27.205806 25243 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:27.208660 25285 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:27.209929 25855 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:27.210350 25855 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:27.210741 25855 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.256794 25436 tablet_bootstrap.cc:492] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:27.257923 25436 ts_tablet_manager.cc:1403] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.182s	user 0.009s	sys 0.004s
I20260709 13:07:27.261721 25634 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:27.278103 25856 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:27.278028 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:27.278347 25855 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.278848 25394 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:27.280413 25855 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:27.281405 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:27.282159 25394 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.280812 25810 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:27.305825 25394 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:27.307571 25287 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:27.311159 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:27.311384 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:27.311807 25240 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:27.313694 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:27.314234 25246 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.317435 25359 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:27.318502 25576 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:27.318825 25576 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:27.319125 25576 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:27.320984 25192 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:07:27.323472 25246 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:27.337247 25437 tablet_bootstrap.cc:492] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:27.337719 25437 ts_tablet_manager.cc:1403] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.254s	user 0.012s	sys 0.014s
I20260709 13:07:27.340296 25437 raft_consensus.cc:359] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.340955 25437 raft_consensus.cc:385] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:27.341269 25437 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:27.324822 25857 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:27.348760 25437 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.351176 25436 raft_consensus.cc:359] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.352131 25436 raft_consensus.cc:385] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:27.352492 25436 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:27.357604 25436 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:07:27.362103 25358 leader_election.cc:343] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:27.364336 25436 ts_tablet_manager.cc:1434] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.106s	user 0.003s	sys 0.004s
I20260709 13:07:27.360095 25576 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.366142 25576 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:27.368438 25144 catalog_manager.cc:5697] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:27.358896 25857 raft_consensus.cc:697] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:27.382061 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:07:27.386399 25358 leader_election.cc:343] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:27.387081 25436 tablet_bootstrap.cc:492] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:27.390990 25437 ts_tablet_manager.cc:1434] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.053s	user 0.006s	sys 0.001s
I20260709 13:07:27.401595 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:27.402499 25246 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.400456 25857 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.407852 25142 catalog_manager.cc:5697] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:27.412691 25437 tablet_bootstrap.cc:492] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:27.416777 25246 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:27.421845 25359 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:27.423942 25819 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:27.424459 25819 raft_consensus.cc:697] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:27.425482 25855 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:27.425381 25819 consensus_queue.cc:237] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.426206 25855 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.429248 25855 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:27.431607 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:27.432425 25244 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:27.434096 25285 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:27.434957 25855 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:27.436980 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:27.438154 25817 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:07:27.442488 25287 leader_election.cc:343] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:27.437511 25855 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:27.453266 25855 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.474709 25144 catalog_manager.cc:5697] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:27.484892 25142 catalog_manager.cc:5697] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:27.499558 25855 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.502156 25437 tablet_bootstrap.cc:654] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:27.504021 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:27.504627 25241 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.505280 25855 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:27.506737 25436 tablet_bootstrap.cc:654] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:27.507400 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:07:27.509209 25287 leader_election.cc:343] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:27.522388 25634 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:27.522541 25241 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:27.523823 25285 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:27.524627 25855 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:27.525188 25855 raft_consensus.cc:697] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:27.526100 25855 consensus_queue.cc:237] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:07:27.541563 25358 consensus_peers.cc:597] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:27.549353 25437 tablet_bootstrap.cc:492] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:27.549855 25437 ts_tablet_manager.cc:1403] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.151s	user 0.010s	sys 0.011s
I20260709 13:07:27.552362 25437 raft_consensus.cc:359] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.553021 25437 raft_consensus.cc:385] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:27.553318 25437 raft_consensus.cc:740] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:27.556511 25437 consensus_queue.cc:260] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.559702 25143 catalog_manager.cc:5697] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:27.564668 25436 tablet_bootstrap.cc:492] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:27.565165 25436 ts_tablet_manager.cc:1403] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.189s	user 0.009s	sys 0.008s
I20260709 13:07:27.567680 25436 raft_consensus.cc:359] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.568334 25436 raft_consensus.cc:385] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:27.568596 25436 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:27.569240 25436 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.570215 25437 ts_tablet_manager.cc:1434] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.000s
I20260709 13:07:27.571136 25437 tablet_bootstrap.cc:492] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61: Bootstrap starting.
I20260709 13:07:27.571516 25436 ts_tablet_manager.cc:1434] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.002s
I20260709 13:07:27.572423 25436 tablet_bootstrap.cc:492] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:27.580053 25863 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:27.583393 25436 tablet_bootstrap.cc:654] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:27.591387 25437 tablet_bootstrap.cc:654] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:27.596053 25436 tablet_bootstrap.cc:492] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:27.596503 25436 ts_tablet_manager.cc:1403] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.024s	user 0.003s	sys 0.011s
I20260709 13:07:27.598920 25436 raft_consensus.cc:359] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.599606 25436 raft_consensus.cc:385] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:27.600855 25436 raft_consensus.cc:740] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:27.601497 25436 consensus_queue.cc:260] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.603497 25436 ts_tablet_manager.cc:1434] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.004s
I20260709 13:07:27.604418 25436 tablet_bootstrap.cc:492] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:27.611826 25436 tablet_bootstrap.cc:654] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:27.612666 25437 tablet_bootstrap.cc:492] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61: No bootstrap required, opened a new log
I20260709 13:07:27.614557 25437 ts_tablet_manager.cc:1403] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent bootstrapping tablet: real 0.044s	user 0.008s	sys 0.014s
I20260709 13:07:27.617039 25437 raft_consensus.cc:359] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.618441 25437 raft_consensus.cc:385] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:27.618719 25437 raft_consensus.cc:740] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Initialized, Role: FOLLOWER
I20260709 13:07:27.619338 25437 consensus_queue.cc:260] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.621596 25437 ts_tablet_manager.cc:1434] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.001s
I20260709 13:07:27.622824 25436 tablet_bootstrap.cc:492] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:27.623272 25436 ts_tablet_manager.cc:1403] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.019s	user 0.014s	sys 0.001s
W20260709 13:07:27.624404 25209 consensus_peers.cc:597] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:27.628867 25576 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:27.629315 25576 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.630659 25436 raft_consensus.cc:359] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.630990 25576 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:27.631351 25436 raft_consensus.cc:385] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:27.631608 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:27.631623 25436 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:27.632212 25246 raft_consensus.cc:2393] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 1: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:07:27.632439 25436 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.634686 25436 ts_tablet_manager.cc:1434] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 13:07:27.635761 25436 tablet_bootstrap.cc:492] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:27.641341 25436 tablet_bootstrap.cc:654] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:27.642237 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:27.645323 25358 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:27.646657 25817 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.653340 25436 tablet_bootstrap.cc:492] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:27.653816 25436 ts_tablet_manager.cc:1403] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.018s	user 0.005s	sys 0.012s
I20260709 13:07:27.656497 25817 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:27.656802 25436 raft_consensus.cc:359] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.657497 25436 raft_consensus.cc:385] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:27.657698 25436 raft_consensus.cc:740] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:27.658310 25436 consensus_queue.cc:260] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.660233 25436 ts_tablet_manager.cc:1434] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 13:07:27.661887 25436 tablet_bootstrap.cc:492] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:27.666385 25436 tablet_bootstrap.cc:654] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:27.670621 25864 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:27.671504 25436 tablet_bootstrap.cc:492] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:27.671900 25436 ts_tablet_manager.cc:1403] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.004s
I20260709 13:07:27.674376 25436 raft_consensus.cc:359] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.674860 25436 raft_consensus.cc:385] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:27.675104 25436 raft_consensus.cc:740] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:27.675721 25436 consensus_queue.cc:260] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.676064 25316 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.677448 25436 ts_tablet_manager.cc:1434] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.005s	user 0.007s	sys 0.000s
I20260709 13:07:27.678550 25436 tablet_bootstrap.cc:492] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567: Bootstrap starting.
I20260709 13:07:27.678926 25855 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:27.679417 25855 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.682097 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:27.682662 25246 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:27.684226 25436 tablet_bootstrap.cc:654] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:27.687458 25285 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:27.688550 25697 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:27.688871 25697 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:27.689195 25697 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.697162 25855 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:27.699900 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:27.700556 25393 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 0.
I20260709 13:07:27.708617 25697 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.711184 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:27.711683 25246 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.714423 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:27.715320 25393 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.715989 25855 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:27.716456 25855 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.717851 25246 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:27.718942 25285 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:27.719720 25857 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:27.720182 25857 raft_consensus.cc:697] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:27.721294 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:27.721055 25857 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.734746 25855 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:27.737170 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:27.737946 25697 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:27.741020 25287 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:27.743779 25697 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:07:27.745113 25863 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:27.747130 25817 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:27.747843 25817 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.750479 25817 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:27.752564 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:07:27.762717 25316 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:27.763988 25358 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:27.766028 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:27.766768 25436 tablet_bootstrap.cc:492] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567: No bootstrap required, opened a new log
I20260709 13:07:27.766852 25866 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:27.767274 25866 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:27.767408 25436 ts_tablet_manager.cc:1403] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567: Time spent bootstrapping tablet: real 0.089s	user 0.041s	sys 0.040s
I20260709 13:07:27.767604 25246 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 0.
I20260709 13:07:27.767894 25143 catalog_manager.cc:5697] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:27.770011 25436 raft_consensus.cc:359] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.770655 25436 raft_consensus.cc:385] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:27.770920 25436 raft_consensus.cc:740] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Initialized, Role: FOLLOWER
I20260709 13:07:27.771525 25436 consensus_queue.cc:260] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.773097 25436 ts_tablet_manager.cc:1434] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:07:27.767647 25866 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.779546 25393 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:07:27.783064 25855 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:27.783440 25855 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.785187 25855 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:27.790681 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:27.790848 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:27.792819 25287 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:27.793713 25855 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:27.795420 25866 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.796974 25866 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:27.797617 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:27.797926 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:27.798197 25241 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.798525 25321 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.804229 25241 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:27.803297 25321 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:07:27.807318 25359 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:27.808254 25866 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:27.808645 25866 raft_consensus.cc:697] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:27.809301 25866 consensus_queue.cc:237] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.818895 25142 catalog_manager.cc:5697] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:27.820696 25316 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.845368 25855 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:27.845759 25855 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.847581 25855 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:27.848198 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:27.848528 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:27.848810 25241 raft_consensus.cc:2393] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 1: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:07:27.849808 25285 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:27.850340 25855 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:27.855854 25855 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:27.860600 25866 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:27.874096 25634 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:27.892076 25855 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:27.901968 25855 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:27.920624 25855 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:27.921059 25855 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:27.924146 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:07:27.924206 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:07:27.925064 25855 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:27.925593 25287 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:27.926482 25855 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:27.969904 25316 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:28.001211 25856 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:28.009912 25857 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:28.044957 25321 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:28.227698 25855 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:28.236850 25855 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:28.335222 25819 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:28.345336 25817 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:28.958262 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:07:28.983882 25900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:28.983939 25899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:28.984356 25902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:28.985724 24113 server_base.cc:1061] running on GCE node
I20260709 13:07:28.987309 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:07:28.987496 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:07:28.987641 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602448987629 us; error 0 us; skew 500 ppm
I20260709 13:07:28.988121 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:07:28.990914 24113 webserver.cc:533] Webserver started at http://127.23.140.68:42305/ using document root <none> and password file <none>
I20260709 13:07:28.991487 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:07:28.991691 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:07:28.992055 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:07:28.993597 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-3-root/instance:
uuid: "f646dfd532974dceadfe00005d2bbf59"
format_stamp: "Formatted at 2026-07-09 13:07:28 on dist-test-slave-k73n"
I20260709 13:07:28.997957 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.004s
I20260709 13:07:29.001245 25907 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:29.001977 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260709 13:07:29.002365 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-3-root
uuid: "f646dfd532974dceadfe00005d2bbf59"
format_stamp: "Formatted at 2026-07-09 13:07:28 on dist-test-slave-k73n"
I20260709 13:07:29.002772 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:07:29.043380 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:07:29.044946 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:07:29.047267 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:07:29.050055 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:07:29.050274 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:29.050523 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:07:29.050714 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:29.112887 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.68:39211
I20260709 13:07:29.113011 25969 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.68:39211 every 8 connection(s)
I20260709 13:07:29.118031 24113 auto_leader_rebalancer-test.cc:384] add a tserver: f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:29.132635 25970 heartbeater.cc:344] Connected to a master server at 127.23.140.126:33305
I20260709 13:07:29.133067 25970 heartbeater.cc:461] Registering TS with master...
I20260709 13:07:29.133816 25970 heartbeater.cc:507] Master 127.23.140.126:33305 requested a full tablet report, sending...
I20260709 13:07:29.135743 25143 ts_manager.cc:194] Registered new tserver with Master: f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:07:29.136998 25143 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53264
I20260709 13:07:29.464751 25241 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:29.470949 25316 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:29.471273 25391 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:29.472906 25863 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:29.475140 25863 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:29.484061 25762 raft_consensus.cc:2955] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:29.486270 25321 raft_consensus.cc:2955] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:29.487681 25391 raft_consensus.cc:2955] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
W20260709 13:07:29.492159 25209 consensus_peers.cc:597] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 991bcf9b91a14242837dfb4e18891e91. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:29.504235 25142 catalog_manager.cc:5697] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:29.521785 25393 consensus_queue.cc:237] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:29.541222 25316 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:29.543004 25241 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:29.545254 25988 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:29.547236 25898 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:07:29.555076 25358 consensus_peers.cc:597] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98122796f52e40968d1d8ccd7b032b32. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:29.566577 25817 raft_consensus.cc:2955] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:29.567265 25241 raft_consensus.cc:2955] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:29.569568 25316 raft_consensus.cc:2955] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:29.582252 25246 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
W20260709 13:07:29.593588 25209 consensus_peers.cc:597] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17ef4786898b4ad696baed95b4209788. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:29.594420 25316 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:29.594420 25393 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:29.591082 25143 catalog_manager.cc:5697] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:29.595790 25981 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:29.597446 25762 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:29.607524 25981 raft_consensus.cc:2955] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:29.609436 25316 raft_consensus.cc:2955] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:29.622673 25393 raft_consensus.cc:2955] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:29.629921 25391 consensus_queue.cc:237] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:29.643334 25143 catalog_manager.cc:5697] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
W20260709 13:07:29.676851 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 56442.1us
I20260709 13:07:29.684540 25246 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:07:29.684669 25358 consensus_peers.cc:597] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b5c9f65605b4dfca1bbfe0f09629c6d. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:29.685417 25315 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:29.687077 25817 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:29.688571 25817 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:29.707748 25817 raft_consensus.cc:2955] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:29.709460 25246 raft_consensus.cc:2955] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:29.720530 25316 raft_consensus.cc:2955] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:29.729447 25142 catalog_manager.cc:5697] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:29.945784 26000 ts_tablet_manager.cc:933] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:29.982260 26000 tablet_copy_client.cc:323] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.67:34661
I20260709 13:07:29.995064 25405 tablet_copy_service.cc:140] P a2a77ecf2c0941618b5bc79d4c04db61: Received BeginTabletCopySession request for tablet 98122796f52e40968d1d8ccd7b032b32 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:41830)
I20260709 13:07:29.995663 25405 tablet_copy_service.cc:161] P a2a77ecf2c0941618b5bc79d4c04db61: Beginning new tablet copy session on tablet 98122796f52e40968d1d8ccd7b032b32 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:41830: session id = f646dfd532974dceadfe00005d2bbf59-98122796f52e40968d1d8ccd7b032b32
I20260709 13:07:30.003685 25405 tablet_copy_source_session.cc:215] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:30.007885 26000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98122796f52e40968d1d8ccd7b032b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:30.021555 26000 tablet_copy_client.cc:806] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:30.022150 26000 tablet_copy_client.cc:670] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:30.026752 26000 tablet_copy_client.cc:538] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:30.042786 26000 tablet_bootstrap.cc:492] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:30.062901 26000 tablet_bootstrap.cc:492] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:30.063891 26000 tablet_bootstrap.cc:492] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:30.064527 26000 ts_tablet_manager.cc:1403] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.022s	user 0.020s	sys 0.000s
I20260709 13:07:30.067683 26000 raft_consensus.cc:359] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:30.068516 26000 raft_consensus.cc:740] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:30.069159 26000 consensus_queue.cc:260] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:30.071064 25970 heartbeater.cc:499] Master 127.23.140.126:33305 was elected leader, sending a full tablet report...
I20260709 13:07:30.071655 26000 ts_tablet_manager.cc:1434] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:07:30.074609 25405 tablet_copy_service.cc:342] P a2a77ecf2c0941618b5bc79d4c04db61: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-98122796f52e40968d1d8ccd7b032b32 received from {username='slave'} at 127.0.0.1:41830
I20260709 13:07:30.075103 25405 tablet_copy_service.cc:434] P a2a77ecf2c0941618b5bc79d4c04db61: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-98122796f52e40968d1d8ccd7b032b32 on tablet 98122796f52e40968d1d8ccd7b032b32 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:30.078589 26006 ts_tablet_manager.cc:933] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:07:30.092628 26000 ts_tablet_manager.cc:933] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:07:30.095777 26005 ts_tablet_manager.cc:933] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:30.097474 26000 tablet_copy_client.cc:323] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.65:45313
I20260709 13:07:30.102802 26006 tablet_copy_client.cc:323] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.65:45313
I20260709 13:07:30.105662 26005 tablet_copy_client.cc:323] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.67:34661
I20260709 13:07:30.107707 25405 tablet_copy_service.cc:140] P a2a77ecf2c0941618b5bc79d4c04db61: Received BeginTabletCopySession request for tablet 9b5c9f65605b4dfca1bbfe0f09629c6d from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:41830)
I20260709 13:07:30.108215 25405 tablet_copy_service.cc:161] P a2a77ecf2c0941618b5bc79d4c04db61: Beginning new tablet copy session on tablet 9b5c9f65605b4dfca1bbfe0f09629c6d from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:41830: session id = f646dfd532974dceadfe00005d2bbf59-9b5c9f65605b4dfca1bbfe0f09629c6d
I20260709 13:07:30.115199 25405 tablet_copy_source_session.cc:215] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:30.118198 26005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b5c9f65605b4dfca1bbfe0f09629c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:30.123485 25256 tablet_copy_service.cc:140] P 46de6919f85341c6a9b574dff112e35a: Received BeginTabletCopySession request for tablet 991bcf9b91a14242837dfb4e18891e91 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:44954)
I20260709 13:07:30.124096 25256 tablet_copy_service.cc:161] P 46de6919f85341c6a9b574dff112e35a: Beginning new tablet copy session on tablet 991bcf9b91a14242837dfb4e18891e91 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:44954: session id = f646dfd532974dceadfe00005d2bbf59-991bcf9b91a14242837dfb4e18891e91
I20260709 13:07:30.129906 25255 tablet_copy_service.cc:140] P 46de6919f85341c6a9b574dff112e35a: Received BeginTabletCopySession request for tablet 17ef4786898b4ad696baed95b4209788 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:44954)
I20260709 13:07:30.130506 25255 tablet_copy_service.cc:161] P 46de6919f85341c6a9b574dff112e35a: Beginning new tablet copy session on tablet 17ef4786898b4ad696baed95b4209788 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:44954: session id = f646dfd532974dceadfe00005d2bbf59-17ef4786898b4ad696baed95b4209788
I20260709 13:07:30.131141 25256 tablet_copy_source_session.cc:215] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:30.135171 26005 tablet_copy_client.cc:806] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:30.135716 26005 tablet_copy_client.cc:670] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:30.136390 26000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 991bcf9b91a14242837dfb4e18891e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:30.140069 25255 tablet_copy_source_session.cc:215] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:30.141230 26005 tablet_copy_client.cc:538] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:30.143573 26006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17ef4786898b4ad696baed95b4209788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:30.150494 26005 tablet_bootstrap.cc:492] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:30.152582 26000 tablet_copy_client.cc:806] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:30.153115 26000 tablet_copy_client.cc:670] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:30.157054 26000 tablet_copy_client.cc:538] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:30.163688 26006 tablet_copy_client.cc:806] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:30.164429 26006 tablet_copy_client.cc:670] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:30.166254 26000 tablet_bootstrap.cc:492] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:30.168525 26006 tablet_copy_client.cc:538] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:30.179016 26006 tablet_bootstrap.cc:492] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:30.197345 26000 tablet_bootstrap.cc:492] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:30.198196 26000 tablet_bootstrap.cc:492] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:30.198808 26000 ts_tablet_manager.cc:1403] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.033s	user 0.013s	sys 0.011s
I20260709 13:07:30.201567 26005 tablet_bootstrap.cc:492] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:30.202515 26005 tablet_bootstrap.cc:492] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:30.203135 26005 ts_tablet_manager.cc:1403] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.053s	user 0.014s	sys 0.014s
I20260709 13:07:30.209251 26006 tablet_bootstrap.cc:492] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:30.210148 26006 tablet_bootstrap.cc:492] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:30.211024 26006 ts_tablet_manager.cc:1403] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.032s	user 0.021s	sys 0.008s
I20260709 13:07:30.212399 26005 raft_consensus.cc:359] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:30.201611 26000 raft_consensus.cc:359] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:30.213296 26005 raft_consensus.cc:740] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:30.213407 26000 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:30.214345 26006 raft_consensus.cc:359] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:30.214455 26005 consensus_queue.cc:260] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:30.214524 26000 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:30.215368 26006 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:30.226626 26006 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:30.216758 26005 ts_tablet_manager.cc:1434] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.000s
I20260709 13:07:30.240661 25405 tablet_copy_service.cc:342] P a2a77ecf2c0941618b5bc79d4c04db61: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-9b5c9f65605b4dfca1bbfe0f09629c6d received from {username='slave'} at 127.0.0.1:41830
I20260709 13:07:30.241160 25405 tablet_copy_service.cc:434] P a2a77ecf2c0941618b5bc79d4c04db61: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-9b5c9f65605b4dfca1bbfe0f09629c6d on tablet 9b5c9f65605b4dfca1bbfe0f09629c6d with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:30.250581 26006 ts_tablet_manager.cc:1434] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.039s	user 0.005s	sys 0.001s
I20260709 13:07:30.267333 25255 tablet_copy_service.cc:342] P 46de6919f85341c6a9b574dff112e35a: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-17ef4786898b4ad696baed95b4209788 received from {username='slave'} at 127.0.0.1:44954
I20260709 13:07:30.267814 25255 tablet_copy_service.cc:434] P 46de6919f85341c6a9b574dff112e35a: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-17ef4786898b4ad696baed95b4209788 on tablet 17ef4786898b4ad696baed95b4209788 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:30.269106 26000 ts_tablet_manager.cc:1434] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.070s	user 0.007s	sys 0.003s
I20260709 13:07:30.270802 25255 tablet_copy_service.cc:342] P 46de6919f85341c6a9b574dff112e35a: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-991bcf9b91a14242837dfb4e18891e91 received from {username='slave'} at 127.0.0.1:44954
I20260709 13:07:30.271181 25255 tablet_copy_service.cc:434] P 46de6919f85341c6a9b574dff112e35a: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-991bcf9b91a14242837dfb4e18891e91 on tablet 991bcf9b91a14242837dfb4e18891e91 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:30.439694 25945 raft_consensus.cc:1217] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:30.500615 25945 raft_consensus.cc:1217] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:30.629784 25945 raft_consensus.cc:1217] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:30.683753 25945 raft_consensus.cc:1217] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:31.043798 25898 raft_consensus.cc:1064] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:31.046476 25898 consensus_queue.cc:237] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:31.058226 25945 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:31.058454 25246 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:31.059870 25988 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:31.060166 25316 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:31.061105 25898 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:31.063017 25997 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:31.078150 25995 raft_consensus.cc:1064] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:31.080601 25995 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:31.096455 25316 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:31.096832 25944 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:31.097652 25390 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:31.098829 25863 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:31.100366 25863 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:31.103176 25997 raft_consensus.cc:2955] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.105907 25863 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:31.106165 25945 raft_consensus.cc:2955] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.139796 25144 catalog_manager.cc:5697] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:31.145293 25988 raft_consensus.cc:1064] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:31.150971 25988 consensus_queue.cc:237] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:31.152474 25246 raft_consensus.cc:2955] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.167142 25321 raft_consensus.cc:2955] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:07:31.174808 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 75042.1us
I20260709 13:07:31.196111 25863 raft_consensus.cc:1064] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:31.200237 25995 raft_consensus.cc:2955] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.202531 25246 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:31.203979 25997 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:31.198767 25863 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:31.209843 25945 raft_consensus.cc:2955] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.214413 25316 raft_consensus.cc:2955] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.221308 25314 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:31.221741 25142 catalog_manager.cc:5697] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:31.224170 25393 raft_consensus.cc:2955] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.225574 25944 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:31.232647 25944 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:31.234213 26022 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:31.234655 25762 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:31.235366 25390 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:31.238631 25314 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:31.239451 25995 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:31.235988 26022 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:31.244418 25244 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:31.245921 26020 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:31.253448 25390 consensus_queue.cc:237] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:31.266875 25942 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:31.266952 25315 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:31.265666 25945 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:31.269627 25863 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:31.274111 25997 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:31.276472 25995 raft_consensus.cc:2955] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.280290 25762 raft_consensus.cc:2955] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.282325 25314 raft_consensus.cc:2955] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.288498 25129 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 991bcf9b91a14242837dfb4e18891e91 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:31.340481 25318 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:31.293206 26022 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:07:31.343485 25945 raft_consensus.cc:2955] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:07:31.325507 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 51282.8us
W20260709 13:07:31.346395 25129 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 57963.8us
I20260709 13:07:31.351613 26024 raft_consensus.cc:2955] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.360469 26020 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:31.360344 25942 raft_consensus.cc:2955] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.384166 25988 raft_consensus.cc:2955] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.391932 25143 catalog_manager.cc:5697] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:31.397645 25316 raft_consensus.cc:2955] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.407369 25131 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 98122796f52e40968d1d8ccd7b032b32 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:31.408649 25142 catalog_manager.cc:5697] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.423110 25378 tablet_service.cc:1558] Processing DeleteTablet for tablet 991bcf9b91a14242837dfb4e18891e91 with delete_type TABLET_DATA_TOMBSTONED (TS a2a77ecf2c0941618b5bc79d4c04db61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60924
I20260709 13:07:31.429031 25942 raft_consensus.cc:2955] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.431329 25144 catalog_manager.cc:5697] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:31.444134 26041 tablet_replica.cc:333] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:07:31.448431 25945 raft_consensus.cc:2955] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.452212 25241 raft_consensus.cc:2955] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.458843 25316 raft_consensus.cc:2955] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.469681 25394 raft_consensus.cc:2955] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.471037 25321 raft_consensus.cc:2955] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.481736 25140 catalog_manager.cc:5697] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.501201 25230 tablet_service.cc:1558] Processing DeleteTablet for tablet 98122796f52e40968d1d8ccd7b032b32 with delete_type TABLET_DATA_TOMBSTONED (TS 46de6919f85341c6a9b574dff112e35a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32990
I20260709 13:07:31.502696 26041 raft_consensus.cc:2243] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:31.505174 26041 raft_consensus.cc:2272] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:31.505537 26042 tablet_replica.cc:333] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:07:31.506639 26042 raft_consensus.cc:2243] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:31.507372 26042 raft_consensus.cc:2272] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:31.508708 26041 ts_tablet_manager.cc:1916] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:31.512290 26042 ts_tablet_manager.cc:1916] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:31.535249 26042 ts_tablet_manager.cc:1929] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:31.535813 26042 log.cc:1199] T 98122796f52e40968d1d8ccd7b032b32 P 46de6919f85341c6a9b574dff112e35a: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-0-root/wals/98122796f52e40968d1d8ccd7b032b32
I20260709 13:07:31.539671 25129 catalog_manager.cc:5028] TS 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): tablet 98122796f52e40968d1d8ccd7b032b32 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:31.543179 26041 ts_tablet_manager.cc:1929] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:31.543696 26041 log.cc:1199] T 991bcf9b91a14242837dfb4e18891e91 P a2a77ecf2c0941618b5bc79d4c04db61: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-2-root/wals/991bcf9b91a14242837dfb4e18891e91
I20260709 13:07:31.545550 25131 catalog_manager.cc:5028] TS a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): tablet 991bcf9b91a14242837dfb4e18891e91 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:31.549737 25241 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:31.555737 25394 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:31.555737 25942 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:31.557453 26038 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:31.558403 26040 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:31.561024 25394 consensus_queue.cc:237] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:31.570111 25995 raft_consensus.cc:2955] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.572914 25945 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:31.573086 25318 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:31.573316 25942 raft_consensus.cc:2955] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.578163 25129 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 17ef4786898b4ad696baed95b4209788 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:31.578907 25898 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:31.580719 25898 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:31.583812 25394 raft_consensus.cc:2955] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.585724 25144 catalog_manager.cc:5697] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.593878 26022 raft_consensus.cc:2955] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.595593 25314 raft_consensus.cc:2955] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.605225 25131 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9b5c9f65605b4dfca1bbfe0f09629c6d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:31.613658 25942 raft_consensus.cc:2955] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.616765 25141 catalog_manager.cc:5697] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:31.618039 25310 tablet_service.cc:1558] Processing DeleteTablet for tablet 17ef4786898b4ad696baed95b4209788 with delete_type TABLET_DATA_TOMBSTONED (TS 2329e428ebbf484fa3ea9452133df567 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36848
I20260709 13:07:31.627898 26043 tablet_replica.cc:333] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:07:31.628880 26043 raft_consensus.cc:2243] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:31.629582 26043 raft_consensus.cc:2272] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:31.631613 25230 tablet_service.cc:1558] Processing DeleteTablet for tablet 9b5c9f65605b4dfca1bbfe0f09629c6d with delete_type TABLET_DATA_TOMBSTONED (TS 46de6919f85341c6a9b574dff112e35a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32990
I20260709 13:07:31.633538 26042 tablet_replica.cc:333] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:07:31.634471 26042 raft_consensus.cc:2243] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:31.635175 26042 raft_consensus.cc:2272] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:31.636909 26043 ts_tablet_manager.cc:1916] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:31.640646 26042 ts_tablet_manager.cc:1916] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:31.653986 26042 ts_tablet_manager.cc:1929] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:31.654325 26042 log.cc:1199] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 46de6919f85341c6a9b574dff112e35a: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-0-root/wals/9b5c9f65605b4dfca1bbfe0f09629c6d
I20260709 13:07:31.655514 25129 catalog_manager.cc:5028] TS 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): tablet 9b5c9f65605b4dfca1bbfe0f09629c6d (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:31.656188 26043 ts_tablet_manager.cc:1929] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:31.656598 26043 log.cc:1199] T 17ef4786898b4ad696baed95b4209788 P 2329e428ebbf484fa3ea9452133df567: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-1-root/wals/17ef4786898b4ad696baed95b4209788
I20260709 13:07:31.657992 25128 catalog_manager.cc:5028] TS 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): tablet 17ef4786898b4ad696baed95b4209788 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:32.705518 25241 consensus_queue.cc:237] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
W20260709 13:07:32.711115 25209 consensus_peers.cc:597] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56914a99b2d8417592e1f08979cfadf6. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:32.711514 25314 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:32.711884 25393 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:32.712913 26045 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:32.713730 26038 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:32.720475 26045 raft_consensus.cc:2955] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:32.721911 25393 raft_consensus.cc:2955] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:32.723948 25314 raft_consensus.cc:2955] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:32.742338 25140 catalog_manager.cc:5697] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:32.763412 25241 consensus_queue.cc:237] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:32.771183 25394 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:07:32.771965 25209 consensus_peers.cc:597] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f209a6539f54de28906492f142efb6b. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:32.772574 26050 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:32.772835 25318 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:32.775955 26050 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:32.807548 26038 raft_consensus.cc:2955] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:32.815177 25314 raft_consensus.cc:2955] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:32.830140 25394 raft_consensus.cc:2955] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:32.853628 25142 catalog_manager.cc:5697] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:32.878615 25314 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:32.886678 25394 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:32.888479 25855 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:32.894671 25246 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:32.895938 25855 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:32.899955 25697 raft_consensus.cc:2955] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:32.906703 25394 raft_consensus.cc:2955] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:32.906915 25243 raft_consensus.cc:2955] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:32.917822 25246 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:32.927270 25142 catalog_manager.cc:5697] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:32.939447 25394 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:07:32.940023 25209 consensus_peers.cc:597] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69fee8cad2c84e1492403c23786622a1. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:32.941303 26045 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:32.950788 26038 raft_consensus.cc:2955] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:32.953156 25394 raft_consensus.cc:2955] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:32.969369 25140 catalog_manager.cc:5697] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:32.972705 25316 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:07:32.976236 25284 consensus_peers.cc:597] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 898e00cfe03c4fc68dc9d6e92a55e996. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:32.983320 26051 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:32.991003 25314 raft_consensus.cc:2955] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:33.135690 26070 ts_tablet_manager.cc:933] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:07:33.137550 26070 tablet_copy_client.cc:323] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.65:45313
I20260709 13:07:33.139005 25255 tablet_copy_service.cc:140] P 46de6919f85341c6a9b574dff112e35a: Received BeginTabletCopySession request for tablet 56914a99b2d8417592e1f08979cfadf6 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:44954)
I20260709 13:07:33.139513 25255 tablet_copy_service.cc:161] P 46de6919f85341c6a9b574dff112e35a: Beginning new tablet copy session on tablet 56914a99b2d8417592e1f08979cfadf6 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:44954: session id = f646dfd532974dceadfe00005d2bbf59-56914a99b2d8417592e1f08979cfadf6
I20260709 13:07:33.146535 25255 tablet_copy_source_session.cc:215] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:33.149487 26070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56914a99b2d8417592e1f08979cfadf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:33.160916 26070 tablet_copy_client.cc:806] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:33.161552 26070 tablet_copy_client.cc:670] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:33.165325 26070 tablet_copy_client.cc:538] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:33.180035 26070 tablet_bootstrap.cc:492] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:33.181804 26072 ts_tablet_manager.cc:933] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:07:33.183874 26072 tablet_copy_client.cc:323] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.65:45313
I20260709 13:07:33.185612 25255 tablet_copy_service.cc:140] P 46de6919f85341c6a9b574dff112e35a: Received BeginTabletCopySession request for tablet 8f209a6539f54de28906492f142efb6b from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:44954)
I20260709 13:07:33.186131 25255 tablet_copy_service.cc:161] P 46de6919f85341c6a9b574dff112e35a: Beginning new tablet copy session on tablet 8f209a6539f54de28906492f142efb6b from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:44954: session id = f646dfd532974dceadfe00005d2bbf59-8f209a6539f54de28906492f142efb6b
I20260709 13:07:33.195010 25255 tablet_copy_source_session.cc:215] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:33.198125 26072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f209a6539f54de28906492f142efb6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:33.201546 26070 tablet_bootstrap.cc:492] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:33.202311 26070 tablet_bootstrap.cc:492] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:33.202925 26070 ts_tablet_manager.cc:1403] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.023s	user 0.010s	sys 0.009s
I20260709 13:07:33.205631 26070 raft_consensus.cc:359] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:33.206393 26070 raft_consensus.cc:740] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:33.207005 26070 consensus_queue.cc:260] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:33.210435 26070 ts_tablet_manager.cc:1434] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:07:33.213660 25255 tablet_copy_service.cc:342] P 46de6919f85341c6a9b574dff112e35a: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-56914a99b2d8417592e1f08979cfadf6 received from {username='slave'} at 127.0.0.1:44954
I20260709 13:07:33.214063 25255 tablet_copy_service.cc:434] P 46de6919f85341c6a9b574dff112e35a: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-56914a99b2d8417592e1f08979cfadf6 on tablet 56914a99b2d8417592e1f08979cfadf6 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:33.214069 26072 tablet_copy_client.cc:806] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:33.215924 26072 tablet_copy_client.cc:670] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:33.219889 26072 tablet_copy_client.cc:538] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:33.228073 26072 tablet_bootstrap.cc:492] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:33.248471 26072 tablet_bootstrap.cc:492] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:33.249346 26072 tablet_bootstrap.cc:492] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:33.249953 26072 ts_tablet_manager.cc:1403] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.022s	user 0.022s	sys 0.000s
I20260709 13:07:33.253705 26072 raft_consensus.cc:359] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:33.254516 26072 raft_consensus.cc:740] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:33.255131 26072 consensus_queue.cc:260] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:33.257504 26072 ts_tablet_manager.cc:1434] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:07:33.259503 25255 tablet_copy_service.cc:342] P 46de6919f85341c6a9b574dff112e35a: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-8f209a6539f54de28906492f142efb6b received from {username='slave'} at 127.0.0.1:44954
I20260709 13:07:33.259904 25255 tablet_copy_service.cc:434] P 46de6919f85341c6a9b574dff112e35a: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-8f209a6539f54de28906492f142efb6b on tablet 8f209a6539f54de28906492f142efb6b with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:33.337356 26072 ts_tablet_manager.cc:933] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:33.338971 26072 tablet_copy_client.cc:323] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.66:42315
I20260709 13:07:33.350050 25331 tablet_copy_service.cc:140] P 2329e428ebbf484fa3ea9452133df567: Received BeginTabletCopySession request for tablet 898e00cfe03c4fc68dc9d6e92a55e996 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:40928)
I20260709 13:07:33.350641 25331 tablet_copy_service.cc:161] P 2329e428ebbf484fa3ea9452133df567: Beginning new tablet copy session on tablet 898e00cfe03c4fc68dc9d6e92a55e996 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:40928: session id = f646dfd532974dceadfe00005d2bbf59-898e00cfe03c4fc68dc9d6e92a55e996
I20260709 13:07:33.358001 25331 tablet_copy_source_session.cc:215] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:33.363799 26072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898e00cfe03c4fc68dc9d6e92a55e996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:33.384584 26072 tablet_copy_client.cc:806] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:33.385160 26072 tablet_copy_client.cc:670] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:33.388806 26072 tablet_copy_client.cc:538] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:33.402230 26072 tablet_bootstrap.cc:492] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:33.423265 26072 tablet_bootstrap.cc:492] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:33.424198 26072 tablet_bootstrap.cc:492] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:33.424907 26072 ts_tablet_manager.cc:1403] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.023s	user 0.010s	sys 0.007s
I20260709 13:07:33.425194 26070 ts_tablet_manager.cc:933] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:07:33.428079 26072 raft_consensus.cc:359] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:33.428901 26072 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:33.429563 26072 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:33.431874 26072 ts_tablet_manager.cc:1434] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.004s
I20260709 13:07:33.432080 26070 tablet_copy_client.cc:323] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.65:45313
I20260709 13:07:33.434468 25331 tablet_copy_service.cc:342] P 2329e428ebbf484fa3ea9452133df567: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-898e00cfe03c4fc68dc9d6e92a55e996 received from {username='slave'} at 127.0.0.1:40928
I20260709 13:07:33.434887 25331 tablet_copy_service.cc:434] P 2329e428ebbf484fa3ea9452133df567: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-898e00cfe03c4fc68dc9d6e92a55e996 on tablet 898e00cfe03c4fc68dc9d6e92a55e996 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:33.436034 25255 tablet_copy_service.cc:140] P 46de6919f85341c6a9b574dff112e35a: Received BeginTabletCopySession request for tablet 69fee8cad2c84e1492403c23786622a1 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:44954)
I20260709 13:07:33.436506 25255 tablet_copy_service.cc:161] P 46de6919f85341c6a9b574dff112e35a: Beginning new tablet copy session on tablet 69fee8cad2c84e1492403c23786622a1 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:44954: session id = f646dfd532974dceadfe00005d2bbf59-69fee8cad2c84e1492403c23786622a1
I20260709 13:07:33.445447 25255 tablet_copy_source_session.cc:215] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:33.448443 26070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69fee8cad2c84e1492403c23786622a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:33.466666 26070 tablet_copy_client.cc:806] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:33.467293 26070 tablet_copy_client.cc:670] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:33.483788 26070 tablet_copy_client.cc:538] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:33.491947 26070 tablet_bootstrap.cc:492] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:33.572436 25942 raft_consensus.cc:1217] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:33.605468 26070 tablet_bootstrap.cc:492] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:33.606948 26070 tablet_bootstrap.cc:492] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:33.609210 26070 ts_tablet_manager.cc:1403] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.117s	user 0.019s	sys 0.013s
I20260709 13:07:33.613467 26070 raft_consensus.cc:359] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:33.615211 26070 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:33.616637 26070 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:33.635370 26070 ts_tablet_manager.cc:1434] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.026s	user 0.009s	sys 0.001s
I20260709 13:07:33.644447 25255 tablet_copy_service.cc:342] P 46de6919f85341c6a9b574dff112e35a: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-69fee8cad2c84e1492403c23786622a1 received from {username='slave'} at 127.0.0.1:44954
I20260709 13:07:33.645174 25255 tablet_copy_service.cc:434] P 46de6919f85341c6a9b574dff112e35a: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-69fee8cad2c84e1492403c23786622a1 on tablet 69fee8cad2c84e1492403c23786622a1 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:33.699640 25942 raft_consensus.cc:1217] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:33.795397 25942 raft_consensus.cc:1217] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:33.872125 25942 raft_consensus.cc:1217] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:34.218178 26038 raft_consensus.cc:1064] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:34.220801 26038 consensus_queue.cc:237] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:34.236445 26038 raft_consensus.cc:1064] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:34.237041 25942 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:34.238795 26038 consensus_queue.cc:237] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:34.239400 26084 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:34.241511 25315 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:34.251250 26084 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:34.279088 25387 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260709 13:07:34.282840 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 45461us
I20260709 13:07:34.302289 26089 raft_consensus.cc:1064] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:34.305533 26089 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:34.329160 25942 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:34.332314 25316 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:34.334867 26045 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:34.337781 26098 raft_consensus.cc:2955] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.340574 25855 raft_consensus.cc:1064] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:34.341080 26045 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:34.343873 25855 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:34.348016 25316 raft_consensus.cc:2955] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.359907 25945 raft_consensus.cc:2955] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.384859 26092 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:34.393162 25945 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:34.393600 25318 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:34.398841 25944 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:34.399931 25387 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:34.400085 25243 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:34.410118 25855 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:34.411118 26085 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:34.412904 25697 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:34.432430 25394 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:34.439445 25390 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:34.440281 25144 catalog_manager.cc:5697] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.441669 26085 raft_consensus.cc:2955] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.446403 26098 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:34.448941 26084 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:34.457360 26084 raft_consensus.cc:2955] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.464951 25390 raft_consensus.cc:2955] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.467088 26098 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:34.468892 26045 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:34.476449 25244 raft_consensus.cc:2955] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.482276 25387 raft_consensus.cc:2955] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.496246 25942 raft_consensus.cc:2955] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.497457 25142 catalog_manager.cc:5697] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:34.508673 25944 raft_consensus.cc:2955] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.529263 25394 raft_consensus.cc:2955] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.543284 25144 catalog_manager.cc:5697] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.546923 26102 raft_consensus.cc:2955] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.553081 25244 consensus_queue.cc:237] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:34.557717 25387 raft_consensus.cc:2955] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.563388 25314 raft_consensus.cc:2955] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.570010 25942 raft_consensus.cc:2955] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.574901 25316 raft_consensus.cc:2955] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.583778 25144 catalog_manager.cc:5697] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.604530 25942 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:34.607229 25314 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:34.610471 26092 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:34.611594 26084 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:34.619779 26084 raft_consensus.cc:2955] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.623703 25942 raft_consensus.cc:2955] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.629475 25129 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8f209a6539f54de28906492f142efb6b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:34.636977 25140 catalog_manager.cc:5697] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.638114 25244 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:34.642982 25387 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:34.644871 25942 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:34.647027 26099 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:07:34.648010 25244 consensus_queue.cc:237] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:34.650187 25378 tablet_service.cc:1558] Processing DeleteTablet for tablet 8f209a6539f54de28906492f142efb6b with delete_type TABLET_DATA_TOMBSTONED (TS a2a77ecf2c0941618b5bc79d4c04db61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60924
I20260709 13:07:34.650527 26045 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:34.659981 26098 raft_consensus.cc:2955] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.661800 25393 raft_consensus.cc:2955] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.665155 25390 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:34.665725 25942 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:34.667296 26099 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:07:34.669168 26045 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:34.671957 25129 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 69fee8cad2c84e1492403c23786622a1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:34.673421 25942 raft_consensus.cc:2955] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.678540 25314 raft_consensus.cc:2955] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.689396 25140 catalog_manager.cc:5697] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.692044 26045 raft_consensus.cc:2955] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.698567 25944 raft_consensus.cc:2955] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.703709 25387 raft_consensus.cc:2955] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.715873 25310 tablet_service.cc:1558] Processing DeleteTablet for tablet 69fee8cad2c84e1492403c23786622a1 with delete_type TABLET_DATA_TOMBSTONED (TS 2329e428ebbf484fa3ea9452133df567 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36848
I20260709 13:07:34.719619 25129 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 56914a99b2d8417592e1f08979cfadf6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:34.733814 25140 catalog_manager.cc:5697] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.735731 26121 tablet_replica.cc:333] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:07:34.738583 26121 raft_consensus.cc:2243] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:34.739423 26121 raft_consensus.cc:2272] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:34.751175 26121 ts_tablet_manager.cc:1916] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:34.751526 25310 tablet_service.cc:1558] Processing DeleteTablet for tablet 56914a99b2d8417592e1f08979cfadf6 with delete_type TABLET_DATA_TOMBSTONED (TS 2329e428ebbf484fa3ea9452133df567 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36848
I20260709 13:07:34.753371 26120 tablet_replica.cc:333] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:07:34.756145 26120 raft_consensus.cc:2243] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:34.756889 26120 raft_consensus.cc:2272] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:34.773033 26121 ts_tablet_manager.cc:1929] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:34.774672 26120 ts_tablet_manager.cc:1916] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:34.775099 26121 log.cc:1199] T 69fee8cad2c84e1492403c23786622a1 P 2329e428ebbf484fa3ea9452133df567: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-1-root/wals/69fee8cad2c84e1492403c23786622a1
I20260709 13:07:34.777022 25128 catalog_manager.cc:5028] TS 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): tablet 69fee8cad2c84e1492403c23786622a1 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:34.778193 26121 tablet_replica.cc:333] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:07:34.779249 26121 raft_consensus.cc:2243] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:34.779958 26121 raft_consensus.cc:2272] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:34.783377 26121 ts_tablet_manager.cc:1916] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:34.790395 26120 ts_tablet_manager.cc:1929] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:34.790845 26120 log.cc:1199] T 8f209a6539f54de28906492f142efb6b P a2a77ecf2c0941618b5bc79d4c04db61: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-2-root/wals/8f209a6539f54de28906492f142efb6b
I20260709 13:07:34.792780 25131 catalog_manager.cc:5028] TS a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): tablet 8f209a6539f54de28906492f142efb6b (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:34.796326 26121 ts_tablet_manager.cc:1929] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:34.796693 26121 log.cc:1199] T 56914a99b2d8417592e1f08979cfadf6 P 2329e428ebbf484fa3ea9452133df567: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-1-root/wals/56914a99b2d8417592e1f08979cfadf6
I20260709 13:07:34.797957 25128 catalog_manager.cc:5028] TS 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): tablet 56914a99b2d8417592e1f08979cfadf6 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:34.866815 25314 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:34.871377 25944 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:34.871942 25387 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:34.872895 26111 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:07:34.875011 26085 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:34.880142 26111 raft_consensus.cc:2955] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.882031 25944 raft_consensus.cc:2955] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.890765 25387 raft_consensus.cc:2955] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:34.905385 25128 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 898e00cfe03c4fc68dc9d6e92a55e996 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:34.908897 25142 catalog_manager.cc:5697] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:34.928774 25230 tablet_service.cc:1558] Processing DeleteTablet for tablet 898e00cfe03c4fc68dc9d6e92a55e996 with delete_type TABLET_DATA_TOMBSTONED (TS 46de6919f85341c6a9b574dff112e35a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32990
I20260709 13:07:34.931332 26122 tablet_replica.cc:333] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:07:34.933175 26122 raft_consensus.cc:2243] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:34.934020 26122 raft_consensus.cc:2272] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:34.937469 26122 ts_tablet_manager.cc:1916] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:34.949450 26122 ts_tablet_manager.cc:1929] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:34.949813 26122 log.cc:1199] T 898e00cfe03c4fc68dc9d6e92a55e996 P 46de6919f85341c6a9b574dff112e35a: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-0-root/wals/898e00cfe03c4fc68dc9d6e92a55e996
I20260709 13:07:34.951239 25129 catalog_manager.cc:5028] TS 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): tablet 898e00cfe03c4fc68dc9d6e92a55e996 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
W20260709 13:07:35.101315 25267 debug-util.cc:398] Leaking SignalData structure 0x7b08001f8fc0 after lost signal to thread 24116
W20260709 13:07:35.102231 25267 debug-util.cc:398] Leaking SignalData structure 0x7b08000ca360 after lost signal to thread 25178
W20260709 13:07:35.103147 25267 debug-util.cc:398] Leaking SignalData structure 0x7b08002f2820 after lost signal to thread 25270
W20260709 13:07:35.103950 25267 debug-util.cc:398] Leaking SignalData structure 0x7b080030f6e0 after lost signal to thread 25345
W20260709 13:07:35.104725 25267 debug-util.cc:398] Leaking SignalData structure 0x7b08001eb6c0 after lost signal to thread 25419
W20260709 13:07:35.105536 25267 debug-util.cc:398] Leaking SignalData structure 0x7b0800306200 after lost signal to thread 25969
I20260709 13:07:36.010015 25243 consensus_queue.cc:237] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
W20260709 13:07:36.020531 25209 consensus_peers.cc:597] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4d43163cfdd4a4094e92fe68bc9259a. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:36.020875 25393 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:36.022161 25318 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:36.023496 26123 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:36.024299 26045 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:36.056895 26098 raft_consensus.cc:2955] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:36.059621 25393 raft_consensus.cc:2955] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:36.062083 25318 raft_consensus.cc:2955] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:36.073900 25387 consensus_queue.cc:237] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:36.080031 25142 catalog_manager.cc:5697] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:36.108186 25316 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:36.111068 26082 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:07:36.116540 25358 consensus_peers.cc:597] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46ab4fd91d3f4766a7b97848f5244360. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:36.121068 25244 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:36.123241 26090 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:36.141582 26133 raft_consensus.cc:2955] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:36.149730 25243 raft_consensus.cc:2955] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:36.155396 25318 raft_consensus.cc:2955] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:36.158635 25393 consensus_queue.cc:237] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:36.170032 25144 catalog_manager.cc:5697] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:36.182911 25243 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:36.184264 26082 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:36.185438 25318 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:07:36.185071 25358 consensus_peers.cc:597] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6d19448306e4052a13c08a582f4ccfa. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:36.189719 26082 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:36.232678 26133 raft_consensus.cc:2955] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:36.239568 25244 raft_consensus.cc:2955] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:36.252885 25318 consensus_queue.cc:237] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:36.263181 25394 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:07:36.263733 25284 consensus_peers.cc:597] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7555c0d2bb294af19f602573931ceb90. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:36.266352 25243 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:36.265530 25144 catalog_manager.cc:5697] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:36.267724 26085 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:36.268621 26111 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:36.290328 25246 raft_consensus.cc:2955] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:36.300462 25318 raft_consensus.cc:2955] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:36.287137 26085 raft_consensus.cc:2955] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:36.340093 25394 raft_consensus.cc:2955] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:36.354717 25140 catalog_manager.cc:5697] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:36.418473 26147 ts_tablet_manager.cc:933] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:07:36.420908 26147 tablet_copy_client.cc:323] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.65:45313
I20260709 13:07:36.422722 25255 tablet_copy_service.cc:140] P 46de6919f85341c6a9b574dff112e35a: Received BeginTabletCopySession request for tablet d4d43163cfdd4a4094e92fe68bc9259a from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:44954)
I20260709 13:07:36.423265 25255 tablet_copy_service.cc:161] P 46de6919f85341c6a9b574dff112e35a: Beginning new tablet copy session on tablet d4d43163cfdd4a4094e92fe68bc9259a from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:44954: session id = f646dfd532974dceadfe00005d2bbf59-d4d43163cfdd4a4094e92fe68bc9259a
I20260709 13:07:36.504746 25255 tablet_copy_source_session.cc:215] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:36.510707 26147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4d43163cfdd4a4094e92fe68bc9259a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:36.560021 26147 tablet_copy_client.cc:806] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:36.560727 26147 tablet_copy_client.cc:670] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:36.564810 26147 tablet_copy_client.cc:538] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:36.580664 26149 ts_tablet_manager.cc:933] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:36.581667 26147 tablet_bootstrap.cc:492] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:36.589658 26150 ts_tablet_manager.cc:933] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:36.597434 26149 tablet_copy_client.cc:323] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.67:34661
I20260709 13:07:36.599574 25405 tablet_copy_service.cc:140] P a2a77ecf2c0941618b5bc79d4c04db61: Received BeginTabletCopySession request for tablet b6d19448306e4052a13c08a582f4ccfa from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:41830)
I20260709 13:07:36.600099 25405 tablet_copy_service.cc:161] P a2a77ecf2c0941618b5bc79d4c04db61: Beginning new tablet copy session on tablet b6d19448306e4052a13c08a582f4ccfa from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:41830: session id = f646dfd532974dceadfe00005d2bbf59-b6d19448306e4052a13c08a582f4ccfa
I20260709 13:07:36.607452 25405 tablet_copy_source_session.cc:215] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:36.610781 26149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d19448306e4052a13c08a582f4ccfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:36.614053 26150 tablet_copy_client.cc:323] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.67:34661
I20260709 13:07:36.619127 25405 tablet_copy_service.cc:140] P a2a77ecf2c0941618b5bc79d4c04db61: Received BeginTabletCopySession request for tablet 46ab4fd91d3f4766a7b97848f5244360 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:41830)
I20260709 13:07:36.619699 25405 tablet_copy_service.cc:161] P a2a77ecf2c0941618b5bc79d4c04db61: Beginning new tablet copy session on tablet 46ab4fd91d3f4766a7b97848f5244360 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:41830: session id = f646dfd532974dceadfe00005d2bbf59-46ab4fd91d3f4766a7b97848f5244360
I20260709 13:07:36.626600 25405 tablet_copy_source_session.cc:215] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:36.630113 26150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46ab4fd91d3f4766a7b97848f5244360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:36.641970 26149 tablet_copy_client.cc:806] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:36.642581 26149 tablet_copy_client.cc:670] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:36.650393 26149 tablet_copy_client.cc:538] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:36.654667 26150 tablet_copy_client.cc:806] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:36.655246 26150 tablet_copy_client.cc:670] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:36.659488 26150 tablet_copy_client.cc:538] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:36.669690 26147 tablet_bootstrap.cc:492] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:36.670612 26147 tablet_bootstrap.cc:492] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:36.670778 26149 tablet_bootstrap.cc:492] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:36.671241 26147 ts_tablet_manager.cc:1403] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.092s	user 0.025s	sys 0.009s
I20260709 13:07:36.674620 26147 raft_consensus.cc:359] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:36.675414 26147 raft_consensus.cc:740] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:36.676051 26147 consensus_queue.cc:260] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:36.678422 26150 tablet_bootstrap.cc:492] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:36.708698 26147 ts_tablet_manager.cc:1434] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.037s	user 0.019s	sys 0.011s
I20260709 13:07:36.706506 26156 ts_tablet_manager.cc:933] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:36.718739 26149 tablet_bootstrap.cc:492] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:36.723935 26149 tablet_bootstrap.cc:492] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:36.724669 26149 ts_tablet_manager.cc:1403] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.054s	user 0.019s	sys 0.009s
I20260709 13:07:36.725085 25255 tablet_copy_service.cc:342] P 46de6919f85341c6a9b574dff112e35a: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-d4d43163cfdd4a4094e92fe68bc9259a received from {username='slave'} at 127.0.0.1:44954
I20260709 13:07:36.725576 25255 tablet_copy_service.cc:434] P 46de6919f85341c6a9b574dff112e35a: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-d4d43163cfdd4a4094e92fe68bc9259a on tablet d4d43163cfdd4a4094e92fe68bc9259a with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:36.727597 26149 raft_consensus.cc:359] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:36.728494 26149 raft_consensus.cc:740] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:36.741557 26150 tablet_bootstrap.cc:492] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:36.742547 26150 tablet_bootstrap.cc:492] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:36.743438 26150 ts_tablet_manager.cc:1403] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.065s	user 0.024s	sys 0.001s
I20260709 13:07:36.744264 26149 consensus_queue.cc:260] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:36.747138 26156 tablet_copy_client.cc:323] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.66:42315
I20260709 13:07:36.746887 26150 raft_consensus.cc:359] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:36.747748 26150 raft_consensus.cc:740] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:36.748428 26150 consensus_queue.cc:260] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:36.749004 25331 tablet_copy_service.cc:140] P 2329e428ebbf484fa3ea9452133df567: Received BeginTabletCopySession request for tablet 7555c0d2bb294af19f602573931ceb90 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:40928)
I20260709 13:07:36.749557 25331 tablet_copy_service.cc:161] P 2329e428ebbf484fa3ea9452133df567: Beginning new tablet copy session on tablet 7555c0d2bb294af19f602573931ceb90 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:40928: session id = f646dfd532974dceadfe00005d2bbf59-7555c0d2bb294af19f602573931ceb90
I20260709 13:07:36.751151 26150 ts_tablet_manager.cc:1434] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:07:36.752200 26149 ts_tablet_manager.cc:1434] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.027s	user 0.007s	sys 0.000s
I20260709 13:07:36.754136 25405 tablet_copy_service.cc:342] P a2a77ecf2c0941618b5bc79d4c04db61: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-b6d19448306e4052a13c08a582f4ccfa received from {username='slave'} at 127.0.0.1:41830
I20260709 13:07:36.754566 25405 tablet_copy_service.cc:434] P a2a77ecf2c0941618b5bc79d4c04db61: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-b6d19448306e4052a13c08a582f4ccfa on tablet b6d19448306e4052a13c08a582f4ccfa with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:36.757248 25331 tablet_copy_source_session.cc:215] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:36.757335 25405 tablet_copy_service.cc:342] P a2a77ecf2c0941618b5bc79d4c04db61: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-46ab4fd91d3f4766a7b97848f5244360 received from {username='slave'} at 127.0.0.1:41830
I20260709 13:07:36.757895 25405 tablet_copy_service.cc:434] P a2a77ecf2c0941618b5bc79d4c04db61: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-46ab4fd91d3f4766a7b97848f5244360 on tablet 46ab4fd91d3f4766a7b97848f5244360 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:36.761628 26156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7555c0d2bb294af19f602573931ceb90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:36.796058 26156 tablet_copy_client.cc:806] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:36.796624 26156 tablet_copy_client.cc:670] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:36.803277 25942 raft_consensus.cc:1217] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:36.820822 26156 tablet_copy_client.cc:538] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:36.831321 26156 tablet_bootstrap.cc:492] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:36.853399 26156 tablet_bootstrap.cc:492] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:36.854296 26156 tablet_bootstrap.cc:492] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:36.854954 26156 ts_tablet_manager.cc:1403] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.024s	user 0.015s	sys 0.007s
I20260709 13:07:36.857999 26156 raft_consensus.cc:359] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:36.858856 26156 raft_consensus.cc:740] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:36.859544 26156 consensus_queue.cc:260] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:36.874094 26156 ts_tablet_manager.cc:1434] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20260709 13:07:36.875789 25331 tablet_copy_service.cc:342] P 2329e428ebbf484fa3ea9452133df567: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-7555c0d2bb294af19f602573931ceb90 received from {username='slave'} at 127.0.0.1:40928
I20260709 13:07:36.876226 25331 tablet_copy_service.cc:434] P 2329e428ebbf484fa3ea9452133df567: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-7555c0d2bb294af19f602573931ceb90 on tablet 7555c0d2bb294af19f602573931ceb90 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:36.969719 25942 raft_consensus.cc:1217] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:37.166898 25942 raft_consensus.cc:1217] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:37.287484 26129 raft_consensus.cc:1064] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:37.293247 26129 consensus_queue.cc:237] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:37.301450 25942 raft_consensus.cc:1217] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:37.307247 25944 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:37.312772 25314 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:37.318306 26098 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:37.319382 26159 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:37.326609 25393 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:37.337672 26129 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260709 13:07:37.374500 26129 raft_consensus.cc:2955] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.381505 25393 raft_consensus.cc:2955] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.386307 25318 raft_consensus.cc:2955] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.412979 25144 catalog_manager.cc:5697] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.418152 26140 raft_consensus.cc:1064] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:37.420629 26140 consensus_queue.cc:237] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:37.438251 25942 raft_consensus.cc:2955] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.447160 25316 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:37.448627 25944 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:37.452922 25244 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:37.453996 26168 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:37.455986 26082 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:37.456939 26133 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:37.498373 26168 raft_consensus.cc:2955] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.502985 25318 raft_consensus.cc:2955] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.513526 25244 consensus_queue.cc:237] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:37.521860 25141 catalog_manager.cc:5697] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.545384 25318 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:37.548609 25944 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:37.549404 26166 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:37.550825 26129 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:37.560153 26166 raft_consensus.cc:2955] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.565044 25944 raft_consensus.cc:2955] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:07:37.569548 26176 log.cc:927] Time spent T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 13:07:37.571182 25318 raft_consensus.cc:2955] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.577843 25241 raft_consensus.cc:2955] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.580269 25141 catalog_manager.cc:5697] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.591869 25378 tablet_service.cc:1558] Processing DeleteTablet for tablet d4d43163cfdd4a4094e92fe68bc9259a with delete_type TABLET_DATA_TOMBSTONED (TS a2a77ecf2c0941618b5bc79d4c04db61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60924
I20260709 13:07:37.606112 26177 tablet_replica.cc:333] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:07:37.607188 26177 raft_consensus.cc:2243] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:37.607913 26177 raft_consensus.cc:2272] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:37.610772 26177 ts_tablet_manager.cc:1916] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:37.621785 25393 consensus_queue.cc:237] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:37.632112 26177 ts_tablet_manager.cc:1929] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:37.632601 26177 log.cc:1199] T d4d43163cfdd4a4094e92fe68bc9259a P a2a77ecf2c0941618b5bc79d4c04db61: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-2-root/wals/d4d43163cfdd4a4094e92fe68bc9259a
I20260709 13:07:37.634405 25131 catalog_manager.cc:5028] TS a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): tablet d4d43163cfdd4a4094e92fe68bc9259a (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:37.596474 25944 raft_consensus.cc:2955] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.652172 25129 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet d4d43163cfdd4a4094e92fe68bc9259a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:37.672487 25316 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:37.676818 26082 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:37.680783 26168 raft_consensus.cc:1064] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:37.683559 25942 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:37.690362 26170 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:37.691597 26168 consensus_queue.cc:237] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:37.694021 26082 raft_consensus.cc:2955] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.695830 25318 raft_consensus.cc:2955] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.700584 25942 raft_consensus.cc:2955] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.703392 25131 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 46ab4fd91d3f4766a7b97848f5244360 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:37.727990 25142 catalog_manager.cc:5697] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:37.748090 25230 tablet_service.cc:1558] Processing DeleteTablet for tablet 46ab4fd91d3f4766a7b97848f5244360 with delete_type TABLET_DATA_TOMBSTONED (TS 46de6919f85341c6a9b574dff112e35a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32990
I20260709 13:07:37.748270 25241 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:37.748653 25316 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:37.750972 26140 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:37.752936 26140 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:37.754225 25942 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:37.764256 26140 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:37.780949 26179 tablet_replica.cc:333] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:07:37.782579 26179 raft_consensus.cc:2243] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:37.785710 26179 raft_consensus.cc:2272] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:37.791178 26179 ts_tablet_manager.cc:1916] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:37.796416 26168 raft_consensus.cc:2955] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.798372 25318 raft_consensus.cc:2955] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.808243 25942 raft_consensus.cc:2955] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.816876 25142 catalog_manager.cc:5697] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.834939 26111 raft_consensus.cc:1064] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:37.837491 26111 consensus_queue.cc:237] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:37.841176 25241 raft_consensus.cc:2955] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.849303 25390 consensus_queue.cc:237] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:37.855794 25944 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:37.858753 26082 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:37.869072 26179 ts_tablet_manager.cc:1929] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:37.869617 26179 log.cc:1199] T 46ab4fd91d3f4766a7b97848f5244360 P 46de6919f85341c6a9b574dff112e35a: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-0-root/wals/46ab4fd91d3f4766a7b97848f5244360
I20260709 13:07:37.871205 25129 catalog_manager.cc:5028] TS 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): tablet 46ab4fd91d3f4766a7b97848f5244360 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:37.878371 25944 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:37.880560 25240 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:37.884032 26085 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:37.887624 25390 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:37.885093 26185 raft_consensus.cc:2955] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.893778 25942 raft_consensus.cc:2955] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.885488 26184 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260709 13:07:37.908669 26085 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:37.933653 25131 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet b6d19448306e4052a13c08a582f4ccfa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:37.936048 25142 catalog_manager.cc:5697] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:07:37.942828 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 52812.5us
I20260709 13:07:37.960839 25310 tablet_service.cc:1558] Processing DeleteTablet for tablet b6d19448306e4052a13c08a582f4ccfa with delete_type TABLET_DATA_TOMBSTONED (TS 2329e428ebbf484fa3ea9452133df567 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36848
I20260709 13:07:37.963703 26111 raft_consensus.cc:2955] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.968333 25944 raft_consensus.cc:2955] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:37.985869 25244 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:37.988497 26140 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:37.994552 25142 catalog_manager.cc:5697] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:38.016358 25390 raft_consensus.cc:2955] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:38.022866 26192 tablet_replica.cc:333] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:07:38.023871 26192 raft_consensus.cc:2243] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:38.027678 25244 raft_consensus.cc:2955] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:38.020920 25241 raft_consensus.cc:2955] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:38.048877 25316 consensus_queue.cc:237] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:38.052796 26192 raft_consensus.cc:2272] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:38.064384 25944 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:38.067507 26192 ts_tablet_manager.cc:1916] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:38.067612 26186 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:07:38.071059 25243 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:38.084137 26085 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:38.089648 25944 raft_consensus.cc:2955] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:38.111435 25241 raft_consensus.cc:2955] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:38.114269 26192 ts_tablet_manager.cc:1929] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:38.114693 26192 log.cc:1199] T b6d19448306e4052a13c08a582f4ccfa P 2329e428ebbf484fa3ea9452133df567: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-1-root/wals/b6d19448306e4052a13c08a582f4ccfa
I20260709 13:07:38.116101 25128 catalog_manager.cc:5028] TS 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): tablet b6d19448306e4052a13c08a582f4ccfa (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:38.087800 26186 raft_consensus.cc:2955] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:38.127701 25140 catalog_manager.cc:5697] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:38.138242 25128 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 7555c0d2bb294af19f602573931ceb90 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:38.151330 25378 tablet_service.cc:1558] Processing DeleteTablet for tablet 7555c0d2bb294af19f602573931ceb90 with delete_type TABLET_DATA_TOMBSTONED (TS a2a77ecf2c0941618b5bc79d4c04db61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60924
I20260709 13:07:38.156634 26193 tablet_replica.cc:333] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:07:38.157671 26193 raft_consensus.cc:2243] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:38.158264 26193 raft_consensus.cc:2272] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:38.161026 26193 ts_tablet_manager.cc:1916] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:38.174178 26193 ts_tablet_manager.cc:1929] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:38.174582 26193 log.cc:1199] T 7555c0d2bb294af19f602573931ceb90 P a2a77ecf2c0941618b5bc79d4c04db61: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-2-root/wals/7555c0d2bb294af19f602573931ceb90
I20260709 13:07:38.176370 25131 catalog_manager.cc:5028] TS a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): tablet 7555c0d2bb294af19f602573931ceb90 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:39.244216 25244 consensus_queue.cc:237] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
W20260709 13:07:39.250348 25209 consensus_peers.cc:597] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 993e3b05a73a4403b04c3b55a5b48abe. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:39.251554 25390 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:39.253713 26166 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:39.254528 25314 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:39.256022 26166 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:39.270830 26162 raft_consensus.cc:2955] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:39.274979 25314 raft_consensus.cc:2955] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:39.286537 25390 raft_consensus.cc:2955] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:39.290464 25316 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:39.297583 25141 catalog_manager.cc:5697] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 13:07:39.341840 25284 consensus_peers.cc:597] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2dc23779efbf4518a6ada8bde8243a11. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:39.342392 25390 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:39.343849 26111 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:39.344199 25244 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:39.345547 26111 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:39.371287 26111 raft_consensus.cc:2955] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:39.374578 25241 raft_consensus.cc:2955] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:39.416188 25390 raft_consensus.cc:2955] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:39.438932 25390 consensus_queue.cc:237] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:39.451809 25140 catalog_manager.cc:5697] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:39.458058 25316 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:39.458243 25241 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:07:39.458516 25358 consensus_peers.cc:597] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3cff7b4cb7184e4c9a69372d5901e19a. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:39.459640 26140 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:39.460759 26082 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:39.471683 26209 raft_consensus.cc:2955] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:39.473260 25316 raft_consensus.cc:2955] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:39.475571 25241 raft_consensus.cc:2955] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:39.488535 25140 catalog_manager.cc:5697] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:39.490082 25316 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
W20260709 13:07:39.576849 25284 consensus_peers.cc:597] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e81523f4f4344a04a1e0b03c8de99290. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:39.580020 25244 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:39.581523 26204 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:39.584690 25393 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:39.587309 26194 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:39.598860 26214 raft_consensus.cc:2955] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:39.601776 25390 raft_consensus.cc:2955] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:39.603318 25244 raft_consensus.cc:2955] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:39.627735 25141 catalog_manager.cc:5697] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:39.869810 26217 ts_tablet_manager.cc:933] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:07:39.878531 26217 tablet_copy_client.cc:323] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.65:45313
I20260709 13:07:39.880971 25255 tablet_copy_service.cc:140] P 46de6919f85341c6a9b574dff112e35a: Received BeginTabletCopySession request for tablet 993e3b05a73a4403b04c3b55a5b48abe from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:44954)
I20260709 13:07:39.881484 25255 tablet_copy_service.cc:161] P 46de6919f85341c6a9b574dff112e35a: Beginning new tablet copy session on tablet 993e3b05a73a4403b04c3b55a5b48abe from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:44954: session id = f646dfd532974dceadfe00005d2bbf59-993e3b05a73a4403b04c3b55a5b48abe
I20260709 13:07:39.887517 25255 tablet_copy_source_session.cc:215] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:39.890718 26217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 993e3b05a73a4403b04c3b55a5b48abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:39.901759 26217 tablet_copy_client.cc:806] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:39.902364 26217 tablet_copy_client.cc:670] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:39.906354 26217 tablet_copy_client.cc:538] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:39.913493 26217 tablet_bootstrap.cc:492] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:39.920876 26219 ts_tablet_manager.cc:933] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:39.929368 26219 tablet_copy_client.cc:323] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.66:42315
I20260709 13:07:39.931357 25331 tablet_copy_service.cc:140] P 2329e428ebbf484fa3ea9452133df567: Received BeginTabletCopySession request for tablet 2dc23779efbf4518a6ada8bde8243a11 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:40928)
I20260709 13:07:39.931903 25331 tablet_copy_service.cc:161] P 2329e428ebbf484fa3ea9452133df567: Beginning new tablet copy session on tablet 2dc23779efbf4518a6ada8bde8243a11 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:40928: session id = f646dfd532974dceadfe00005d2bbf59-2dc23779efbf4518a6ada8bde8243a11
I20260709 13:07:39.936286 26217 tablet_bootstrap.cc:492] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:39.937547 26217 tablet_bootstrap.cc:492] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:39.938205 26217 ts_tablet_manager.cc:1403] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.025s	user 0.019s	sys 0.004s
I20260709 13:07:39.938794 25331 tablet_copy_source_session.cc:215] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:39.944458 26219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dc23779efbf4518a6ada8bde8243a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:39.944113 26217 raft_consensus.cc:359] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:39.945569 26217 raft_consensus.cc:740] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:39.946242 26217 consensus_queue.cc:260] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:39.949434 26217 ts_tablet_manager.cc:1434] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 13:07:39.951014 25255 tablet_copy_service.cc:342] P 46de6919f85341c6a9b574dff112e35a: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-993e3b05a73a4403b04c3b55a5b48abe received from {username='slave'} at 127.0.0.1:44954
I20260709 13:07:39.951409 25255 tablet_copy_service.cc:434] P 46de6919f85341c6a9b574dff112e35a: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-993e3b05a73a4403b04c3b55a5b48abe on tablet 993e3b05a73a4403b04c3b55a5b48abe with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:39.961467 26219 tablet_copy_client.cc:806] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:39.962226 26219 tablet_copy_client.cc:670] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:39.965709 26219 tablet_copy_client.cc:538] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:39.974161 26219 tablet_bootstrap.cc:492] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:39.993670 26219 tablet_bootstrap.cc:492] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:39.994637 26219 tablet_bootstrap.cc:492] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:39.995304 26219 ts_tablet_manager.cc:1403] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.021s	user 0.013s	sys 0.008s
I20260709 13:07:39.998178 26219 raft_consensus.cc:359] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:39.999037 26219 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:39.999672 26219 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:40.001996 26219 ts_tablet_manager.cc:1434] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:07:40.003659 25331 tablet_copy_service.cc:342] P 2329e428ebbf484fa3ea9452133df567: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-2dc23779efbf4518a6ada8bde8243a11 received from {username='slave'} at 127.0.0.1:40928
I20260709 13:07:40.004065 25331 tablet_copy_service.cc:434] P 2329e428ebbf484fa3ea9452133df567: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-2dc23779efbf4518a6ada8bde8243a11 on tablet 2dc23779efbf4518a6ada8bde8243a11 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:40.017355 26219 ts_tablet_manager.cc:933] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:40.020242 26219 tablet_copy_client.cc:323] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.67:34661
I20260709 13:07:40.023180 25405 tablet_copy_service.cc:140] P a2a77ecf2c0941618b5bc79d4c04db61: Received BeginTabletCopySession request for tablet 3cff7b4cb7184e4c9a69372d5901e19a from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:41830)
I20260709 13:07:40.023715 25405 tablet_copy_service.cc:161] P a2a77ecf2c0941618b5bc79d4c04db61: Beginning new tablet copy session on tablet 3cff7b4cb7184e4c9a69372d5901e19a from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:41830: session id = f646dfd532974dceadfe00005d2bbf59-3cff7b4cb7184e4c9a69372d5901e19a
I20260709 13:07:40.034894 25405 tablet_copy_source_session.cc:215] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:40.038048 26219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cff7b4cb7184e4c9a69372d5901e19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:40.053180 26219 tablet_copy_client.cc:806] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:40.053747 26219 tablet_copy_client.cc:670] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:40.057648 26219 tablet_copy_client.cc:538] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:40.065433 26219 tablet_bootstrap.cc:492] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:40.083335 26219 tablet_bootstrap.cc:492] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:40.084096 26219 tablet_bootstrap.cc:492] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:40.084708 26219 ts_tablet_manager.cc:1403] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.019s	user 0.014s	sys 0.000s
I20260709 13:07:40.088361 26219 raft_consensus.cc:359] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:40.089185 26219 raft_consensus.cc:740] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:40.089766 26219 consensus_queue.cc:260] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:40.093858 26219 ts_tablet_manager.cc:1434] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260709 13:07:40.098217 25405 tablet_copy_service.cc:342] P a2a77ecf2c0941618b5bc79d4c04db61: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-3cff7b4cb7184e4c9a69372d5901e19a received from {username='slave'} at 127.0.0.1:41830
I20260709 13:07:40.098637 25405 tablet_copy_service.cc:434] P a2a77ecf2c0941618b5bc79d4c04db61: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-3cff7b4cb7184e4c9a69372d5901e19a on tablet 3cff7b4cb7184e4c9a69372d5901e19a with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:40.160929 26219 ts_tablet_manager.cc:933] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:40.162729 26219 tablet_copy_client.cc:323] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.66:42315
I20260709 13:07:40.164044 25331 tablet_copy_service.cc:140] P 2329e428ebbf484fa3ea9452133df567: Received BeginTabletCopySession request for tablet e81523f4f4344a04a1e0b03c8de99290 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:40928)
I20260709 13:07:40.164456 25331 tablet_copy_service.cc:161] P 2329e428ebbf484fa3ea9452133df567: Beginning new tablet copy session on tablet e81523f4f4344a04a1e0b03c8de99290 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:40928: session id = f646dfd532974dceadfe00005d2bbf59-e81523f4f4344a04a1e0b03c8de99290
I20260709 13:07:40.169440 25331 tablet_copy_source_session.cc:215] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:40.173055 26219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e81523f4f4344a04a1e0b03c8de99290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:40.186529 26219 tablet_copy_client.cc:806] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:40.187071 26219 tablet_copy_client.cc:670] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:40.190410 26219 tablet_copy_client.cc:538] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:40.197463 26219 tablet_bootstrap.cc:492] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:40.212690 26219 tablet_bootstrap.cc:492] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:40.213410 26219 tablet_bootstrap.cc:492] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:40.213892 26219 ts_tablet_manager.cc:1403] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.017s	user 0.007s	sys 0.008s
I20260709 13:07:40.215806 26219 raft_consensus.cc:359] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:40.216352 26219 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:40.216804 26219 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:40.218676 26219 ts_tablet_manager.cc:1434] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:07:40.223452 25331 tablet_copy_service.cc:342] P 2329e428ebbf484fa3ea9452133df567: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-e81523f4f4344a04a1e0b03c8de99290 received from {username='slave'} at 127.0.0.1:40928
I20260709 13:07:40.223862 25331 tablet_copy_service.cc:434] P 2329e428ebbf484fa3ea9452133df567: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-e81523f4f4344a04a1e0b03c8de99290 on tablet e81523f4f4344a04a1e0b03c8de99290 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:40.382540 25944 raft_consensus.cc:1217] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:40.406433 25944 raft_consensus.cc:1217] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:40.483582 25942 raft_consensus.cc:1217] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:40.561257 25942 raft_consensus.cc:1217] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:40.818840 26198 raft_consensus.cc:1064] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:40.820998 26198 consensus_queue.cc:237] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:40.835547 25942 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:40.836844 25314 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:40.837627 25390 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:40.838346 26198 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:40.839515 26229 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:40.840905 26166 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:40.861793 26229 raft_consensus.cc:2955] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:40.865468 25390 raft_consensus.cc:2955] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:40.883929 25140 catalog_manager.cc:5697] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:40.885622 25314 raft_consensus.cc:2955] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:40.885756 25942 raft_consensus.cc:2955] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:40.944564 25241 consensus_queue.cc:237] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:40.956955 25942 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:40.957262 25390 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:40.959198 26236 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:40.963542 26198 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:40.970558 25393 raft_consensus.cc:2955] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:40.973810 25942 raft_consensus.cc:2955] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:40.968462 26198 raft_consensus.cc:2955] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:40.988950 25142 catalog_manager.cc:5697] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:40.994700 25129 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 993e3b05a73a4403b04c3b55a5b48abe with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:41.005720 25310 tablet_service.cc:1558] Processing DeleteTablet for tablet 993e3b05a73a4403b04c3b55a5b48abe with delete_type TABLET_DATA_TOMBSTONED (TS 2329e428ebbf484fa3ea9452133df567 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36848
I20260709 13:07:41.015342 26240 tablet_replica.cc:333] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:07:41.016425 26240 raft_consensus.cc:2243] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:41.017189 26240 raft_consensus.cc:2272] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:41.020004 26240 ts_tablet_manager.cc:1916] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:41.023891 26225 raft_consensus.cc:1064] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:41.028321 26225 consensus_queue.cc:237] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:41.033790 26214 raft_consensus.cc:1064] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:41.036271 26214 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:41.040073 26240 ts_tablet_manager.cc:1929] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:41.040550 26240 log.cc:1199] T 993e3b05a73a4403b04c3b55a5b48abe P 2329e428ebbf484fa3ea9452133df567: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-1-root/wals/993e3b05a73a4403b04c3b55a5b48abe
I20260709 13:07:41.042127 25128 catalog_manager.cc:5028] TS 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): tablet 993e3b05a73a4403b04c3b55a5b48abe (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:41.046324 25393 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:41.047606 25944 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:41.048105 25241 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:41.049356 26225 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:41.049928 25944 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:41.050433 26213 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:41.053170 26213 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:41.054361 26082 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:41.058817 25314 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:41.062137 26209 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:41.071194 26082 raft_consensus.cc:2955] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.073086 25244 raft_consensus.cc:2955] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.100047 25314 raft_consensus.cc:2955] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.095887 25142 catalog_manager.cc:5697] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.104046 25243 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:41.106586 26230 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 13:07:41.134336 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 83244.6us
I20260709 13:07:41.143117 25942 raft_consensus.cc:2955] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.152370 26213 raft_consensus.cc:1064] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:41.159852 26213 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:41.166363 26230 raft_consensus.cc:2955] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.171420 25393 consensus_queue.cc:237] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:41.177114 25942 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:41.177594 25314 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:41.181851 26225 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:41.183282 26225 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:41.180843 25944 raft_consensus.cc:2955] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.188809 26225 raft_consensus.cc:2955] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.190905 25314 raft_consensus.cc:2955] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.198886 25131 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 3cff7b4cb7184e4c9a69372d5901e19a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:41.197902 25942 raft_consensus.cc:2955] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.201285 25241 raft_consensus.cc:2955] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.227594 25393 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:41.227938 25942 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:41.231217 26214 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:41.229354 25142 catalog_manager.cc:5697] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.235544 25142 catalog_manager.cc:5697] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.236250 26214 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:41.251524 25241 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:41.252980 26255 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:41.268332 25230 tablet_service.cc:1558] Processing DeleteTablet for tablet 3cff7b4cb7184e4c9a69372d5901e19a with delete_type TABLET_DATA_TOMBSTONED (TS 46de6919f85341c6a9b574dff112e35a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32990
I20260709 13:07:41.279433 25314 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:41.292140 25243 raft_consensus.cc:2955] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.281230 26259 tablet_replica.cc:333] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:07:41.281929 25394 raft_consensus.cc:2955] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.293012 25390 raft_consensus.cc:2955] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.294597 26259 raft_consensus.cc:2243] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:41.295301 26259 raft_consensus.cc:2272] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:41.299522 25942 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:41.299670 26259 ts_tablet_manager.cc:1916] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:41.304713 26214 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:41.288432 26213 raft_consensus.cc:2955] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.318303 25241 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:41.318938 26214 raft_consensus.cc:2955] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.319862 26243 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:41.324119 25244 raft_consensus.cc:2955] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.332202 26259 ts_tablet_manager.cc:1929] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:41.332700 26259 log.cc:1199] T 3cff7b4cb7184e4c9a69372d5901e19a P 46de6919f85341c6a9b574dff112e35a: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-0-root/wals/3cff7b4cb7184e4c9a69372d5901e19a
I20260709 13:07:41.334393 25128 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet e81523f4f4344a04a1e0b03c8de99290 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:41.334656 25129 catalog_manager.cc:5028] TS 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): tablet 3cff7b4cb7184e4c9a69372d5901e19a (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:41.324676 25944 raft_consensus.cc:2955] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.339728 25142 catalog_manager.cc:5697] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:07:41.346865 25128 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet e81523f4f4344a04a1e0b03c8de99290 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 4
I20260709 13:07:41.353540 25140 catalog_manager.cc:5697] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.355497 25942 raft_consensus.cc:2955] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.370013 25378 tablet_service.cc:1558] Processing DeleteTablet for tablet e81523f4f4344a04a1e0b03c8de99290 with delete_type TABLET_DATA_TOMBSTONED (TS a2a77ecf2c0941618b5bc79d4c04db61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60924
I20260709 13:07:41.397221 26261 tablet_replica.cc:333] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:07:41.398258 26261 raft_consensus.cc:2243] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:41.399003 26261 raft_consensus.cc:2272] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:41.402786 26261 ts_tablet_manager.cc:1916] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:41.412305 25314 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:41.420573 26261 ts_tablet_manager.cc:1929] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:41.421119 26261 log.cc:1199] T e81523f4f4344a04a1e0b03c8de99290 P a2a77ecf2c0941618b5bc79d4c04db61: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-2-root/wals/e81523f4f4344a04a1e0b03c8de99290
I20260709 13:07:41.421314 25942 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:41.422811 25131 catalog_manager.cc:5028] TS a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): tablet e81523f4f4344a04a1e0b03c8de99290 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:41.423507 26213 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:41.424799 25243 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:41.426133 26213 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:41.429952 26253 raft_consensus.cc:2955] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.435892 25243 raft_consensus.cc:2955] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.439059 25944 raft_consensus.cc:2955] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.459208 25140 catalog_manager.cc:5697] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:41.461812 25128 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2dc23779efbf4518a6ada8bde8243a11 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:41.472455 25378 tablet_service.cc:1558] Processing DeleteTablet for tablet 2dc23779efbf4518a6ada8bde8243a11 with delete_type TABLET_DATA_TOMBSTONED (TS a2a77ecf2c0941618b5bc79d4c04db61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60924
I20260709 13:07:41.473381 26261 tablet_replica.cc:333] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:07:41.474248 26261 raft_consensus.cc:2243] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:41.477468 26261 raft_consensus.cc:2272] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:41.480335 26261 ts_tablet_manager.cc:1916] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:41.495280 26261 ts_tablet_manager.cc:1929] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:41.495733 26261 log.cc:1199] T 2dc23779efbf4518a6ada8bde8243a11 P a2a77ecf2c0941618b5bc79d4c04db61: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-2-root/wals/2dc23779efbf4518a6ada8bde8243a11
I20260709 13:07:41.497104 25131 catalog_manager.cc:5028] TS a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): tablet 2dc23779efbf4518a6ada8bde8243a11 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:42.541971 25244 consensus_queue.cc:237] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
W20260709 13:07:42.554353 25209 consensus_peers.cc:597] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5481170c5c74798bdfd146068f8b539. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:42.559083 25394 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:42.559094 25314 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:42.560488 26229 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:42.561343 26266 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:42.568862 26266 raft_consensus.cc:2955] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:42.570520 25314 raft_consensus.cc:2955] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:42.585350 25394 raft_consensus.cc:2955] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:42.585721 25143 catalog_manager.cc:5697] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:42.586534 25390 consensus_queue.cc:237] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:42.603612 25318 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:07:42.605924 25358 consensus_peers.cc:597] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2730c370dd9a4a1ea5adfb543e3ba120. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:42.606465 25243 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:42.606861 26225 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:42.607949 26263 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:42.623005 26082 raft_consensus.cc:2955] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:42.624444 25318 raft_consensus.cc:2955] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:42.630151 25243 raft_consensus.cc:2955] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:42.635031 25244 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:42.642267 25142 catalog_manager.cc:5697] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:42.645638 25390 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:07:42.646225 25209 consensus_peers.cc:597] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63c3ef75ef234051b7ae1ea395400f54. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:42.647080 26265 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:42.649271 25316 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:42.656571 26266 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:42.658833 26229 raft_consensus.cc:2955] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:42.661854 25390 raft_consensus.cc:2955] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:42.682497 25140 catalog_manager.cc:5697] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:42.685768 25390 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:42.695551 25316 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:42.694613 25318 raft_consensus.cc:2955] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
W20260709 13:07:42.697762 25358 consensus_peers.cc:597] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e29e9ab7eda4242964fa76e29c2a221. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:42.697782 26082 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:42.700448 25243 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:42.701874 26082 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:42.710533 26225 raft_consensus.cc:2955] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:42.714298 25316 raft_consensus.cc:2955] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:42.719415 25243 raft_consensus.cc:2955] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:42.730963 25142 catalog_manager.cc:5697] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:43.095826 26282 ts_tablet_manager.cc:933] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:07:43.097653 26282 tablet_copy_client.cc:323] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.65:45313
I20260709 13:07:43.098974 25255 tablet_copy_service.cc:140] P 46de6919f85341c6a9b574dff112e35a: Received BeginTabletCopySession request for tablet c5481170c5c74798bdfd146068f8b539 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:44954)
I20260709 13:07:43.099392 25255 tablet_copy_service.cc:161] P 46de6919f85341c6a9b574dff112e35a: Beginning new tablet copy session on tablet c5481170c5c74798bdfd146068f8b539 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:44954: session id = f646dfd532974dceadfe00005d2bbf59-c5481170c5c74798bdfd146068f8b539
I20260709 13:07:43.106400 25255 tablet_copy_source_session.cc:215] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:43.109285 26282 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5481170c5c74798bdfd146068f8b539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:43.120568 26282 tablet_copy_client.cc:806] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:43.122033 26282 tablet_copy_client.cc:670] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:43.126143 26282 tablet_copy_client.cc:538] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:43.132630 26284 ts_tablet_manager.cc:933] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:43.134727 26284 tablet_copy_client.cc:323] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.67:34661
I20260709 13:07:43.135046 26282 tablet_bootstrap.cc:492] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:43.136564 25405 tablet_copy_service.cc:140] P a2a77ecf2c0941618b5bc79d4c04db61: Received BeginTabletCopySession request for tablet 4e29e9ab7eda4242964fa76e29c2a221 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:41830)
I20260709 13:07:43.137109 25405 tablet_copy_service.cc:161] P a2a77ecf2c0941618b5bc79d4c04db61: Beginning new tablet copy session on tablet 4e29e9ab7eda4242964fa76e29c2a221 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:41830: session id = f646dfd532974dceadfe00005d2bbf59-4e29e9ab7eda4242964fa76e29c2a221
I20260709 13:07:43.144114 25405 tablet_copy_source_session.cc:215] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:43.147334 26284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e29e9ab7eda4242964fa76e29c2a221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:43.159432 26282 tablet_bootstrap.cc:492] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:43.165663 26282 tablet_bootstrap.cc:492] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:43.166399 26282 ts_tablet_manager.cc:1403] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.032s	user 0.024s	sys 0.001s
I20260709 13:07:43.170382 26282 raft_consensus.cc:359] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:43.171267 26282 raft_consensus.cc:740] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:43.172885 26282 consensus_queue.cc:260] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:43.183522 26282 ts_tablet_manager.cc:1434] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.007s
I20260709 13:07:43.185542 25255 tablet_copy_service.cc:342] P 46de6919f85341c6a9b574dff112e35a: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-c5481170c5c74798bdfd146068f8b539 received from {username='slave'} at 127.0.0.1:44954
I20260709 13:07:43.186089 25255 tablet_copy_service.cc:434] P 46de6919f85341c6a9b574dff112e35a: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-c5481170c5c74798bdfd146068f8b539 on tablet c5481170c5c74798bdfd146068f8b539 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:43.190744 26288 ts_tablet_manager.cc:933] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:43.198354 26290 ts_tablet_manager.cc:933] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:07:43.205425 26288 tablet_copy_client.cc:323] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.67:34661
I20260709 13:07:43.205835 26290 tablet_copy_client.cc:323] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.65:45313
I20260709 13:07:43.206347 26284 tablet_copy_client.cc:806] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:43.206923 26284 tablet_copy_client.cc:670] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:43.207249 25405 tablet_copy_service.cc:140] P a2a77ecf2c0941618b5bc79d4c04db61: Received BeginTabletCopySession request for tablet 2730c370dd9a4a1ea5adfb543e3ba120 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:41830)
I20260709 13:07:43.207777 25405 tablet_copy_service.cc:161] P a2a77ecf2c0941618b5bc79d4c04db61: Beginning new tablet copy session on tablet 2730c370dd9a4a1ea5adfb543e3ba120 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:41830: session id = f646dfd532974dceadfe00005d2bbf59-2730c370dd9a4a1ea5adfb543e3ba120
I20260709 13:07:43.210023 25255 tablet_copy_service.cc:140] P 46de6919f85341c6a9b574dff112e35a: Received BeginTabletCopySession request for tablet 63c3ef75ef234051b7ae1ea395400f54 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:44954)
I20260709 13:07:43.210561 25255 tablet_copy_service.cc:161] P 46de6919f85341c6a9b574dff112e35a: Beginning new tablet copy session on tablet 63c3ef75ef234051b7ae1ea395400f54 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:44954: session id = f646dfd532974dceadfe00005d2bbf59-63c3ef75ef234051b7ae1ea395400f54
I20260709 13:07:43.217988 25405 tablet_copy_source_session.cc:215] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:43.220808 25255 tablet_copy_source_session.cc:215] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:43.222359 26288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2730c370dd9a4a1ea5adfb543e3ba120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:43.222460 26284 tablet_copy_client.cc:538] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:43.223683 26290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63c3ef75ef234051b7ae1ea395400f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:43.238567 26284 tablet_bootstrap.cc:492] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:43.245736 26288 tablet_copy_client.cc:806] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:43.246371 26288 tablet_copy_client.cc:670] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:43.253535 26288 tablet_copy_client.cc:538] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:43.257180 26290 tablet_copy_client.cc:806] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:43.258008 26290 tablet_copy_client.cc:670] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:43.262065 26290 tablet_copy_client.cc:538] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:43.262269 26288 tablet_bootstrap.cc:492] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:43.267321 26284 tablet_bootstrap.cc:492] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:43.269833 26284 tablet_bootstrap.cc:492] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:43.270573 26284 ts_tablet_manager.cc:1403] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.032s	user 0.018s	sys 0.010s
I20260709 13:07:43.273859 26284 raft_consensus.cc:359] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:43.274521 26290 tablet_bootstrap.cc:492] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:43.274735 26284 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:43.275475 26284 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:43.278075 26284 ts_tablet_manager.cc:1434] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:07:43.280987 25405 tablet_copy_service.cc:342] P a2a77ecf2c0941618b5bc79d4c04db61: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-4e29e9ab7eda4242964fa76e29c2a221 received from {username='slave'} at 127.0.0.1:41830
I20260709 13:07:43.281390 25405 tablet_copy_service.cc:434] P a2a77ecf2c0941618b5bc79d4c04db61: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-4e29e9ab7eda4242964fa76e29c2a221 on tablet 4e29e9ab7eda4242964fa76e29c2a221 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:43.316192 26288 tablet_bootstrap.cc:492] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:43.317641 26288 tablet_bootstrap.cc:492] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:43.318526 26288 ts_tablet_manager.cc:1403] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.056s	user 0.019s	sys 0.009s
I20260709 13:07:43.321914 26288 raft_consensus.cc:359] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:43.322701 26288 raft_consensus.cc:740] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:43.323318 26288 consensus_queue.cc:260] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:43.328833 26288 ts_tablet_manager.cc:1434] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.003s
I20260709 13:07:43.330629 26290 tablet_bootstrap.cc:492] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:43.331740 26290 tablet_bootstrap.cc:492] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:43.333319 26290 ts_tablet_manager.cc:1403] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.059s	user 0.019s	sys 0.009s
I20260709 13:07:43.333429 25405 tablet_copy_service.cc:342] P a2a77ecf2c0941618b5bc79d4c04db61: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-2730c370dd9a4a1ea5adfb543e3ba120 received from {username='slave'} at 127.0.0.1:41830
I20260709 13:07:43.333916 25405 tablet_copy_service.cc:434] P a2a77ecf2c0941618b5bc79d4c04db61: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-2730c370dd9a4a1ea5adfb543e3ba120 on tablet 2730c370dd9a4a1ea5adfb543e3ba120 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:43.336122 26290 raft_consensus.cc:359] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:43.345018 26290 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:43.345728 26290 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:43.348242 26290 ts_tablet_manager.cc:1434] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.015s	user 0.003s	sys 0.000s
I20260709 13:07:43.350344 25255 tablet_copy_service.cc:342] P 46de6919f85341c6a9b574dff112e35a: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-63c3ef75ef234051b7ae1ea395400f54 received from {username='slave'} at 127.0.0.1:44954
I20260709 13:07:43.350749 25255 tablet_copy_service.cc:434] P 46de6919f85341c6a9b574dff112e35a: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-63c3ef75ef234051b7ae1ea395400f54 on tablet 63c3ef75ef234051b7ae1ea395400f54 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:43.714754 25944 raft_consensus.cc:1217] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:43.714636 25945 raft_consensus.cc:1217] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:43.801061 25944 raft_consensus.cc:1217] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:43.825325 25943 raft_consensus.cc:1217] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:44.131573 26266 raft_consensus.cc:1064] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:44.134377 26266 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:44.152235 26299 raft_consensus.cc:1064] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:44.159422 25943 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:44.159821 25318 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:44.161311 26262 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:44.155155 26299 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:44.162621 25390 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:44.163798 26266 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:44.167547 26297 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:44.180935 25246 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:44.181831 25945 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:44.182634 26225 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:44.183614 25316 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:44.186803 26302 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:44.187885 26225 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:44.229770 25318 raft_consensus.cc:2955] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.242630 26262 raft_consensus.cc:2955] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.249877 25390 raft_consensus.cc:2955] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.250085 26225 raft_consensus.cc:2955] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.267167 26229 raft_consensus.cc:1064] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:44.269659 26229 consensus_queue.cc:237] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:44.269342 25945 raft_consensus.cc:2955] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.291342 25246 raft_consensus.cc:2955] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.294379 25142 catalog_manager.cc:5697] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.302714 25945 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:44.303893 25314 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:44.310118 25141 catalog_manager.cc:5697] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.311398 26301 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:44.313946 26266 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 13:07:44.317866 26310 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 13:07:44.324077 25390 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:44.310077 25945 raft_consensus.cc:2955] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.338181 26229 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:44.349684 25315 raft_consensus.cc:2955] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.356535 26229 raft_consensus.cc:2955] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.379364 25246 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:44.372334 25314 raft_consensus.cc:2955] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.394320 25943 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:44.397037 26297 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:44.405696 25394 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:44.412164 25141 catalog_manager.cc:5697] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.414314 25393 raft_consensus.cc:2955] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.417161 26299 raft_consensus.cc:1064] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:44.417909 26266 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:44.419683 26299 consensus_queue.cc:237] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:44.432297 26297 raft_consensus.cc:2955] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.422217 25943 raft_consensus.cc:2955] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.473039 25314 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:44.476249 25246 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:44.480477 26225 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:44.481684 26302 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:44.443182 25394 raft_consensus.cc:2955] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:07:44.507305 26319 log.cc:927] Time spent T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 13:07:44.512688 25942 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:44.458536 25945 raft_consensus.cc:2955] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.519748 26302 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:44.530575 25129 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 63c3ef75ef234051b7ae1ea395400f54 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:44.532827 26320 raft_consensus.cc:2955] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.537523 25944 raft_consensus.cc:2955] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.542344 25314 raft_consensus.cc:2955] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.552732 25141 catalog_manager.cc:5697] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:44.555053 25143 catalog_manager.cc:5697] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.568547 25246 raft_consensus.cc:2955] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.581105 25310 tablet_service.cc:1558] Processing DeleteTablet for tablet 63c3ef75ef234051b7ae1ea395400f54 with delete_type TABLET_DATA_TOMBSTONED (TS 2329e428ebbf484fa3ea9452133df567 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36848
I20260709 13:07:44.593180 25246 consensus_queue.cc:237] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:44.596657 26325 tablet_replica.cc:333] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:07:44.605429 26325 raft_consensus.cc:2243] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:44.606314 26325 raft_consensus.cc:2272] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:44.607439 25393 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:44.608383 25394 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:44.610347 25945 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:44.611791 26229 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:44.617918 26229 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:44.620774 25318 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:44.622669 26324 raft_consensus.cc:2955] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.626180 26325 ts_tablet_manager.cc:1916] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:44.627239 25943 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:44.636991 26225 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:07:44.636973 25943 raft_consensus.cc:2955] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.638161 25390 raft_consensus.cc:2955] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.639737 26225 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:44.636927 25393 consensus_queue.cc:237] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:44.651528 25129 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet c5481170c5c74798bdfd146068f8b539 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:44.656569 26302 raft_consensus.cc:2955] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.669248 25131 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 4e29e9ab7eda4242964fa76e29c2a221 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:44.669320 25318 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:44.671521 25945 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:44.674647 25140 catalog_manager.cc:5697] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.675047 25318 raft_consensus.cc:2955] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.683693 26325 ts_tablet_manager.cc:1929] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:44.684233 26325 log.cc:1199] T 63c3ef75ef234051b7ae1ea395400f54 P 2329e428ebbf484fa3ea9452133df567: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-1-root/wals/63c3ef75ef234051b7ae1ea395400f54
I20260709 13:07:44.687108 25128 catalog_manager.cc:5028] TS 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): tablet 63c3ef75ef234051b7ae1ea395400f54 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:44.689107 25310 tablet_service.cc:1558] Processing DeleteTablet for tablet c5481170c5c74798bdfd146068f8b539 with delete_type TABLET_DATA_TOMBSTONED (TS 2329e428ebbf484fa3ea9452133df567 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36848
I20260709 13:07:44.692364 26225 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:44.693994 25943 raft_consensus.cc:2955] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.699402 26082 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:44.703794 26325 tablet_replica.cc:333] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:07:44.710357 25140 catalog_manager.cc:5697] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.717641 26302 raft_consensus.cc:2955] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.721190 25230 tablet_service.cc:1558] Processing DeleteTablet for tablet 4e29e9ab7eda4242964fa76e29c2a221 with delete_type TABLET_DATA_TOMBSTONED (TS 46de6919f85341c6a9b574dff112e35a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32990
I20260709 13:07:44.730752 25131 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2730c370dd9a4a1ea5adfb543e3ba120 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:44.734524 25945 raft_consensus.cc:2955] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.748035 26325 raft_consensus.cc:2243] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:44.748886 26325 raft_consensus.cc:2272] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:44.751585 25141 catalog_manager.cc:5697] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.755159 26325 ts_tablet_manager.cc:1916] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:44.763790 25314 raft_consensus.cc:2955] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:44.766072 26326 tablet_replica.cc:333] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:07:44.767586 25230 tablet_service.cc:1558] Processing DeleteTablet for tablet 2730c370dd9a4a1ea5adfb543e3ba120 with delete_type TABLET_DATA_TOMBSTONED (TS 46de6919f85341c6a9b574dff112e35a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32990
I20260709 13:07:44.770509 26326 raft_consensus.cc:2243] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:44.772398 26326 raft_consensus.cc:2272] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:44.781538 26325 ts_tablet_manager.cc:1929] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:44.781980 26325 log.cc:1199] T c5481170c5c74798bdfd146068f8b539 P 2329e428ebbf484fa3ea9452133df567: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-1-root/wals/c5481170c5c74798bdfd146068f8b539
I20260709 13:07:44.784016 25128 catalog_manager.cc:5028] TS 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): tablet c5481170c5c74798bdfd146068f8b539 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:44.784435 26326 ts_tablet_manager.cc:1916] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:44.801375 26326 ts_tablet_manager.cc:1929] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:44.801717 26326 log.cc:1199] T 4e29e9ab7eda4242964fa76e29c2a221 P 46de6919f85341c6a9b574dff112e35a: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-0-root/wals/4e29e9ab7eda4242964fa76e29c2a221
I20260709 13:07:44.803200 26326 tablet_replica.cc:333] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:07:44.803191 25129 catalog_manager.cc:5028] TS 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): tablet 4e29e9ab7eda4242964fa76e29c2a221 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:44.804207 26326 raft_consensus.cc:2243] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:44.804951 26326 raft_consensus.cc:2272] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:44.807801 26326 ts_tablet_manager.cc:1916] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:44.817829 26326 ts_tablet_manager.cc:1929] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:44.818153 26326 log.cc:1199] T 2730c370dd9a4a1ea5adfb543e3ba120 P 46de6919f85341c6a9b574dff112e35a: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-0-root/wals/2730c370dd9a4a1ea5adfb543e3ba120
I20260709 13:07:44.819378 25129 catalog_manager.cc:5028] TS 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): tablet 2730c370dd9a4a1ea5adfb543e3ba120 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:45.841939 25314 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:45.847939 25393 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:07:45.848490 25284 consensus_peers.cc:597] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2cfa29915b34839a60321576f9a67d6. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:45.849643 25244 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:45.850687 26213 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:45.851502 26328 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:45.859395 26328 raft_consensus.cc:2955] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:45.873081 25244 raft_consensus.cc:2955] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:45.882715 25246 consensus_queue.cc:237] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:45.888619 25140 catalog_manager.cc:5697] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:45.891564 25390 raft_consensus.cc:2955] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
W20260709 13:07:45.904803 25209 consensus_peers.cc:597] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ff9a7b00f104dd294125870b06a7df4. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:45.905951 25393 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:45.907752 25314 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:45.908185 26324 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:45.910302 26324 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:45.921839 26324 raft_consensus.cc:2955] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:45.925679 25390 raft_consensus.cc:2955] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:45.927667 25314 raft_consensus.cc:2955] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:45.937201 25315 consensus_queue.cc:237] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:45.956192 25141 catalog_manager.cc:5697] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
W20260709 13:07:45.959070 25284 consensus_peers.cc:597] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a53b1fb6477d486e98bff9549cc487df. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:45.959753 25246 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:45.960031 25390 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:45.961642 26335 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:45.963454 26335 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:45.978824 26333 raft_consensus.cc:2955] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:45.982304 25244 raft_consensus.cc:2955] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:45.985352 25393 raft_consensus.cc:2955] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:45.999068 25393 consensus_queue.cc:237] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:46.000802 25140 catalog_manager.cc:5697] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:46.005808 25318 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:46.009037 25244 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:46.015806 26302 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:46.020115 26302 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:07:46.023356 25358 consensus_peers.cc:597] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d45c0f8b003848989966aa931c6a7c11. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:46.034190 26302 raft_consensus.cc:2955] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:46.036119 25246 raft_consensus.cc:2955] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:46.037823 25318 raft_consensus.cc:2955] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:46.053866 25143 catalog_manager.cc:5697] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:46.342706 26353 ts_tablet_manager.cc:933] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:07:46.344959 26353 tablet_copy_client.cc:323] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.65:45313
I20260709 13:07:46.346731 25255 tablet_copy_service.cc:140] P 46de6919f85341c6a9b574dff112e35a: Received BeginTabletCopySession request for tablet 6ff9a7b00f104dd294125870b06a7df4 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:44954)
I20260709 13:07:46.347257 25255 tablet_copy_service.cc:161] P 46de6919f85341c6a9b574dff112e35a: Beginning new tablet copy session on tablet 6ff9a7b00f104dd294125870b06a7df4 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:44954: session id = f646dfd532974dceadfe00005d2bbf59-6ff9a7b00f104dd294125870b06a7df4
I20260709 13:07:46.352941 25255 tablet_copy_source_session.cc:215] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:46.356081 26353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ff9a7b00f104dd294125870b06a7df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:46.374194 26353 tablet_copy_client.cc:806] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:46.374828 26353 tablet_copy_client.cc:670] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:46.378544 26353 tablet_copy_client.cc:538] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:46.384639 26353 tablet_bootstrap.cc:492] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:46.393239 26355 ts_tablet_manager.cc:933] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:46.396950 26355 tablet_copy_client.cc:323] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.66:42315
I20260709 13:07:46.398829 25331 tablet_copy_service.cc:140] P 2329e428ebbf484fa3ea9452133df567: Received BeginTabletCopySession request for tablet b2cfa29915b34839a60321576f9a67d6 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:40928)
I20260709 13:07:46.399569 25331 tablet_copy_service.cc:161] P 2329e428ebbf484fa3ea9452133df567: Beginning new tablet copy session on tablet b2cfa29915b34839a60321576f9a67d6 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:40928: session id = f646dfd532974dceadfe00005d2bbf59-b2cfa29915b34839a60321576f9a67d6
I20260709 13:07:46.405251 26353 tablet_bootstrap.cc:492] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:46.406221 26353 tablet_bootstrap.cc:492] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:46.406919 26353 ts_tablet_manager.cc:1403] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.023s	user 0.013s	sys 0.012s
I20260709 13:07:46.410090 26353 raft_consensus.cc:359] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:46.410965 26353 raft_consensus.cc:740] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:46.411860 25331 tablet_copy_source_session.cc:215] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:46.411613 26353 consensus_queue.cc:260] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:46.415445 26355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2cfa29915b34839a60321576f9a67d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:46.416177 26353 ts_tablet_manager.cc:1434] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.003s
I20260709 13:07:46.419286 25255 tablet_copy_service.cc:342] P 46de6919f85341c6a9b574dff112e35a: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-6ff9a7b00f104dd294125870b06a7df4 received from {username='slave'} at 127.0.0.1:44954
I20260709 13:07:46.419737 25255 tablet_copy_service.cc:434] P 46de6919f85341c6a9b574dff112e35a: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-6ff9a7b00f104dd294125870b06a7df4 on tablet 6ff9a7b00f104dd294125870b06a7df4 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:46.434105 26355 tablet_copy_client.cc:806] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:46.434676 26355 tablet_copy_client.cc:670] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:46.438117 26355 tablet_copy_client.cc:538] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:46.446091 26355 tablet_bootstrap.cc:492] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:46.463709 26355 tablet_bootstrap.cc:492] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:46.464381 26355 tablet_bootstrap.cc:492] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:46.464910 26355 ts_tablet_manager.cc:1403] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.019s	user 0.012s	sys 0.003s
I20260709 13:07:46.468190 26355 raft_consensus.cc:359] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:46.469020 26355 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:46.469543 26355 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:46.471390 26355 ts_tablet_manager.cc:1434] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 13:07:46.473584 25331 tablet_copy_service.cc:342] P 2329e428ebbf484fa3ea9452133df567: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-b2cfa29915b34839a60321576f9a67d6 received from {username='slave'} at 127.0.0.1:40928
I20260709 13:07:46.473961 25331 tablet_copy_service.cc:434] P 2329e428ebbf484fa3ea9452133df567: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-b2cfa29915b34839a60321576f9a67d6 on tablet b2cfa29915b34839a60321576f9a67d6 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:46.490883 26355 ts_tablet_manager.cc:933] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:46.492758 26355 tablet_copy_client.cc:323] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.66:42315
I20260709 13:07:46.495848 25331 tablet_copy_service.cc:140] P 2329e428ebbf484fa3ea9452133df567: Received BeginTabletCopySession request for tablet a53b1fb6477d486e98bff9549cc487df from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:40928)
I20260709 13:07:46.496330 25331 tablet_copy_service.cc:161] P 2329e428ebbf484fa3ea9452133df567: Beginning new tablet copy session on tablet a53b1fb6477d486e98bff9549cc487df from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:40928: session id = f646dfd532974dceadfe00005d2bbf59-a53b1fb6477d486e98bff9549cc487df
I20260709 13:07:46.505143 25331 tablet_copy_source_session.cc:215] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:46.508425 26355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a53b1fb6477d486e98bff9549cc487df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:46.527621 26355 tablet_copy_client.cc:806] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:46.528223 26355 tablet_copy_client.cc:670] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:46.533118 26355 tablet_copy_client.cc:538] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:46.541657 26355 tablet_bootstrap.cc:492] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:46.563659 26353 ts_tablet_manager.cc:933] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:46.569052 26355 tablet_bootstrap.cc:492] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:46.570132 26355 tablet_bootstrap.cc:492] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:46.571297 26355 ts_tablet_manager.cc:1403] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.030s	user 0.010s	sys 0.015s
I20260709 13:07:46.575289 26355 raft_consensus.cc:359] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:46.579388 26355 raft_consensus.cc:740] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:46.580612 26355 consensus_queue.cc:260] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:46.583599 26355 ts_tablet_manager.cc:1434] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.001s
I20260709 13:07:46.585470 25331 tablet_copy_service.cc:342] P 2329e428ebbf484fa3ea9452133df567: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-a53b1fb6477d486e98bff9549cc487df received from {username='slave'} at 127.0.0.1:40928
I20260709 13:07:46.585880 25331 tablet_copy_service.cc:434] P 2329e428ebbf484fa3ea9452133df567: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-a53b1fb6477d486e98bff9549cc487df on tablet a53b1fb6477d486e98bff9549cc487df with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:46.625362 26353 tablet_copy_client.cc:323] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.67:34661
I20260709 13:07:46.627589 25405 tablet_copy_service.cc:140] P a2a77ecf2c0941618b5bc79d4c04db61: Received BeginTabletCopySession request for tablet d45c0f8b003848989966aa931c6a7c11 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:41830)
I20260709 13:07:46.628105 25405 tablet_copy_service.cc:161] P a2a77ecf2c0941618b5bc79d4c04db61: Beginning new tablet copy session on tablet d45c0f8b003848989966aa931c6a7c11 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:41830: session id = f646dfd532974dceadfe00005d2bbf59-d45c0f8b003848989966aa931c6a7c11
I20260709 13:07:46.635679 25405 tablet_copy_source_session.cc:215] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:46.638852 26353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d45c0f8b003848989966aa931c6a7c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:46.653098 26353 tablet_copy_client.cc:806] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:46.653684 26353 tablet_copy_client.cc:670] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:46.657042 26353 tablet_copy_client.cc:538] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:46.664958 26353 tablet_bootstrap.cc:492] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:46.682358 26353 tablet_bootstrap.cc:492] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:46.683220 26353 tablet_bootstrap.cc:492] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:46.683756 26353 ts_tablet_manager.cc:1403] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.019s	user 0.018s	sys 0.000s
I20260709 13:07:46.685943 26353 raft_consensus.cc:359] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:46.686518 26353 raft_consensus.cc:740] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:46.687019 26353 consensus_queue.cc:260] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:46.689085 26353 ts_tablet_manager.cc:1434] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20260709 13:07:46.691681 25405 tablet_copy_service.cc:342] P a2a77ecf2c0941618b5bc79d4c04db61: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-d45c0f8b003848989966aa931c6a7c11 received from {username='slave'} at 127.0.0.1:41830
I20260709 13:07:46.692117 25405 tablet_copy_service.cc:434] P a2a77ecf2c0941618b5bc79d4c04db61: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-d45c0f8b003848989966aa931c6a7c11 on tablet d45c0f8b003848989966aa931c6a7c11 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:46.894958 25945 raft_consensus.cc:1217] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:46.938016 25945 raft_consensus.cc:1217] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:47.024689 25945 raft_consensus.cc:1217] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:47.078248 25945 raft_consensus.cc:1217] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:47.283461 26364 raft_consensus.cc:1064] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:47.285385 26364 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:47.292435 25393 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:47.293826 26366 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:47.294358 25241 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:47.292922 25945 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:47.295794 26365 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:47.297056 26364 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:47.315145 26365 raft_consensus.cc:2955] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.318147 25241 raft_consensus.cc:2955] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.317804 25393 raft_consensus.cc:2955] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.329696 25945 raft_consensus.cc:2955] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.334334 25140 catalog_manager.cc:5697] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:47.399319 25318 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:47.409070 25945 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:47.411055 26333 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:47.412103 25241 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:47.414417 26333 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:47.418112 26366 raft_consensus.cc:2955] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.428449 25945 raft_consensus.cc:2955] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.432561 25128 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet b2cfa29915b34839a60321576f9a67d6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:47.441296 25143 catalog_manager.cc:5697] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:47.455516 25378 tablet_service.cc:1558] Processing DeleteTablet for tablet b2cfa29915b34839a60321576f9a67d6 with delete_type TABLET_DATA_TOMBSTONED (TS a2a77ecf2c0941618b5bc79d4c04db61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60924
I20260709 13:07:47.462651 26378 tablet_replica.cc:333] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:07:47.470641 26302 raft_consensus.cc:1064] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:47.472816 26302 consensus_queue.cc:237] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:47.475430 25241 raft_consensus.cc:2955] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.489940 26378 raft_consensus.cc:2243] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:47.494206 26378 raft_consensus.cc:2272] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:47.505432 25318 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:47.508679 25241 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:47.511423 26349 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 13:07:47.513552 25945 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:47.517275 26380 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:47.519439 26378 ts_tablet_manager.cc:1916] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:47.519416 26381 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:47.536370 26344 raft_consensus.cc:1064] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:47.539213 26344 consensus_queue.cc:237] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:47.543939 26382 raft_consensus.cc:2955] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.552933 25244 raft_consensus.cc:2955] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.574517 25141 catalog_manager.cc:5697] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.587112 25314 raft_consensus.cc:2955] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.592103 25944 raft_consensus.cc:2955] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.593876 26366 raft_consensus.cc:1064] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:47.599251 26378 ts_tablet_manager.cc:1929] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:47.599757 26378 log.cc:1199] T b2cfa29915b34839a60321576f9a67d6 P a2a77ecf2c0941618b5bc79d4c04db61: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-2-root/wals/b2cfa29915b34839a60321576f9a67d6
I20260709 13:07:47.601505 25131 catalog_manager.cc:5028] TS a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): tablet b2cfa29915b34839a60321576f9a67d6 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:47.596342 26366 consensus_queue.cc:237] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:47.608086 25393 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:47.609436 25943 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:47.611099 26343 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:47.614464 26369 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:47.626022 25390 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:47.626196 25241 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:47.627713 26333 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:47.629482 26364 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:47.635540 25945 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:47.638166 26333 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:47.642946 25318 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:47.644567 26344 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:47.647359 25394 consensus_queue.cc:237] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:47.657512 25314 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:47.659421 25944 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:47.660037 26302 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:47.660847 26382 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:47.670245 26380 raft_consensus.cc:2955] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.672053 25942 raft_consensus.cc:2955] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.682469 25314 raft_consensus.cc:2955] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.688905 25131 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet d45c0f8b003848989966aa931c6a7c11 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:47.688468 25143 catalog_manager.cc:5697] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.711450 25230 tablet_service.cc:1558] Processing DeleteTablet for tablet d45c0f8b003848989966aa931c6a7c11 with delete_type TABLET_DATA_TOMBSTONED (TS 46de6919f85341c6a9b574dff112e35a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32990
I20260709 13:07:47.761296 26344 raft_consensus.cc:2955] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.763691 26394 tablet_replica.cc:333] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:07:47.765630 25315 raft_consensus.cc:2955] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.772449 26364 raft_consensus.cc:2955] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.774561 25942 raft_consensus.cc:2955] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.782821 25944 raft_consensus.cc:2955] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.779264 25390 raft_consensus.cc:2955] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.797649 25142 catalog_manager.cc:5697] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.798828 26394 raft_consensus.cc:2243] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:47.799567 26394 raft_consensus.cc:2272] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:47.801321 25244 raft_consensus.cc:2955] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.803805 26394 ts_tablet_manager.cc:1916] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:47.806480 25143 catalog_manager.cc:5697] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:47.811769 25390 raft_consensus.cc:2955] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.827553 25241 consensus_queue.cc:237] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:47.835562 25944 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:47.840430 26344 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:47.842321 25390 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:47.844188 26343 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:47.850481 25315 consensus_queue.cc:237] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:47.858937 26369 raft_consensus.cc:2955] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.864575 25390 raft_consensus.cc:2955] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.866436 25241 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:47.867897 26366 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:47.868242 25944 raft_consensus.cc:2955] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.875003 25942 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:47.876308 26366 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:47.878116 26365 raft_consensus.cc:2955] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.885677 25945 raft_consensus.cc:2955] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.892745 25241 raft_consensus.cc:2955] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.895996 25143 catalog_manager.cc:5697] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:47.897969 25128 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet a53b1fb6477d486e98bff9549cc487df with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:47.902763 26394 ts_tablet_manager.cc:1929] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:47.903182 26394 log.cc:1199] T d45c0f8b003848989966aa931c6a7c11 P 46de6919f85341c6a9b574dff112e35a: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-0-root/wals/d45c0f8b003848989966aa931c6a7c11
I20260709 13:07:47.904835 25129 catalog_manager.cc:5028] TS 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): tablet d45c0f8b003848989966aa931c6a7c11 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:47.906594 25141 catalog_manager.cc:5697] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:47.913206 25129 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 6ff9a7b00f104dd294125870b06a7df4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:47.921588 25378 tablet_service.cc:1558] Processing DeleteTablet for tablet a53b1fb6477d486e98bff9549cc487df with delete_type TABLET_DATA_TOMBSTONED (TS a2a77ecf2c0941618b5bc79d4c04db61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60924
I20260709 13:07:47.922480 26378 tablet_replica.cc:333] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:07:47.923396 26378 raft_consensus.cc:2243] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:47.924124 26378 raft_consensus.cc:2272] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:47.933283 26378 ts_tablet_manager.cc:1916] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:47.950975 25310 tablet_service.cc:1558] Processing DeleteTablet for tablet 6ff9a7b00f104dd294125870b06a7df4 with delete_type TABLET_DATA_TOMBSTONED (TS 2329e428ebbf484fa3ea9452133df567 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36848
I20260709 13:07:47.951776 26378 ts_tablet_manager.cc:1929] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:47.952294 26378 log.cc:1199] T a53b1fb6477d486e98bff9549cc487df P a2a77ecf2c0941618b5bc79d4c04db61: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-2-root/wals/a53b1fb6477d486e98bff9549cc487df
I20260709 13:07:47.954107 25131 catalog_manager.cc:5028] TS a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): tablet a53b1fb6477d486e98bff9549cc487df (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:47.959618 26396 tablet_replica.cc:333] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:07:47.960673 26396 raft_consensus.cc:2243] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:47.963143 26396 raft_consensus.cc:2272] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:47.967367 26396 ts_tablet_manager.cc:1916] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:47.977547 26396 ts_tablet_manager.cc:1929] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:47.977807 26396 log.cc:1199] T 6ff9a7b00f104dd294125870b06a7df4 P 2329e428ebbf484fa3ea9452133df567: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-1-root/wals/6ff9a7b00f104dd294125870b06a7df4
I20260709 13:07:47.979211 25128 catalog_manager.cc:5028] TS 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): tablet 6ff9a7b00f104dd294125870b06a7df4 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:49.031991 25318 consensus_queue.cc:237] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
W20260709 13:07:49.038033 25284 consensus_peers.cc:597] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de96ce5d28144cf3894e100bfc4959fa. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:49.038582 25393 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:49.040388 25244 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:49.041162 26400 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:49.042596 26364 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:49.050874 26400 raft_consensus.cc:2955] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:49.052767 25393 raft_consensus.cc:2955] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:49.067183 25315 consensus_queue.cc:237] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:49.079471 25244 raft_consensus.cc:2955] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
W20260709 13:07:49.086241 25284 consensus_peers.cc:597] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf0501919f944794bbab0ffae6a334e1. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:49.085826 25143 catalog_manager.cc:5697] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:49.088225 25393 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:49.091475 26403 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:49.107239 25244 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:49.109766 26364 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:49.141669 26404 raft_consensus.cc:2955] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:49.143676 25393 raft_consensus.cc:2955] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:49.144001 25244 raft_consensus.cc:2955] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:49.160781 25393 consensus_queue.cc:237] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:49.162106 25143 catalog_manager.cc:5697] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:49.174769 25244 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:07:49.176187 25358 consensus_peers.cc:597] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ead2e92f0614db79c9249687c4246f9. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:49.176448 26397 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:49.177776 25315 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:49.179582 26397 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:49.185858 26380 raft_consensus.cc:2955] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:49.189751 25315 raft_consensus.cc:2955] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:49.196290 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:49.194448 25244 raft_consensus.cc:2955] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:49.203907 25241 consensus_queue.cc:237] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:49.207986 25141 catalog_manager.cc:5697] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:49.226181 25393 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:49.226186 25315 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:07:49.228721 25209 consensus_peers.cc:597] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8fa7e9050d44475381c97025c2b11711. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:49.228792 26398 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:49.233174 26398 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:49.242527 26395 raft_consensus.cc:2955] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:49.244074 25315 raft_consensus.cc:2955] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:49.245489 25393 raft_consensus.cc:2955] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:49.260607 25140 catalog_manager.cc:5697] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:49.280203 25241 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "17ef4786898b4ad696baed95b4209788"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
mode: GRACEFUL
new_leader_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:37596
I20260709 13:07:49.280869 25241 raft_consensus.cc:606] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Received request to transfer leadership to TS f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:49.309765 25393 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
mode: GRACEFUL
new_leader_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:52318
I20260709 13:07:49.310287 25393 raft_consensus.cc:606] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Received request to transfer leadership to TS 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:49.312353 26343 raft_consensus.cc:993] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a: : Instructing follower f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:49.312810 25241 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
mode: GRACEFUL
new_leader_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:37596
I20260709 13:07:49.312851 26398 raft_consensus.cc:1081] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Signalling peer f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:49.313408 25241 raft_consensus.cc:606] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Received request to transfer leadership to TS f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:49.315089 25942 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "17ef4786898b4ad696baed95b4209788"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:59304
I20260709 13:07:49.315553 25942 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:49.315835 25942 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:49.320672 25942 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:49.321965 25241 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b62f4877af9742298b1e149789ca3090"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
mode: GRACEFUL
new_leader_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:37596
I20260709 13:07:49.322520 25241 raft_consensus.cc:606] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Received request to transfer leadership to TS 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:49.325670 25942 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:49.326815 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:49.327281 25244 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
mode: GRACEFUL
new_leader_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:37596
I20260709 13:07:49.327421 25394 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:49.327811 25244 raft_consensus.cc:606] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Received request to transfer leadership to TS 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:49.332789 25244 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d1bed174e340476591fddc8217b8f644"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
mode: GRACEFUL
new_leader_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:37596
I20260709 13:07:49.332810 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:49.333360 25244 raft_consensus.cc:606] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Received request to transfer leadership to TS 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:49.333516 25246 raft_consensus.cc:3055] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:49.333902 25246 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:49.334482 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:07:49.336007 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:07:49.335831 25246 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:49.336915 25246 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:49.340879 25246 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:49.332911 25394 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:49.342317 25911 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:07:49.344094 26422 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:49.345428 26422 raft_consensus.cc:697] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:07:49.347235 26422 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:49.356144 25143 catalog_manager.cc:5697] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 1 to 2, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 2 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:49.359504 26343 raft_consensus.cc:993] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: : Instructing follower 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:49.359988 26398 raft_consensus.cc:1081] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Signalling peer 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:49.361792 25315 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b62f4877af9742298b1e149789ca3090"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:36864
I20260709 13:07:49.362282 25315 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:49.362597 25315 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:49.371631 25315 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:49.374576 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:49.375089 25246 raft_consensus.cc:3055] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:49.375365 25246 raft_consensus.cc:740] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:49.376067 25246 consensus_queue.cc:260] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:49.377106 25246 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:49.383476 25246 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:49.384506 25315 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:49.384531 25285 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:49.385496 26400 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:49.385667 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:49.386004 26400 raft_consensus.cc:697] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:49.386235 25394 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:49.386893 26400 consensus_queue.cc:237] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:49.395706 25394 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:49.414224 25143 catalog_manager.cc:5697] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 2, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:49.438338 26343 raft_consensus.cc:993] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: : Instructing follower 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:49.438800 26398 raft_consensus.cc:1081] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Signalling peer 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:49.440387 25315 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d1bed174e340476591fddc8217b8f644"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:36864
I20260709 13:07:49.440892 25315 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:49.441232 25315 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:49.447871 25315 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:49.450907 25315 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:49.451385 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:49.451733 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:49.452091 25246 raft_consensus.cc:3055] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:49.452239 25394 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:49.452457 25246 raft_consensus.cc:740] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:49.453224 25246 consensus_queue.cc:260] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:49.454205 25246 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:49.455156 26398 raft_consensus.cc:993] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: : Instructing follower f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:49.455683 26398 raft_consensus.cc:1081] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Signalling peer f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:49.460456 25394 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:49.461149 25246 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:49.462016 25287 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:49.462815 26400 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:49.463249 26400 raft_consensus.cc:697] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:49.464046 26400 consensus_queue.cc:237] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:49.470116 25942 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:59304
I20260709 13:07:49.471086 25942 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:49.471481 25942 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:49.479252 25942 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:49.482360 25942 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:49.483412 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:49.484158 25394 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:49.486480 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:49.487026 25246 raft_consensus.cc:3055] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:49.487299 25246 raft_consensus.cc:740] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:49.487963 25246 consensus_queue.cc:260] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:49.489151 25246 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:49.499536 25394 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:49.500763 25911 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:07:49.501585 26422 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:49.502105 26422 raft_consensus.cc:697] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:07:49.503010 26427 ts_tablet_manager.cc:933] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:49.502923 26422 consensus_queue.cc:237] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:49.503656 25143 catalog_manager.cc:5697] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 2, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:49.505259 26427 tablet_copy_client.cc:323] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.66:42315
I20260709 13:07:49.507887 25246 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:49.509379 25331 tablet_copy_service.cc:140] P 2329e428ebbf484fa3ea9452133df567: Received BeginTabletCopySession request for tablet de96ce5d28144cf3894e100bfc4959fa from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:40928)
I20260709 13:07:49.509881 25331 tablet_copy_service.cc:161] P 2329e428ebbf484fa3ea9452133df567: Beginning new tablet copy session on tablet de96ce5d28144cf3894e100bfc4959fa from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:40928: session id = f646dfd532974dceadfe00005d2bbf59-de96ce5d28144cf3894e100bfc4959fa
I20260709 13:07:49.512306 25140 catalog_manager.cc:5697] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 1 to 2, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 2 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:49.516844 25331 tablet_copy_source_session.cc:215] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:49.519754 26427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de96ce5d28144cf3894e100bfc4959fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:49.533207 26427 tablet_copy_client.cc:806] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:49.533794 26427 tablet_copy_client.cc:670] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:49.537278 26427 tablet_copy_client.cc:538] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:49.543720 26427 tablet_bootstrap.cc:492] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:49.551199 26395 raft_consensus.cc:993] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: : Instructing follower 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:49.551723 26395 raft_consensus.cc:1081] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Signalling peer 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:49.554394 25315 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:36864
I20260709 13:07:49.554939 25315 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:49.555294 25315 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:49.566619 25315 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:49.569684 26427 tablet_bootstrap.cc:492] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:49.570013 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:49.570724 25246 raft_consensus.cc:3055] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:49.570801 26427 tablet_bootstrap.cc:492] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:49.571125 25246 raft_consensus.cc:740] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:49.571723 26427 ts_tablet_manager.cc:1403] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.028s	user 0.017s	sys 0.012s
I20260709 13:07:49.572063 25246 consensus_queue.cc:260] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:49.574339 25315 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:49.574854 26427 raft_consensus.cc:359] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:49.575742 26427 raft_consensus.cc:740] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:49.576349 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:49.576967 25394 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:49.576414 26427 consensus_queue.cc:260] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:49.577342 25246 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:49.578907 26427 ts_tablet_manager.cc:1434] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:07:49.581353 25331 tablet_copy_service.cc:342] P 2329e428ebbf484fa3ea9452133df567: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-de96ce5d28144cf3894e100bfc4959fa received from {username='slave'} at 127.0.0.1:40928
I20260709 13:07:49.581754 25331 tablet_copy_service.cc:434] P 2329e428ebbf484fa3ea9452133df567: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-de96ce5d28144cf3894e100bfc4959fa on tablet de96ce5d28144cf3894e100bfc4959fa with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:49.586894 25246 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:49.587999 25285 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:49.589120 26400 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:49.589600 26400 raft_consensus.cc:697] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:49.590437 26400 consensus_queue.cc:237] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:49.591245 25394 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:49.593699 26427 ts_tablet_manager.cc:933] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:49.643203 26427 tablet_copy_client.cc:323] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.67:34661
I20260709 13:07:49.646095 25405 tablet_copy_service.cc:140] P a2a77ecf2c0941618b5bc79d4c04db61: Received BeginTabletCopySession request for tablet 5ead2e92f0614db79c9249687c4246f9 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:41830)
I20260709 13:07:49.646615 25405 tablet_copy_service.cc:161] P a2a77ecf2c0941618b5bc79d4c04db61: Beginning new tablet copy session on tablet 5ead2e92f0614db79c9249687c4246f9 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:41830: session id = f646dfd532974dceadfe00005d2bbf59-5ead2e92f0614db79c9249687c4246f9
I20260709 13:07:49.648464 25140 catalog_manager.cc:5697] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 2, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:49.657069 25405 tablet_copy_source_session.cc:215] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:49.660497 26427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ead2e92f0614db79c9249687c4246f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:49.676204 26427 tablet_copy_client.cc:806] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:49.676821 26427 tablet_copy_client.cc:670] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:49.680099 26427 tablet_copy_client.cc:538] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:49.688676 26427 tablet_bootstrap.cc:492] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:49.705914 26434 ts_tablet_manager.cc:933] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:49.712337 26434 tablet_copy_client.cc:323] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.66:42315
I20260709 13:07:49.713835 25331 tablet_copy_service.cc:140] P 2329e428ebbf484fa3ea9452133df567: Received BeginTabletCopySession request for tablet cf0501919f944794bbab0ffae6a334e1 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:40928)
I20260709 13:07:49.714339 25331 tablet_copy_service.cc:161] P 2329e428ebbf484fa3ea9452133df567: Beginning new tablet copy session on tablet cf0501919f944794bbab0ffae6a334e1 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:40928: session id = f646dfd532974dceadfe00005d2bbf59-cf0501919f944794bbab0ffae6a334e1
I20260709 13:07:49.715039 26427 tablet_bootstrap.cc:492] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:49.715880 26427 tablet_bootstrap.cc:492] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:49.716485 26427 ts_tablet_manager.cc:1403] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.028s	user 0.022s	sys 0.003s
I20260709 13:07:49.719306 26427 raft_consensus.cc:359] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:49.720104 26427 raft_consensus.cc:740] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:49.720644 26397 raft_consensus.cc:993] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: : Instructing follower 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:49.720781 25331 tablet_copy_source_session.cc:215] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:49.721163 26397 raft_consensus.cc:1081] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Signalling peer 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:49.721272 26427 consensus_queue.cc:260] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:49.723501 25314 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:36876
I20260709 13:07:49.723758 26427 ts_tablet_manager.cc:1434] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.003s
I20260709 13:07:49.723974 25314 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:49.724350 25314 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:49.727646 26434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf0501919f944794bbab0ffae6a334e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:49.729681 25405 tablet_copy_service.cc:342] P a2a77ecf2c0941618b5bc79d4c04db61: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-5ead2e92f0614db79c9249687c4246f9 received from {username='slave'} at 127.0.0.1:41830
I20260709 13:07:49.730176 25405 tablet_copy_service.cc:434] P a2a77ecf2c0941618b5bc79d4c04db61: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-5ead2e92f0614db79c9249687c4246f9 on tablet 5ead2e92f0614db79c9249687c4246f9 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:49.737660 25314 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:49.742873 26434 tablet_copy_client.cc:806] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:49.743496 25314 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:49.743470 26434 tablet_copy_client.cc:670] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:49.744778 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:49.745364 25241 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:49.749204 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:49.749817 25393 raft_consensus.cc:3055] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:49.750133 25393 raft_consensus.cc:740] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:49.750921 25393 consensus_queue.cc:260] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:49.751614 26434 tablet_copy_client.cc:538] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:49.752334 25393 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:49.752668 26427 ts_tablet_manager.cc:933] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:07:49.755765 25241 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:49.756907 25285 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:49.757671 26400 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:49.758112 26400 raft_consensus.cc:697] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:49.758920 26400 consensus_queue.cc:237] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:49.761921 26434 tablet_bootstrap.cc:492] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:49.774173 25393 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:49.777020 25394 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:49.779388 26422 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:49.779881 26427 tablet_copy_client.cc:323] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.65:45313
I20260709 13:07:49.781415 25255 tablet_copy_service.cc:140] P 46de6919f85341c6a9b574dff112e35a: Received BeginTabletCopySession request for tablet 8fa7e9050d44475381c97025c2b11711 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:44954)
I20260709 13:07:49.781970 25255 tablet_copy_service.cc:161] P 46de6919f85341c6a9b574dff112e35a: Beginning new tablet copy session on tablet 8fa7e9050d44475381c97025c2b11711 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:44954: session id = f646dfd532974dceadfe00005d2bbf59-8fa7e9050d44475381c97025c2b11711
I20260709 13:07:49.793678 25255 tablet_copy_source_session.cc:215] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:49.797083 25140 catalog_manager.cc:5697] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 2, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:49.799530 26427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fa7e9050d44475381c97025c2b11711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:49.827135 26434 tablet_bootstrap.cc:492] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:49.828083 26434 tablet_bootstrap.cc:492] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:49.828734 26434 ts_tablet_manager.cc:1403] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.067s	user 0.023s	sys 0.004s
I20260709 13:07:49.835161 25393 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:49.839820 26400 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 13:07:49.840950 26434 raft_consensus.cc:359] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:49.841842 26434 raft_consensus.cc:740] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:49.842509 26434 consensus_queue.cc:260] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:49.844871 26434 ts_tablet_manager.cc:1434] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.000s
I20260709 13:07:49.851415 26427 tablet_copy_client.cc:806] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:49.851940 26427 tablet_copy_client.cc:670] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:49.853508 25331 tablet_copy_service.cc:342] P 2329e428ebbf484fa3ea9452133df567: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-cf0501919f944794bbab0ffae6a334e1 received from {username='slave'} at 127.0.0.1:40928
I20260709 13:07:49.854102 25331 tablet_copy_service.cc:434] P 2329e428ebbf484fa3ea9452133df567: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-cf0501919f944794bbab0ffae6a334e1 on tablet cf0501919f944794bbab0ffae6a334e1 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:49.855285 26427 tablet_copy_client.cc:538] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:49.864900 26427 tablet_bootstrap.cc:492] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:49.878897 25390 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:49.884011 26404 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:49.904887 25942 raft_consensus.cc:1217] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:49.907752 25241 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:49.917853 26422 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:49.929598 26427 tablet_bootstrap.cc:492] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:49.939495 26427 tablet_bootstrap.cc:492] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:49.940934 26427 ts_tablet_manager.cc:1403] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.076s	user 0.017s	sys 0.006s
I20260709 13:07:49.944162 26427 raft_consensus.cc:359] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:49.949116 26427 raft_consensus.cc:740] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:49.950290 26427 consensus_queue.cc:260] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:49.956475 26427 ts_tablet_manager.cc:1434] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.014s	user 0.002s	sys 0.002s
I20260709 13:07:49.956691 25246 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:49.962730 25255 tablet_copy_service.cc:342] P 46de6919f85341c6a9b574dff112e35a: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-8fa7e9050d44475381c97025c2b11711 received from {username='slave'} at 127.0.0.1:44954
I20260709 13:07:49.963145 25255 tablet_copy_service.cc:434] P 46de6919f85341c6a9b574dff112e35a: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-8fa7e9050d44475381c97025c2b11711 on tablet 8fa7e9050d44475381c97025c2b11711 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:50.010808 25942 raft_consensus.cc:1217] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:50.011658 25241 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:49.982671 26400 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
W20260709 13:07:50.016567 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 47309.5us
I20260709 13:07:50.024502 26364 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:07:50.026872 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 101829us
I20260709 13:07:50.048718 25246 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:50.050189 26422 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:50.064654 25394 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:50.066097 26423 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:50.105674 25394 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:50.107309 26364 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:50.120276 25246 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:50.123258 26404 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:50.306669 25394 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:50.307905 26404 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:50.321326 25241 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:50.322285 25945 raft_consensus.cc:1217] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:50.327700 26440 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:50.337162 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:50.356818 25942 raft_consensus.cc:1217] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:50.407058 26302 raft_consensus.cc:1064] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:50.409379 26302 consensus_queue.cc:237] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:50.411554 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:50.415472 25315 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:50.417680 26415 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:50.420338 25942 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:50.422664 25241 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:50.424855 26415 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:50.440757 26447 raft_consensus.cc:1064] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:50.443300 26447 consensus_queue.cc:237] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:50.454536 25942 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:50.455282 26397 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:50.456044 26440 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:50.461405 25394 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:50.462994 26447 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:50.465114 25246 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:50.466746 26440 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:50.481951 26302 raft_consensus.cc:2955] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:50.489828 26447 raft_consensus.cc:2955] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:50.504585 25942 raft_consensus.cc:2955] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:50.494917 25241 raft_consensus.cc:2955] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:50.527980 25142 catalog_manager.cc:5697] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:50.532662 25243 raft_consensus.cc:2955] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:50.536995 25945 raft_consensus.cc:2955] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:50.546149 25394 raft_consensus.cc:2955] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:50.549088 25141 catalog_manager.cc:5697] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:50.577447 25315 raft_consensus.cc:2955] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:50.597404 25318 consensus_queue.cc:237] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:50.596637 25387 consensus_queue.cc:237] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:50.610997 25394 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:50.614257 25945 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:50.615255 25942 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:50.616945 26404 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:07:50.619508 26397 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:50.619541 26404 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:07:50.625823 26404 raft_consensus.cc:2955] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:50.628878 25246 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:50.642251 25387 raft_consensus.cc:2955] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:50.646412 26472 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:50.640498 25942 raft_consensus.cc:2955] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:50.656054 26302 raft_consensus.cc:2955] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:50.657601 25945 raft_consensus.cc:2955] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:50.666713 25246 raft_consensus.cc:2955] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:50.692049 25131 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5ead2e92f0614db79c9249687c4246f9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 13:07:50.862072 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 180551us
I20260709 13:07:50.891841 25144 catalog_manager.cc:5697] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:50.903818 25246 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
mode: GRACEFUL
new_leader_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:37596
I20260709 13:07:50.904371 25246 raft_consensus.cc:606] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Received request to transfer leadership to TS f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:50.918606 25128 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet de96ce5d28144cf3894e100bfc4959fa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:50.957738 26479 raft_consensus.cc:1064] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:50.959946 26479 consensus_queue.cc:237] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:50.968492 25141 catalog_manager.cc:5244] ChangeConfig:REMOVE_PEER RPC for tablet 5ead2e92f0614db79c9249687c4246f9 with cas_config_opid_index 3: aborting the task: latest config opid_index 4; task opid_index 3
I20260709 13:07:50.973354 25387 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:52318
I20260709 13:07:50.973925 25387 raft_consensus.cc:606] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:50.978197 25310 tablet_service.cc:1558] Processing DeleteTablet for tablet 5ead2e92f0614db79c9249687c4246f9 with delete_type TABLET_DATA_TOMBSTONED (TS 2329e428ebbf484fa3ea9452133df567 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36848
I20260709 13:07:50.988651 25143 catalog_manager.cc:5697] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:51.003989 26484 tablet_replica.cc:333] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:07:51.004182 26364 raft_consensus.cc:1064] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
W20260709 13:07:51.007381 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 52618.1us
I20260709 13:07:51.007180 26364 consensus_queue.cc:237] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:51.010098 26478 raft_consensus.cc:993] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: : Instructing follower f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:51.010497 26478 raft_consensus.cc:1081] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Signalling peer f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:51.008728 25230 tablet_service.cc:1558] Processing DeleteTablet for tablet de96ce5d28144cf3894e100bfc4959fa with delete_type TABLET_DATA_TOMBSTONED (TS 46de6919f85341c6a9b574dff112e35a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32990
I20260709 13:07:51.023290 26487 tablet_replica.cc:333] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:07:51.023329 25942 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:51.023913 26484 raft_consensus.cc:2243] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:51.025794 25318 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:51.027114 26484 raft_consensus.cc:2272] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:51.032951 25243 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:51.036768 26440 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:51.046092 26484 ts_tablet_manager.cc:1916] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:51.054791 25942 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:51.056715 26482 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:51.072088 26484 ts_tablet_manager.cc:1929] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:51.072692 26484 log.cc:1199] T 5ead2e92f0614db79c9249687c4246f9 P 2329e428ebbf484fa3ea9452133df567: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-1-root/wals/5ead2e92f0614db79c9249687c4246f9
I20260709 13:07:51.075001 25945 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:59304
I20260709 13:07:51.075521 25945 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:51.075835 25945 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:51.082952 26487 raft_consensus.cc:2243] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:51.109866 25128 catalog_manager.cc:5028] TS 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): tablet 5ead2e92f0614db79c9249687c4246f9 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:51.122895 26487 raft_consensus.cc:2272] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:51.123684 25945 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:51.130059 25394 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:51.143070 25394 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:51.143497 25945 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:51.146504 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:51.147131 25393 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:07:51.147897 26489 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
I20260709 13:07:51.172796 26395 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:51.178238 26487 ts_tablet_manager.cc:1916] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:51.186611 26481 raft_consensus.cc:993] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61: : Instructing follower 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:07:51.187135 26481 raft_consensus.cc:1081] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Signalling peer 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:07:51.188827 25246 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:32994
I20260709 13:07:51.189333 25246 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:51.189641 25246 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:51.195807 25246 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:51.207583 26404 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:51.208468 25246 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:51.199304 25393 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:51.200785 26482 raft_consensus.cc:2955] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:51.202373 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:51.213349 25243 raft_consensus.cc:3055] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:51.213644 25243 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:51.215612 25243 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:51.216725 25243 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:51.219534 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:51.220104 25312 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:51.221166 26487 ts_tablet_manager.cc:1929] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:51.221632 26487 log.cc:1199] T de96ce5d28144cf3894e100bfc4959fa P 46de6919f85341c6a9b574dff112e35a: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-0-root/wals/de96ce5d28144cf3894e100bfc4959fa
I20260709 13:07:51.221876 25911 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:07:51.222874 26422 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:51.204465 25245 raft_consensus.cc:2955] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:51.205281 26395 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:07:51.198369 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 43735.2us
I20260709 13:07:51.223318 25129 catalog_manager.cc:5028] TS 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): tablet de96ce5d28144cf3894e100bfc4959fa (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:51.223340 26422 raft_consensus.cc:697] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:07:51.224920 26422 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:51.229856 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:51.230392 25390 raft_consensus.cc:3055] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:51.230664 25390 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:51.231351 25390 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:51.232347 25390 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:51.233261 26478 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:51.237269 25243 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:51.249253 25942 raft_consensus.cc:2955] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:51.256932 25390 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:07:51.263103 25212 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:51.264237 26491 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:51.264714 26491 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 2 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:51.265497 26491 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:51.268409 25143 catalog_manager.cc:5697] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:51.280015 25387 raft_consensus.cc:2955] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:51.281327 25321 consensus_queue.cc:237] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:51.293318 25942 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:51.292186 25143 catalog_manager.cc:5697] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 1 to 2, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 2 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:51.295007 25312 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:07:51.296376 26478 raft_consensus.cc:2955] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:51.298563 26482 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:51.303884 25390 raft_consensus.cc:2955] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:51.305497 25945 raft_consensus.cc:2955] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:51.318583 25245 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:51.322070 25144 catalog_manager.cc:5697] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:51.323508 26477 raft_consensus.cc:2955] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:51.325206 26447 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:51.326506 25321 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:07:51.327031 25321 raft_consensus.cc:606] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Received request to transfer leadership to TS f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:51.332063 25245 raft_consensus.cc:2955] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:51.335263 25128 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet cf0501919f944794bbab0ffae6a334e1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:51.348413 25144 catalog_manager.cc:5697] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:51.352406 25243 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
mode: GRACEFUL
new_leader_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:37596
I20260709 13:07:51.352969 25243 raft_consensus.cc:606] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Received request to transfer leadership to TS 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:51.320218 25312 raft_consensus.cc:2955] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:51.338428 25942 raft_consensus.cc:2955] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:51.361729 25245 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
mode: GRACEFUL
new_leader_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:37596
I20260709 13:07:51.343873 25143 catalog_manager.cc:5697] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 1 to 2, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 2 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:51.362332 25245 raft_consensus.cc:606] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Received request to transfer leadership to TS f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:51.368234 25378 tablet_service.cc:1558] Processing DeleteTablet for tablet cf0501919f944794bbab0ffae6a334e1 with delete_type TABLET_DATA_TOMBSTONED (TS a2a77ecf2c0941618b5bc79d4c04db61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60924
I20260709 13:07:51.371546 25245 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
mode: GRACEFUL
new_leader_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:37596
I20260709 13:07:51.372253 25245 raft_consensus.cc:606] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Received request to transfer leadership to TS 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:51.373611 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:07:51.375152 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:07:51.379463 26500 tablet_replica.cc:333] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:07:51.381291 26500 raft_consensus.cc:2243] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:51.381990 26500 raft_consensus.cc:2272] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:51.385779 26500 ts_tablet_manager.cc:1916] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:51.401734 25245 consensus_queue.cc:237] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:51.404095 26500 ts_tablet_manager.cc:1929] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:51.404527 26500 log.cc:1199] T cf0501919f944794bbab0ffae6a334e1 P a2a77ecf2c0941618b5bc79d4c04db61: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-2-root/wals/cf0501919f944794bbab0ffae6a334e1
I20260709 13:07:51.406317 25131 catalog_manager.cc:5028] TS a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): tablet cf0501919f944794bbab0ffae6a334e1 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:51.407051 25942 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:51.408095 25312 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:51.409801 26491 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:51.411067 26478 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:51.417344 26491 raft_consensus.cc:2955] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:51.422174 25942 raft_consensus.cc:2955] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:51.423926 25312 raft_consensus.cc:2955] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:51.432118 25129 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8fa7e9050d44475381c97025c2b11711 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:51.435075 26395 raft_consensus.cc:993] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a: : Instructing follower 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:51.435575 26395 raft_consensus.cc:1081] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Signalling peer 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:51.437021 25321 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:36864
I20260709 13:07:51.437551 25321 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:51.437884 25321 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:51.443651 25321 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:51.445420 25321 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:51.446537 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:51.447093 25245 raft_consensus.cc:3055] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:51.447767 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:51.448282 25390 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:51.447386 25245 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:51.449334 25144 catalog_manager.cc:5697] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:51.449627 25245 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:51.450840 25245 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:51.458165 25390 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:51.459532 25287 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:51.463007 26404 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:51.463479 26404 raft_consensus.cc:697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:51.464301 26404 consensus_queue.cc:237] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:51.466320 25245 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:51.468465 25378 tablet_service.cc:1558] Processing DeleteTablet for tablet 8fa7e9050d44475381c97025c2b11711 with delete_type TABLET_DATA_TOMBSTONED (TS a2a77ecf2c0941618b5bc79d4c04db61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60924
I20260709 13:07:51.469415 26500 tablet_replica.cc:333] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:07:51.470539 26500 raft_consensus.cc:2243] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:51.471115 26500 raft_consensus.cc:2272] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:51.474684 26500 ts_tablet_manager.cc:1916] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:51.482067 25141 catalog_manager.cc:5697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 2, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:51.499578 26500 ts_tablet_manager.cc:1929] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:51.500039 26500 log.cc:1199] T 8fa7e9050d44475381c97025c2b11711 P a2a77ecf2c0941618b5bc79d4c04db61: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-2-root/wals/8fa7e9050d44475381c97025c2b11711
I20260709 13:07:51.501507 25131 catalog_manager.cc:5028] TS a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): tablet 8fa7e9050d44475381c97025c2b11711 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:51.530189 26482 raft_consensus.cc:993] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567: : Instructing follower f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:51.530755 26404 raft_consensus.cc:1081] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Signalling peer f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:51.532279 25942 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:59324
I20260709 13:07:51.532799 25942 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:51.533154 25942 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:51.542161 25942 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:51.544265 25942 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:51.545598 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:51.545511 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:51.546231 25321 raft_consensus.cc:3055] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:51.546289 25245 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:51.546624 25321 raft_consensus.cc:740] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:51.547351 25321 consensus_queue.cc:260] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:51.548482 25321 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:51.554422 25245 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:51.555409 25909 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:07:51.556125 26422 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:51.556576 26422 raft_consensus.cc:697] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:07:51.557909 25321 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:51.557586 26422 consensus_queue.cc:237] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:51.565989 25141 catalog_manager.cc:5697] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 1 to 2, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 2 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:51.613209 26491 raft_consensus.cc:993] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a: : Instructing follower 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:51.613710 26478 raft_consensus.cc:1081] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Signalling peer 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:51.615342 25321 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:36864
I20260709 13:07:51.615928 25321 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:51.616323 25321 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:51.622588 25321 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:51.624769 25321 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:51.626035 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:51.626607 25246 raft_consensus.cc:3055] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:51.626909 25246 raft_consensus.cc:740] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:51.627415 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:51.627945 25387 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:51.627657 25246 consensus_queue.cc:260] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:51.628852 25246 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:51.634207 25387 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:51.634473 26478 raft_consensus.cc:993] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a: : Instructing follower f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:51.634923 26478 raft_consensus.cc:1081] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Signalling peer f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:51.635466 25246 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:51.636361 25287 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:51.636421 25943 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:59304
I20260709 13:07:51.637157 25943 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:51.637491 25943 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:51.637648 26404 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:51.638168 26404 raft_consensus.cc:697] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:51.639062 26404 consensus_queue.cc:237] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:51.644928 25943 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:51.647050 25943 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:51.647174 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:51.647578 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:51.647902 25246 raft_consensus.cc:3055] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:51.648145 25312 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:51.648329 25246 raft_consensus.cc:740] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:51.649075 25246 consensus_queue.cc:260] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:51.649655 25141 catalog_manager.cc:5697] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 2, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:51.650264 25246 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:51.654654 25312 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:51.655843 25908 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:07:51.656711 26422 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:51.656867 25246 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:51.657407 26422 raft_consensus.cc:697] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:07:51.658246 26422 consensus_queue.cc:237] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:51.659297 25246 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:51.661084 26423 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:51.667297 25141 catalog_manager.cc:5697] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 1 to 2, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 2 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:51.672745 25387 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:51.676981 26422 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:51.745203 25312 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:51.746284 26478 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:07:51.747264 25175 debug-util.cc:398] Leaking SignalData structure 0x7b0800091640 after lost signal to thread 24116
W20260709 13:07:51.748250 25175 debug-util.cc:398] Leaking SignalData structure 0x7b080003e060 after lost signal to thread 25178
W20260709 13:07:51.749401 25175 debug-util.cc:398] Leaking SignalData structure 0x7b08002f40e0 after lost signal to thread 25270
W20260709 13:07:51.750478 25175 debug-util.cc:398] Leaking SignalData structure 0x7b080032d320 after lost signal to thread 25345
W20260709 13:07:51.751628 25175 debug-util.cc:398] Leaking SignalData structure 0x7b0800304140 after lost signal to thread 25419
W20260709 13:07:51.752580 25175 debug-util.cc:398] Leaking SignalData structure 0x7b08002fdbc0 after lost signal to thread 25969
I20260709 13:07:51.755420 25390 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:51.756948 26478 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:51.860160 25390 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:51.861495 26482 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:51.870697 25246 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:51.885206 26482 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260709 13:07:52.083215 25245 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:52.085381 26422 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:52.096246 25312 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:52.098953 26422 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:52.122287 25245 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:52.125586 26422 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:52.141252 25312 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:52.143486 26422 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:52.196800 25387 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:52.198292 26404 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:52.212352 25245 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:52.213658 26440 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:52.376058 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:52.421293 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:52.470149 25387 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
mode: GRACEFUL
new_leader_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:52318
I20260709 13:07:52.470683 25387 raft_consensus.cc:606] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Received request to transfer leadership to TS 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:52.473366 25246 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8f209a6539f54de28906492f142efb6b"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
mode: GRACEFUL
new_leader_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:37596
I20260709 13:07:52.473901 25246 raft_consensus.cc:606] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Received request to transfer leadership to TS f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:52.477842 25312 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:07:52.478399 25312 raft_consensus.cc:606] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:52.487323 25312 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:07:52.487865 25312 raft_consensus.cc:606] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Received request to transfer leadership to TS f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:52.495311 25387 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
mode: GRACEFUL
new_leader_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:52318
I20260709 13:07:52.496033 25387 raft_consensus.cc:606] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Received request to transfer leadership to TS 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:52.500306 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:07:52.501905 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:07:52.527670 26440 raft_consensus.cc:993] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567: : Instructing follower 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:07:52.528292 26440 raft_consensus.cc:1081] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Signalling peer 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:07:52.530339 25245 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:32992
I20260709 13:07:52.530854 25245 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:52.531167 25245 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:52.539328 25245 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:52.541920 26397 raft_consensus.cc:993] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: : Instructing follower 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:52.541988 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:52.542431 26520 raft_consensus.cc:1081] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Signalling peer 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:52.542692 25387 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:52.543313 25245 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:52.546478 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:52.547032 25312 raft_consensus.cc:3055] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:52.547328 25312 raft_consensus.cc:740] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:52.548055 25312 consensus_queue.cc:260] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:52.549222 25312 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:52.549823 25387 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:07:52.551327 26491 raft_consensus.cc:993] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: : Instructing follower f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:52.551855 26491 raft_consensus.cc:1081] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Signalling peer f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:52.552225 25212 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:52.552991 26523 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:52.553622 26523 raft_consensus.cc:697] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:52.554651 26523 consensus_queue.cc:237] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:52.557405 25318 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:36876
I20260709 13:07:52.557947 25318 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:52.558274 25318 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:52.560760 25943 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8f209a6539f54de28906492f142efb6b"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:59304
I20260709 13:07:52.561409 25943 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:52.561797 25943 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:52.567133 25318 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:52.569490 25312 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:07:52.570039 25318 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:52.572364 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:52.572996 25390 raft_consensus.cc:3055] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:52.573385 25390 raft_consensus.cc:740] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:52.574122 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:52.574733 25246 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:52.574280 25390 consensus_queue.cc:260] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:52.575649 25390 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:52.577557 25943 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:52.579522 25943 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:52.581911 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:52.582566 25243 raft_consensus.cc:3055] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:52.582902 25243 raft_consensus.cc:740] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:52.583796 25243 consensus_queue.cc:260] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:52.585172 25246 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:52.586673 25285 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:52.588063 26404 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:52.588601 26404 raft_consensus.cc:697] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:52.589570 26404 consensus_queue.cc:237] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:52.596014 25387 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:52.599046 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:52.599627 25321 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:52.585237 25243 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:52.606793 25321 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:52.616860 25908 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:07:52.619231 26423 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:52.619801 26423 raft_consensus.cc:697] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:07:52.620826 26423 consensus_queue.cc:237] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:52.641729 25243 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:52.640141 25390 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
W20260709 13:07:52.666941 25358 consensus_peers.cc:597] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8fd399e2f6384e98ada57177a6010d14. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:52.678452 25143 catalog_manager.cc:5697] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 2, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:52.684027 25243 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:52.693850 26397 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:07:52.698495 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 59931.7us
I20260709 13:07:52.710819 25144 catalog_manager.cc:5697] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 2 to 3, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 3 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:52.738489 26482 raft_consensus.cc:993] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: : Instructing follower f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:52.731534 25312 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:07:52.740477 26526 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 13:07:52.742632 26440 raft_consensus.cc:1081] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Signalling peer f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:52.746801 25243 raft_consensus.cc:2955] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:52.749425 26528 raft_consensus.cc:2955] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:52.767647 25141 catalog_manager.cc:5697] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 1 to 2, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 2 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:07:52.786298 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 48335.7us
I20260709 13:07:52.787619 26397 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:07:52.791814 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 60196.6us
I20260709 13:07:52.793906 25943 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:59324
I20260709 13:07:52.794416 25943 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:52.794739 25943 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:52.806082 26397 raft_consensus.cc:993] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61: : Instructing follower 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:52.806631 26397 raft_consensus.cc:1081] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Signalling peer 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:52.808781 25321 raft_consensus.cc:2955] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:52.812336 25312 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:36876
I20260709 13:07:52.812873 25312 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:52.813238 25312 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:52.819947 25312 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:07:52.822479 25192 auto_rebalancer.cc:629] Failed to schedule move for tablet e5cbaabf54304de089abde51df6dc2be: Remote error: Service unavailable: leader transfer in progress
I20260709 13:07:52.825168 25312 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:52.825970 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:52.826532 25243 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:52.823227 25943 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:52.831480 25390 consensus_queue.cc:237] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:52.832830 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:52.833395 25246 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:52.833316 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:52.833899 25387 raft_consensus.cc:3055] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:52.834177 25387 raft_consensus.cc:740] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:52.834857 25387 consensus_queue.cc:260] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:52.835815 25387 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:52.841971 25243 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:52.850505 25387 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:52.847673 25246 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:52.851557 25285 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:52.849561 25943 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:52.849999 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:52.852638 26404 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:52.852730 25318 raft_consensus.cc:3055] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:52.853274 25318 raft_consensus.cc:740] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:52.854296 25318 consensus_queue.cc:260] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:52.855938 25318 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:52.857556 25909 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:07:52.859737 26423 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:52.860241 26423 raft_consensus.cc:697] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:07:52.864931 26423 consensus_queue.cc:237] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:52.865568 25142 catalog_manager.cc:5697] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:52.876904 26404 raft_consensus.cc:697] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:52.877740 26404 consensus_queue.cc:237] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:52.887151 25312 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:07:52.885448 25358 consensus_peers.cc:597] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f406a4d41bb9441bba0a83c1d6a56aa3. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:52.889698 25318 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:52.890771 25246 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:52.894080 26528 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260709 13:07:52.906121 26533 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:52.949795 25143 catalog_manager.cc:5697] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 1 to 2, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 2 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:52.952924 26533 raft_consensus.cc:2955] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:52.960316 25246 raft_consensus.cc:2955] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:52.960893 25312 raft_consensus.cc:2955] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:52.989552 25142 catalog_manager.cc:5697] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:52.995672 25243 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:53.009647 25141 catalog_manager.cc:5697] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 2, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
W20260709 13:07:53.014282 25209 consensus_peers.cc:597] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94b9d971fbc94215a65a2247f2e2acb1. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:53.013674 25390 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:53.016175 25318 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:53.016574 26532 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:53.017822 26523 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:53.023674 25387 raft_consensus.cc:1275] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:53.028175 26530 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:53.029150 25243 raft_consensus.cc:1275] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:53.032440 26530 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:53.036649 25312 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:53.038420 26423 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:07:53.060685 26532 raft_consensus.cc:2955] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:53.075692 25321 raft_consensus.cc:2955] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:53.081384 25393 raft_consensus.cc:2955] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:53.096302 25318 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:53.129976 25246 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:53.148541 26478 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:07:53.195322 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 79626.7us
W20260709 13:07:53.199400 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 85454.9us
I20260709 13:07:53.214594 25390 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:53.221912 26491 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:53.250751 26562 ts_tablet_manager.cc:933] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:53.255030 25143 catalog_manager.cc:5697] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:53.263373 26562 tablet_copy_client.cc:323] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.67:34661
I20260709 13:07:53.267625 25405 tablet_copy_service.cc:140] P a2a77ecf2c0941618b5bc79d4c04db61: Received BeginTabletCopySession request for tablet 8fd399e2f6384e98ada57177a6010d14 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:41830)
I20260709 13:07:53.268359 25405 tablet_copy_service.cc:161] P a2a77ecf2c0941618b5bc79d4c04db61: Beginning new tablet copy session on tablet 8fd399e2f6384e98ada57177a6010d14 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:41830: session id = f646dfd532974dceadfe00005d2bbf59-8fd399e2f6384e98ada57177a6010d14
W20260709 13:07:53.273217 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 138414us
I20260709 13:07:53.275246 26518 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:07:53.283901 25243 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:53.286531 26422 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:07:53.311514 26571 ts_tablet_manager.cc:933] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:53.319574 25318 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:53.321156 26423 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:07:53.328606 25405 tablet_copy_source_session.cc:215] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:53.332373 26562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fd399e2f6384e98ada57177a6010d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:53.345522 25395 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:53.348145 26558 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:53.381004 26571 tablet_copy_client.cc:323] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.67:34661
I20260709 13:07:53.383356 25405 tablet_copy_service.cc:140] P a2a77ecf2c0941618b5bc79d4c04db61: Received BeginTabletCopySession request for tablet f406a4d41bb9441bba0a83c1d6a56aa3 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:41830)
I20260709 13:07:53.383955 25405 tablet_copy_service.cc:161] P a2a77ecf2c0941618b5bc79d4c04db61: Beginning new tablet copy session on tablet f406a4d41bb9441bba0a83c1d6a56aa3 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:41830: session id = f646dfd532974dceadfe00005d2bbf59-f406a4d41bb9441bba0a83c1d6a56aa3
I20260709 13:07:53.391562 25405 tablet_copy_source_session.cc:215] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:53.401675 26571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f406a4d41bb9441bba0a83c1d6a56aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:53.405925 25240 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:53.410099 26558 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:53.420030 26571 tablet_copy_client.cc:806] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:53.420610 26571 tablet_copy_client.cc:670] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:53.424256 26562 tablet_copy_client.cc:806] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:53.425020 26562 tablet_copy_client.cc:670] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:53.429078 26562 tablet_copy_client.cc:538] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:53.435251 26571 tablet_copy_client.cc:538] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:53.451100 26562 tablet_bootstrap.cc:492] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:53.468415 26571 tablet_bootstrap.cc:492] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:53.502759 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:53.505142 26562 tablet_bootstrap.cc:492] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:53.510031 26562 tablet_bootstrap.cc:492] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:53.510710 26562 ts_tablet_manager.cc:1403] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.060s	user 0.014s	sys 0.010s
I20260709 13:07:53.520798 26562 raft_consensus.cc:359] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:53.521777 26562 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:53.522531 26562 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:53.530939 26562 ts_tablet_manager.cc:1434] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.002s
I20260709 13:07:53.534268 25405 tablet_copy_service.cc:342] P a2a77ecf2c0941618b5bc79d4c04db61: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-8fd399e2f6384e98ada57177a6010d14 received from {username='slave'} at 127.0.0.1:41830
I20260709 13:07:53.534780 25405 tablet_copy_service.cc:434] P a2a77ecf2c0941618b5bc79d4c04db61: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-8fd399e2f6384e98ada57177a6010d14 on tablet 8fd399e2f6384e98ada57177a6010d14 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:53.571475 26571 tablet_bootstrap.cc:492] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:53.572479 26571 tablet_bootstrap.cc:492] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:53.573915 26571 ts_tablet_manager.cc:1403] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.107s	user 0.014s	sys 0.018s
I20260709 13:07:53.577523 26571 raft_consensus.cc:359] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:53.578435 26571 raft_consensus.cc:740] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:53.579082 26571 consensus_queue.cc:260] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:53.581403 26571 ts_tablet_manager.cc:1434] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:07:53.589419 25405 tablet_copy_service.cc:342] P a2a77ecf2c0941618b5bc79d4c04db61: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-f406a4d41bb9441bba0a83c1d6a56aa3 received from {username='slave'} at 127.0.0.1:41830
I20260709 13:07:53.589881 25405 tablet_copy_service.cc:434] P a2a77ecf2c0941618b5bc79d4c04db61: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-f406a4d41bb9441bba0a83c1d6a56aa3 on tablet f406a4d41bb9441bba0a83c1d6a56aa3 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:53.598466 26562 ts_tablet_manager.cc:933] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:07:53.623615 26562 tablet_copy_client.cc:323] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.65:45313
I20260709 13:07:53.625248 25255 tablet_copy_service.cc:140] P 46de6919f85341c6a9b574dff112e35a: Received BeginTabletCopySession request for tablet 94b9d971fbc94215a65a2247f2e2acb1 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:44954)
I20260709 13:07:53.625763 25255 tablet_copy_service.cc:161] P 46de6919f85341c6a9b574dff112e35a: Beginning new tablet copy session on tablet 94b9d971fbc94215a65a2247f2e2acb1 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:44954: session id = f646dfd532974dceadfe00005d2bbf59-94b9d971fbc94215a65a2247f2e2acb1
I20260709 13:07:53.633477 25255 tablet_copy_source_session.cc:215] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:53.636605 26562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b9d971fbc94215a65a2247f2e2acb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:53.656075 26562 tablet_copy_client.cc:806] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:53.656703 26562 tablet_copy_client.cc:670] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:53.661816 26562 tablet_copy_client.cc:538] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:53.672933 26562 tablet_bootstrap.cc:492] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:53.673200 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:53.705646 26562 tablet_bootstrap.cc:492] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:53.706638 26562 tablet_bootstrap.cc:492] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:53.707765 26562 ts_tablet_manager.cc:1403] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.035s	user 0.017s	sys 0.011s
I20260709 13:07:53.710758 26562 raft_consensus.cc:359] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:53.712217 26562 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:53.712949 26562 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:53.720620 26562 ts_tablet_manager.cc:1434] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20260709 13:07:53.724960 25255 tablet_copy_service.cc:342] P 46de6919f85341c6a9b574dff112e35a: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-94b9d971fbc94215a65a2247f2e2acb1 received from {username='slave'} at 127.0.0.1:44954
I20260709 13:07:53.725400 25255 tablet_copy_service.cc:434] P 46de6919f85341c6a9b574dff112e35a: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-94b9d971fbc94215a65a2247f2e2acb1 on tablet 94b9d971fbc94215a65a2247f2e2acb1 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:53.806205 25943 raft_consensus.cc:1217] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
W20260709 13:07:53.878767 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 54657.9us
I20260709 13:07:53.918112 25943 raft_consensus.cc:1217] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:53.999339 25394 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
mode: GRACEFUL
new_leader_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:52318
I20260709 13:07:54.000468 25394 raft_consensus.cc:606] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Received request to transfer leadership to TS 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:54.005851 26585 raft_consensus.cc:993] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61: : Instructing follower 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:54.006515 26583 raft_consensus.cc:1081] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Signalling peer 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:54.006788 25318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:07:54.007508 25318 raft_consensus.cc:606] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Received request to transfer leadership to TS f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:54.008579 25312 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:36876
I20260709 13:07:54.009289 25312 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:54.009748 25312 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:54.022826 25312 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:54.025907 25243 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
mode: GRACEFUL
new_leader_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:37596
I20260709 13:07:54.026535 25243 raft_consensus.cc:606] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Received request to transfer leadership to TS f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:54.027674 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:54.026515 25312 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:07:54.028483 25394 raft_consensus.cc:3055] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:54.028852 25394 raft_consensus.cc:740] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:54.029829 25394 consensus_queue.cc:260] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:54.031013 25394 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:54.034292 25395 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:52318
I20260709 13:07:54.034935 25395 raft_consensus.cc:606] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:54.038967 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:07:54.039619 25942 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:54.054244 25312 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d1bed174e340476591fddc8217b8f644"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:07:54.054926 25312 raft_consensus.cc:606] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:54.056241 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:07:54.058640 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:07:54.065443 25942 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:54.073184 25394 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:54.075208 25287 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:54.076517 26559 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:54.077045 26559 raft_consensus.cc:697] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:54.078060 26559 consensus_queue.cc:237] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:54.108269 25141 catalog_manager.cc:5697] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 2, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:54.147732 26585 raft_consensus.cc:993] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: : Instructing follower 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:07:54.148263 26585 raft_consensus.cc:1081] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Signalling peer 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:07:54.149986 25243 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:32994
I20260709 13:07:54.150512 25243 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:54.150853 25243 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:54.149996 25942 raft_consensus.cc:1217] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:54.177871 25243 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:54.182395 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:54.182979 25395 raft_consensus.cc:3055] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:54.183285 25395 raft_consensus.cc:740] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:54.184330 25243 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:07:54.184188 25395 consensus_queue.cc:260] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:54.185217 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:07:54.185370 25395 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:54.185792 25942 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:54.203042 25942 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:07:54.204249 25209 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:07:54.204869 25395 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:07:54.205240 26523 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:54.205801 26523 raft_consensus.cc:697] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 2 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:54.206712 26523 consensus_queue.cc:237] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:54.230482 25141 catalog_manager.cc:5697] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 1 to 2, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 2 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:54.258739 26559 raft_consensus.cc:993] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567: : Instructing follower 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:07:54.259209 26559 raft_consensus.cc:1081] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Signalling peer 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:07:54.263278 25243 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d1bed174e340476591fddc8217b8f644"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:32992
I20260709 13:07:54.263933 25243 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:54.265856 25243 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:54.266670 26587 raft_consensus.cc:1064] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:54.269419 26587 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:54.284685 25243 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:54.294741 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:54.295324 25312 raft_consensus.cc:3055] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:54.295594 25312 raft_consensus.cc:740] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:54.296396 25312 consensus_queue.cc:260] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:54.297461 25312 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:54.299223 25318 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:54.301620 26585 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:54.308200 25312 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:07:54.311825 25942 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:54.313588 25240 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:54.314491 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:54.315256 25394 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:54.312450 25243 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:54.316073 26585 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:54.317880 26585 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:54.321712 25394 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:07:54.323788 25212 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:54.324679 26523 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:54.325158 26523 raft_consensus.cc:697] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:54.326030 26523 consensus_queue.cc:237] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:54.339648 26558 raft_consensus.cc:993] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: : Instructing follower f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:54.340132 26558 raft_consensus.cc:1081] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Signalling peer f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:54.342692 26561 raft_consensus.cc:993] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: : Instructing follower f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:54.343386 25943 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:59324
I20260709 13:07:54.344061 25943 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:54.344496 25943 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:54.349409 26491 raft_consensus.cc:1081] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Signalling peer f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:54.352017 25945 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:59304
I20260709 13:07:54.352672 25945 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:54.353047 25945 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:54.375728 25945 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:54.387126 26472 raft_consensus.cc:2955] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:54.388521 25318 raft_consensus.cc:2955] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:54.390043 25141 catalog_manager.cc:5697] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 2 to 3, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 3 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:54.404454 25943 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:54.407298 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:54.407680 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:54.407897 25321 raft_consensus.cc:3055] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:54.408012 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:54.408279 25321 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:54.408581 25394 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:54.408295 25240 raft_consensus.cc:3055] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:54.409286 25240 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:54.409169 25321 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:54.410454 25321 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:54.410194 25240 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:54.411325 25240 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:54.405937 25243 raft_consensus.cc:2955] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:54.419481 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:54.416188 25394 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:54.420068 25312 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:54.424043 25321 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:54.425321 25911 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:07:54.427102 26423 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:54.428123 26423 raft_consensus.cc:697] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:07:54.429072 26423 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:54.430083 25240 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:54.433934 25909 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:07:54.434562 25312 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:54.434820 26518 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:54.438668 26518 raft_consensus.cc:697] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:07:54.425731 25942 raft_consensus.cc:2955] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:54.439724 26518 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:54.445533 25945 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:07:54.406626 25943 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:54.450455 25143 catalog_manager.cc:5697] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:54.471021 25144 catalog_manager.cc:5697] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 1 to 2, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 2 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:54.477787 26583 raft_consensus.cc:1064] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:54.480645 26583 consensus_queue.cc:237] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:54.496292 25243 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:54.497320 25321 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:54.498833 25942 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:54.500874 26587 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:54.501725 26472 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:54.504451 26583 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:54.529981 25394 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:54.534722 25140 catalog_manager.cc:5697] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 1 to 2, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 2 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:54.539064 25945 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:54.541247 25312 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:54.542898 26587 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:54.546012 26583 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:54.566430 26585 raft_consensus.cc:2955] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:54.568935 26583 raft_consensus.cc:2955] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:54.581045 25943 raft_consensus.cc:2955] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:54.582017 26523 raft_consensus.cc:1064] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:54.592916 25394 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:54.594151 25941 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:54.596930 26584 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:07:54.602382 26523 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:54.594770 25314 raft_consensus.cc:2955] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:54.613164 26566 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:07:54.622498 25321 raft_consensus.cc:2955] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:54.651376 25131 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8fd399e2f6384e98ada57177a6010d14 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:54.652212 25243 raft_consensus.cc:2955] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:54.665257 25943 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:54.665257 25318 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:54.667110 26584 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:54.672621 25391 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:54.672500 26523 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:54.694621 25943 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:54.708348 26612 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:54.723462 25140 catalog_manager.cc:5697] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:54.718941 25144 catalog_manager.cc:5697] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:54.748407 26404 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260709 13:07:54.818507 25387 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:54.830646 26561 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:07:54.832760 25944 raft_consensus.cc:2955] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:54.890130 26627 raft_consensus.cc:2955] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:54.913717 25394 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:54.916095 26518 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:54.934990 25314 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:54.946585 25942 raft_consensus.cc:2955] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:54.944618 25314 raft_consensus.cc:2955] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:54.942157 26627 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260709 13:07:55.041144 25318 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:55.052685 25390 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:55.059898 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:07:55.064399 26616 log.cc:927] Time spent T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.065s	user 0.002s	sys 0.000s
W20260709 13:07:55.065512 26633 log.cc:927] Time spent T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.060s	user 0.002s	sys 0.000s
I20260709 13:07:55.077989 25391 raft_consensus.cc:2955] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:55.081547 25230 tablet_service.cc:1558] Processing DeleteTablet for tablet 8fd399e2f6384e98ada57177a6010d14 with delete_type TABLET_DATA_TOMBSTONED (TS 46de6919f85341c6a9b574dff112e35a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32990
I20260709 13:07:55.085657 26628 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260709 13:07:55.090488 25312 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:55.111405 25243 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:55.133258 26636 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
W20260709 13:07:55.158627 26629 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 13:07:55.121260 25141 catalog_manager.cc:5697] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:55.163395 26639 tablet_replica.cc:333] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:07:55.164603 26639 raft_consensus.cc:2243] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:55.115900 26637 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
I20260709 13:07:55.191219 26636 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:55.165287 26639 raft_consensus.cc:2272] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:55.256067 26639 ts_tablet_manager.cc:1916] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:55.310858 26639 ts_tablet_manager.cc:1929] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:55.311544 26639 log.cc:1199] T 8fd399e2f6384e98ada57177a6010d14 P 46de6919f85341c6a9b574dff112e35a: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-0-root/wals/8fd399e2f6384e98ada57177a6010d14
I20260709 13:07:55.569918 25129 catalog_manager.cc:5028] TS 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): tablet 8fd399e2f6384e98ada57177a6010d14 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:55.602145 25240 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:07:55.634258 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 49269.6us
I20260709 13:07:55.686759 25942 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:55.700270 25318 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:55.718727 26623 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 13:07:55.761330 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 51729.7us
I20260709 13:07:55.765027 26623 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:55.771482 25387 consensus_queue.cc:237] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:55.781419 26623 raft_consensus.cc:2955] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:55.796829 25945 raft_consensus.cc:2955] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:55.796849 25318 raft_consensus.cc:2955] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:55.820626 25129 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 94b9d971fbc94215a65a2247f2e2acb1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:55.828009 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:55.828151 25143 catalog_manager.cc:5697] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index 3 to 4, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New cstate: current_term: 1 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:55.891182 25945 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:55.897006 26472 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:55.902098 25318 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:07:55.904265 26618 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:55.906291 26618 raft_consensus.cc:2955] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:55.932973 25378 tablet_service.cc:1558] Processing DeleteTablet for tablet 94b9d971fbc94215a65a2247f2e2acb1 with delete_type TABLET_DATA_TOMBSTONED (TS a2a77ecf2c0941618b5bc79d4c04db61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60924
I20260709 13:07:55.939385 25945 raft_consensus.cc:2955] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:55.953514 25131 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet f406a4d41bb9441bba0a83c1d6a56aa3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:07:55.966652 26657 tablet_replica.cc:333] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:07:55.967963 26657 raft_consensus.cc:2243] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:55.973016 26657 raft_consensus.cc:2272] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:55.977631 26657 ts_tablet_manager.cc:1916] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:55.988440 25318 raft_consensus.cc:2955] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:56.030709 25143 catalog_manager.cc:5697] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index 3 to 4, VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65) evicted. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:56.056113 25230 tablet_service.cc:1558] Processing DeleteTablet for tablet f406a4d41bb9441bba0a83c1d6a56aa3 with delete_type TABLET_DATA_TOMBSTONED (TS 46de6919f85341c6a9b574dff112e35a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32990
I20260709 13:07:56.063407 26657 ts_tablet_manager.cc:1929] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:56.063948 26657 log.cc:1199] T 94b9d971fbc94215a65a2247f2e2acb1 P a2a77ecf2c0941618b5bc79d4c04db61: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-2-root/wals/94b9d971fbc94215a65a2247f2e2acb1
I20260709 13:07:56.066700 25131 catalog_manager.cc:5028] TS a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): tablet 94b9d971fbc94215a65a2247f2e2acb1 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:56.087981 25387 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
mode: GRACEFUL
new_leader_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:52318
I20260709 13:07:56.088557 25387 raft_consensus.cc:606] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Received request to transfer leadership to TS 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:56.090186 26658 tablet_replica.cc:333] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:07:56.091271 26658 raft_consensus.cc:2243] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:56.091413 25243 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
mode: GRACEFUL
new_leader_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:37596
I20260709 13:07:56.091988 25243 raft_consensus.cc:606] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 2 LEADER]: Received request to transfer leadership to TS f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:56.092046 26658 raft_consensus.cc:2272] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:56.096392 25243 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
mode: GRACEFUL
new_leader_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:37596
I20260709 13:07:56.096915 25243 raft_consensus.cc:606] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Received request to transfer leadership to TS 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:56.099946 26658 ts_tablet_manager.cc:1916] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:07:56.101329 25312 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:07:56.101843 25312 raft_consensus.cc:606] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Received request to transfer leadership to TS f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:56.104393 25387 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:52318
I20260709 13:07:56.104893 25387 raft_consensus.cc:606] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 LEADER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:56.106280 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:07:56.107905 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:07:56.116429 26658 ts_tablet_manager.cc:1929] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:07:56.116937 26658 log.cc:1199] T f406a4d41bb9441bba0a83c1d6a56aa3 P 46de6919f85341c6a9b574dff112e35a: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-0-root/wals/f406a4d41bb9441bba0a83c1d6a56aa3
I20260709 13:07:56.119138 25129 catalog_manager.cc:5028] TS 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): tablet f406a4d41bb9441bba0a83c1d6a56aa3 (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:07:56.214763 26618 raft_consensus.cc:993] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61: : Instructing follower 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:07:56.215413 26472 raft_consensus.cc:1081] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 LEADER]: Signalling peer 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:07:56.216815 25240 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:32994
I20260709 13:07:56.217299 25240 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:56.217607 25240 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:56.222411 25240 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:56.224001 25240 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:56.224810 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:56.224975 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:56.225431 25312 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:56.225597 25390 raft_consensus.cc:3055] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:56.225967 25390 raft_consensus.cc:740] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:56.226821 25390 consensus_queue.cc:260] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:56.227931 25390 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:56.232810 25390 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:07:56.232825 25312 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:07:56.234834 25318 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 2. Current term is 3. Ops: []
I20260709 13:07:56.234714 25212 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:56.235745 26623 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:56.236225 26623 raft_consensus.cc:697] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:56.237179 26623 consensus_queue.cc:237] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:56.252741 25140 catalog_manager.cc:5697] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 2 to 3, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 3 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:56.354871 26650 raft_consensus.cc:993] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567: : Instructing follower f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:56.355384 26650 raft_consensus.cc:1081] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Signalling peer f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:56.362339 25942 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:59324
I20260709 13:07:56.362845 25942 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:56.363180 25942 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:56.369338 25942 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:56.371554 25942 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:56.371991 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:56.373652 25240 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:56.373039 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:56.375820 25312 raft_consensus.cc:3055] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:56.376112 25312 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:56.376854 25312 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:56.377938 25312 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:56.384087 25240 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:56.385399 25909 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:07:56.386304 26661 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:56.386761 26661 raft_consensus.cc:697] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:07:56.387617 26661 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:56.408095 25312 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:07:56.410908 25140 catalog_manager.cc:5697] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 1 to 2, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 2 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:56.455488 26622 raft_consensus.cc:993] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: : Instructing follower 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:56.455958 26622 raft_consensus.cc:1081] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Signalling peer 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:56.457438 25312 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:36876
I20260709 13:07:56.457986 25312 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:56.458341 25312 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:56.465770 25312 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:56.468943 25312 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:07:56.471010 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:56.472748 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:07:56.474524 25390 raft_consensus.cc:3055] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:56.474884 25942 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:56.475024 25390 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:56.477454 25390 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:56.478729 25390 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:56.481983 25942 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:56.483232 25284 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:07:56.486079 26650 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:56.486596 26650 raft_consensus.cc:697] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:56.487258 25390 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:56.487879 26650 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:56.492722 26662 raft_consensus.cc:993] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: : Instructing follower 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:56.493355 26662 raft_consensus.cc:1081] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Signalling peer 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:56.496436 25312 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:36864
I20260709 13:07:56.496973 25312 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:56.497351 25312 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:56.504136 25312 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:56.506644 25312 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:56.507566 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:56.508194 25240 raft_consensus.cc:3055] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:07:56.508548 25240 raft_consensus.cc:740] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:56.507831 25140 catalog_manager.cc:5697] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 2, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:56.509457 25240 consensus_queue.cc:260] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:56.510669 25240 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:56.511461 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:56.511971 25390 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:07:56.518147 25390 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:07:56.522013 25287 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:56.522279 25240 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:07:56.522856 26650 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:07:56.524610 26650 raft_consensus.cc:697] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:56.525475 26650 consensus_queue.cc:237] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:56.542832 25140 catalog_manager.cc:5697] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 3 to 4, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 4 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
W20260709 13:07:56.617015 25966 debug-util.cc:398] Leaking SignalData structure 0x7b08000ffae0 after lost signal to thread 24116
W20260709 13:07:56.618165 25966 debug-util.cc:398] Leaking SignalData structure 0x7b08002dda20 after lost signal to thread 25178
W20260709 13:07:56.619411 25966 debug-util.cc:398] Leaking SignalData structure 0x7b080036ba20 after lost signal to thread 25270
W20260709 13:07:56.620513 25966 debug-util.cc:398] Leaking SignalData structure 0x7b080036c260 after lost signal to thread 25345
W20260709 13:07:56.621582 25966 debug-util.cc:398] Leaking SignalData structure 0x7b08003658c0 after lost signal to thread 25419
W20260709 13:07:56.622750 25966 debug-util.cc:398] Leaking SignalData structure 0x7b080036b620 after lost signal to thread 25969
I20260709 13:07:56.661369 26628 raft_consensus.cc:993] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a: : Instructing follower f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:56.661890 26628 raft_consensus.cc:1081] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 2 LEADER]: Signalling peer f646dfd532974dceadfe00005d2bbf59 to start an election
I20260709 13:07:56.663539 25942 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:59304
I20260709 13:07:56.664121 25942 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:56.664475 25942 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:56.675071 25942 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:56.677924 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:56.678457 25390 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:56.678457 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:56.679498 25239 raft_consensus.cc:3055] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:56.679801 25239 raft_consensus.cc:740] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:56.680553 25239 consensus_queue.cc:260] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:56.684298 25239 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:56.684346 25942 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:56.685391 25390 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:07:56.686370 25911 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:07:56.687183 26660 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:56.687605 26660 raft_consensus.cc:697] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:07:56.688412 26660 consensus_queue.cc:237] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:56.699175 25239 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:07:56.709329 25140 catalog_manager.cc:5697] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 2 to 3, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 3 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:07:56.751358 25390 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:07:56.752583 26623 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:56.783105 25312 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:07:56.785624 26628 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:56.834393 25318 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:56.837173 26661 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:56.888592 25239 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:56.894516 26599 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:07:56.905579 25942 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:56.907238 26609 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:56.973896 25390 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:56.976573 26650 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:57.027693 25387 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:07:57.052564 26634 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:57.075802 25243 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:07:57.078573 26650 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:57.109225 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:57.191720 25390 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:07:57.193464 26660 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:07:57.229080 25239 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:07:57.245018 26661 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:07:57.265605 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:57.388702 25240 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
mode: GRACEFUL
new_leader_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:37596
I20260709 13:07:57.389494 25240 raft_consensus.cc:606] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Received request to transfer leadership to TS 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:57.391857 25394 consensus_queue.cc:237] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:57.405941 25312 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:07:57.406922 25312 raft_consensus.cc:606] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:07:57.408295 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:07:57.410079 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
W20260709 13:07:57.416280 25358 consensus_peers.cc:597] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37bfef8566b143cf94971f26f417678c. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:57.416710 25240 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:57.419445 26472 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:57.439944 25312 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:57.441649 26472 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:57.452064 26650 raft_consensus.cc:993] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: : Instructing follower a2a77ecf2c0941618b5bc79d4c04db61 to start an election
I20260709 13:07:57.452633 26678 raft_consensus.cc:1081] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Signalling peer a2a77ecf2c0941618b5bc79d4c04db61 to start an election
I20260709 13:07:57.456302 25387 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:60946
I20260709 13:07:57.456864 25387 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:57.457247 25387 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:57.468101 25387 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:57.470858 25387 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:07:57.473434 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:57.473977 25318 raft_consensus.cc:3055] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:57.474293 25318 raft_consensus.cc:740] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:57.474190 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:07:57.475028 25942 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:57.475198 25318 consensus_queue.cc:260] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:57.483876 25318 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:57.499575 25942 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:07:57.500447 25945 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 1. Current term is 2. Ops: []
I20260709 13:07:57.505162 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:07:57.505930 26472 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:57.506405 26472 raft_consensus.cc:697] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:57.507169 26472 consensus_queue.cc:237] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:57.511824 25318 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
W20260709 13:07:57.529109 26685 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:07:57.532626 26622 raft_consensus.cc:2955] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:57.535526 25243 raft_consensus.cc:2955] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:57.554029 25140 catalog_manager.cc:5697] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 1 to 2, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 2 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:57.571837 25387 consensus_queue.cc:237] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:57.552255 25312 raft_consensus.cc:2955] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:57.593736 25315 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:57.595346 26622 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:57.599453 25240 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:57.601054 26472 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:07:57.606547 25358 consensus_peers.cc:597] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17584a19690d4010851d65378cfcbafa. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:57.595387 25143 catalog_manager.cc:5697] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:57.645581 26472 raft_consensus.cc:2955] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:57.693306 25240 raft_consensus.cc:2955] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:57.705708 25315 consensus_queue.cc:237] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:57.711817 25312 raft_consensus.cc:2955] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:57.717054 25140 catalog_manager.cc:5697] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:57.734035 25390 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:07:57.734822 25239 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:07:57.735731 25284 consensus_peers.cc:597] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f75b60146e7d4b9b9954b4fb2c414679. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:57.737053 26650 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:57.737951 26679 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:57.770963 26679 raft_consensus.cc:2955] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:57.773056 25390 raft_consensus.cc:2955] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:57.786062 25240 raft_consensus.cc:2955] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:57.789278 25239 consensus_queue.cc:237] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:57.791061 25141 catalog_manager.cc:5697] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index -1 to 2, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:57.812777 25387 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:57.819033 25312 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 13:07:57.819209 25209 consensus_peers.cc:597] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eda233ace2a24b379fef320a5f610c7a. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:57.820564 26623 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:57.821841 26628 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:57.846146 26702 ts_tablet_manager.cc:933] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:57.851233 25387 raft_consensus.cc:2955] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:57.848862 26671 raft_consensus.cc:2955] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:57.896584 25139 catalog_manager.cc:5697] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index -1 to 3, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New cstate: current_term: 3 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:57.912307 26702 tablet_copy_client.cc:323] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.67:34661
I20260709 13:07:57.915259 26662 raft_consensus.cc:993] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: : Instructing follower 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:57.915388 25405 tablet_copy_service.cc:140] P a2a77ecf2c0941618b5bc79d4c04db61: Received BeginTabletCopySession request for tablet 37bfef8566b143cf94971f26f417678c from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:41830)
I20260709 13:07:57.915834 26623 raft_consensus.cc:1081] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Signalling peer 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:57.916003 25405 tablet_copy_service.cc:161] P a2a77ecf2c0941618b5bc79d4c04db61: Beginning new tablet copy session on tablet 37bfef8566b143cf94971f26f417678c from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:41830: session id = f646dfd532974dceadfe00005d2bbf59-37bfef8566b143cf94971f26f417678c
I20260709 13:07:57.921758 25315 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:36864
I20260709 13:07:57.922326 25315 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:57.922676 25315 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:57.929734 25405 tablet_copy_source_session.cc:215] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:57.934625 26702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37bfef8566b143cf94971f26f417678c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:57.948956 25315 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:57.951936 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:57.952101 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:57.954104 25387 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:57.954721 25312 raft_consensus.cc:2955] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:07:57.957650 25240 raft_consensus.cc:3055] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:57.958026 25240 raft_consensus.cc:740] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:57.958797 25240 consensus_queue.cc:260] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:57.959820 25240 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:57.960122 25315 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:57.968711 25387 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:57.970605 25287 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:07:57.971345 26679 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:57.971854 26679 raft_consensus.cc:697] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:57.972748 26679 consensus_queue.cc:237] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:57.974403 25240 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:57.981765 26702 tablet_copy_client.cc:806] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:57.982407 26702 tablet_copy_client.cc:670] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:57.986785 26702 tablet_copy_client.cc:538] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:57.996629 26702 tablet_bootstrap.cc:492] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:58.012974 25140 catalog_manager.cc:5697] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 2, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:58.058483 26702 tablet_bootstrap.cc:492] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:58.059427 26702 tablet_bootstrap.cc:492] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:58.062127 26702 ts_tablet_manager.cc:1403] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.066s	user 0.010s	sys 0.032s
I20260709 13:07:58.065570 26702 raft_consensus.cc:359] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:58.066361 26702 raft_consensus.cc:740] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:58.067003 26702 consensus_queue.cc:260] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:58.070453 26702 ts_tablet_manager.cc:1434] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 13:07:58.072052 25405 tablet_copy_service.cc:342] P a2a77ecf2c0941618b5bc79d4c04db61: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-37bfef8566b143cf94971f26f417678c received from {username='slave'} at 127.0.0.1:41830
I20260709 13:07:58.072530 25405 tablet_copy_service.cc:434] P a2a77ecf2c0941618b5bc79d4c04db61: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-37bfef8566b143cf94971f26f417678c on tablet 37bfef8566b143cf94971f26f417678c with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:58.083484 25942 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:58.086455 26622 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:58.103185 25312 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:07:58.106709 26694 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:07:58.167788 26702 ts_tablet_manager.cc:933] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:58.188565 26702 tablet_copy_client.cc:323] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.67:34661
I20260709 13:07:58.191224 25405 tablet_copy_service.cc:140] P a2a77ecf2c0941618b5bc79d4c04db61: Received BeginTabletCopySession request for tablet 17584a19690d4010851d65378cfcbafa from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:41830)
I20260709 13:07:58.191728 25405 tablet_copy_service.cc:161] P a2a77ecf2c0941618b5bc79d4c04db61: Beginning new tablet copy session on tablet 17584a19690d4010851d65378cfcbafa from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:41830: session id = f646dfd532974dceadfe00005d2bbf59-17584a19690d4010851d65378cfcbafa
I20260709 13:07:58.210718 25405 tablet_copy_source_session.cc:215] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:58.219626 26702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17584a19690d4010851d65378cfcbafa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:58.241530 26713 ts_tablet_manager.cc:933] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:07:58.248664 26713 tablet_copy_client.cc:323] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.66:42315
I20260709 13:07:58.251355 25331 tablet_copy_service.cc:140] P 2329e428ebbf484fa3ea9452133df567: Received BeginTabletCopySession request for tablet f75b60146e7d4b9b9954b4fb2c414679 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:40928)
I20260709 13:07:58.251871 25331 tablet_copy_service.cc:161] P 2329e428ebbf484fa3ea9452133df567: Beginning new tablet copy session on tablet f75b60146e7d4b9b9954b4fb2c414679 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:40928: session id = f646dfd532974dceadfe00005d2bbf59-f75b60146e7d4b9b9954b4fb2c414679
I20260709 13:07:58.266963 25331 tablet_copy_source_session.cc:215] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:58.281386 26713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f75b60146e7d4b9b9954b4fb2c414679. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:58.305868 26702 tablet_copy_client.cc:806] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:58.306931 26702 tablet_copy_client.cc:670] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:58.309624 26715 ts_tablet_manager.cc:933] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:07:58.311436 26702 tablet_copy_client.cc:538] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:58.331008 26702 tablet_bootstrap.cc:492] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:58.346693 26715 tablet_copy_client.cc:323] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.65:45313
I20260709 13:07:58.354033 25255 tablet_copy_service.cc:140] P 46de6919f85341c6a9b574dff112e35a: Received BeginTabletCopySession request for tablet eda233ace2a24b379fef320a5f610c7a from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:44954)
I20260709 13:07:58.354563 25255 tablet_copy_service.cc:161] P 46de6919f85341c6a9b574dff112e35a: Beginning new tablet copy session on tablet eda233ace2a24b379fef320a5f610c7a from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:44954: session id = f646dfd532974dceadfe00005d2bbf59-eda233ace2a24b379fef320a5f610c7a
I20260709 13:07:58.362635 25390 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:58.363551 25255 tablet_copy_source_session.cc:215] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:07:58.364970 26679 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:58.367683 26715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eda233ace2a24b379fef320a5f610c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:58.367381 26713 tablet_copy_client.cc:806] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:58.368892 26713 tablet_copy_client.cc:670] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:58.378139 26713 tablet_copy_client.cc:538] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:07:58.399151 25240 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:58.401322 26678 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:58.411403 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:58.428974 26713 tablet_bootstrap.cc:492] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:58.440181 25945 raft_consensus.cc:1217] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:58.482872 26715 tablet_copy_client.cc:806] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:07:58.485713 26715 tablet_copy_client.cc:670] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:07:58.608402 26715 tablet_copy_client.cc:538] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260709 13:07:58.612073 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 121733us
I20260709 13:07:58.678484 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:58.692022 26715 tablet_bootstrap.cc:492] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:07:58.714054 26702 tablet_bootstrap.cc:492] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:58.715070 26702 tablet_bootstrap.cc:492] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:58.715791 26702 ts_tablet_manager.cc:1403] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.385s	user 0.023s	sys 0.009s
I20260709 13:07:58.740453 26713 tablet_bootstrap.cc:492] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:58.742300 26713 tablet_bootstrap.cc:492] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:58.770838 26713 ts_tablet_manager.cc:1403] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.343s	user 0.030s	sys 0.004s
I20260709 13:07:58.839149 26713 raft_consensus.cc:359] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:58.840251 26713 raft_consensus.cc:740] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:58.851044 26702 raft_consensus.cc:359] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
W20260709 13:07:58.857364 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 66595.9us
I20260709 13:07:58.859370 26702 raft_consensus.cc:740] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:58.860143 26702 consensus_queue.cc:260] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:58.867637 26713 consensus_queue.cc:260] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
W20260709 13:07:58.898794 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 98967.5us
W20260709 13:07:58.900417 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 105121us
W20260709 13:07:58.933059 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 62020.4us
I20260709 13:07:58.953892 26713 ts_tablet_manager.cc:1434] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.170s	user 0.004s	sys 0.004s
W20260709 13:07:58.961803 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 90339.9us
I20260709 13:07:58.979439 26702 ts_tablet_manager.cc:1434] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.263s	user 0.008s	sys 0.004s
W20260709 13:07:59.012458 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 53059.5us
I20260709 13:07:59.021039 25394 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "37bfef8566b143cf94971f26f417678c"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
mode: GRACEFUL
new_leader_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:52318
I20260709 13:07:59.021667 25394 raft_consensus.cc:606] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Received request to transfer leadership to TS 2329e428ebbf484fa3ea9452133df567
I20260709 13:07:59.022384 25405 tablet_copy_service.cc:342] P a2a77ecf2c0941618b5bc79d4c04db61: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-17584a19690d4010851d65378cfcbafa received from {username='slave'} at 127.0.0.1:41830
I20260709 13:07:59.022814 25405 tablet_copy_service.cc:434] P a2a77ecf2c0941618b5bc79d4c04db61: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-17584a19690d4010851d65378cfcbafa on tablet 17584a19690d4010851d65378cfcbafa with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:59.038578 25331 tablet_copy_service.cc:342] P 2329e428ebbf484fa3ea9452133df567: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-f75b60146e7d4b9b9954b4fb2c414679 received from {username='slave'} at 127.0.0.1:40928
I20260709 13:07:59.039076 25331 tablet_copy_service.cc:434] P 2329e428ebbf484fa3ea9452133df567: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-f75b60146e7d4b9b9954b4fb2c414679 on tablet f75b60146e7d4b9b9954b4fb2c414679 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:59.040493 25312 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:07:59.041038 25312 raft_consensus.cc:606] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:07:59.048558 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:07:59.050139 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
W20260709 13:07:59.062067 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 120905us
I20260709 13:07:59.065784 26727 raft_consensus.cc:993] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: : Instructing follower 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:59.066207 26727 raft_consensus.cc:1081] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Signalling peer 2329e428ebbf484fa3ea9452133df567 to start an election
I20260709 13:07:59.085486 26747 raft_consensus.cc:1064] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
W20260709 13:07:59.085901 26747 raft_consensus.cc:1068] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: Unable to promote non-voter f646dfd532974dceadfe00005d2bbf59: Service unavailable: leader transfer in progress
I20260709 13:07:59.099586 25315 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "37bfef8566b143cf94971f26f417678c"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:36876
I20260709 13:07:59.100123 25315 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:59.100508 25315 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:59.107281 25315 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:59.110046 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:07:59.110570 25244 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:59.117281 25315 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:59.125270 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:59.125867 25394 raft_consensus.cc:3055] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:59.126101 26715 tablet_bootstrap.cc:492] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:07:59.126978 26715 tablet_bootstrap.cc:492] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:07:59.127651 26715 ts_tablet_manager.cc:1403] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 0.436s	user 0.032s	sys 0.016s
I20260709 13:07:59.126143 25394 raft_consensus.cc:740] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:07:59.133718 25394 consensus_queue.cc:260] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:59.135049 25394 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:59.140650 25318 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:07:59.142143 26715 raft_consensus.cc:359] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:59.142966 26715 raft_consensus.cc:740] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:07:59.143589 26715 consensus_queue.cc:260] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:59.149281 25244 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:59.151154 25285 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:59.151909 26749 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:59.152387 26749 raft_consensus.cc:697] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:59.153316 26749 consensus_queue.cc:237] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:59.155200 26715 ts_tablet_manager.cc:1434] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.027s	user 0.009s	sys 0.000s
I20260709 13:07:59.172353 25255 tablet_copy_service.cc:342] P 46de6919f85341c6a9b574dff112e35a: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-eda233ace2a24b379fef320a5f610c7a received from {username='slave'} at 127.0.0.1:44954
I20260709 13:07:59.172957 25255 tablet_copy_service.cc:434] P 46de6919f85341c6a9b574dff112e35a: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-eda233ace2a24b379fef320a5f610c7a on tablet eda233ace2a24b379fef320a5f610c7a with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:07:59.176860 25394 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:07:59.196339 25139 catalog_manager.cc:5697] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 2, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:59.228735 25945 raft_consensus.cc:1217] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:59.246536 25318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "37bfef8566b143cf94971f26f417678c"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:36848
I20260709 13:07:59.247172 25318 raft_consensus.cc:606] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Received request to transfer leadership
I20260709 13:07:59.252514 25318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "37bfef8566b143cf94971f26f417678c"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:36848
I20260709 13:07:59.253039 25318 raft_consensus.cc:606] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Received request to transfer leadership
W20260709 13:07:59.254375 25128 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 37bfef8566b143cf94971f26f417678c already in progress [suppressed 1 similar messages]
I20260709 13:07:59.257922 25318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "37bfef8566b143cf94971f26f417678c"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:36848
I20260709 13:07:59.258466 25318 raft_consensus.cc:606] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Received request to transfer leadership
W20260709 13:07:59.265655 25192 auto_rebalancer.cc:893] Removing replace marker failed: Service unavailable: leader transfer in progress
W20260709 13:07:59.266095 25192 auto_rebalancer.cc:901] Could not move replica: Remote error: Service unavailable: leadership transfer for 37bfef8566b143cf94971f26f417678c already in progress
W20260709 13:07:59.266395 25192 auto_rebalancer.cc:330] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 37bfef8566b143cf94971f26f417678c already in progress
I20260709 13:07:59.272856 25945 raft_consensus.cc:1217] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:07:59.353820 25945 raft_consensus.cc:1217] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3]   Dedup: 3.3->[]
I20260709 13:07:59.501609 26745 raft_consensus.cc:993] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567: : Instructing follower 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:07:59.502084 26745 raft_consensus.cc:1081] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Signalling peer 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:07:59.509534 25244 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:32992
I20260709 13:07:59.510105 25244 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:59.510456 25244 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:59.520455 25244 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:59.538136 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:07:59.538713 25312 raft_consensus.cc:3055] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:59.538992 25312 raft_consensus.cc:740] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:07:59.539705 25312 consensus_queue.cc:260] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:59.540738 25312 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:59.547685 25312 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:07:59.549031 25209 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:07:59.550047 26733 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:59.550495 26733 raft_consensus.cc:697] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:07:59.551322 26733 consensus_queue.cc:237] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:07:59.568586 25244 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:07:59.588948 25140 catalog_manager.cc:5697] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 2 to 3, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 3 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:59.593690 25945 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Advancing to term 2
I20260709 13:07:59.611976 25945 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 2 LEARNER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:07:59.613468 26634 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 13:07:59.619973 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 136722us
I20260709 13:07:59.622938 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:07:59.623463 25394 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:59.634305 25394 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:07:59.660161 26634 raft_consensus.cc:1064] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
W20260709 13:07:59.660601 26634 raft_consensus.cc:1068] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: Unable to promote non-voter f646dfd532974dceadfe00005d2bbf59: Service unavailable: leader transfer in progress
I20260709 13:07:59.670614 26750 raft_consensus.cc:1064] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:59.673170 26750 consensus_queue.cc:237] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:59.701682 25318 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:59.703255 26742 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:59.713577 25244 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:07:59.715107 26748 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:59.726811 25243 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:59.727428 25945 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:07:59.729892 26742 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:59.730749 26750 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:59.732524 25394 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:07:59.733964 26748 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 13:07:59.801658 26761 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20260709 13:07:59.825611 26728 raft_consensus.cc:2955] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:07:59.849691 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 55778.9us
I20260709 13:07:59.851516 26769 raft_consensus.cc:993] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: : Instructing follower a2a77ecf2c0941618b5bc79d4c04db61 to start an election
I20260709 13:07:59.856498 26769 raft_consensus.cc:1081] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Signalling peer a2a77ecf2c0941618b5bc79d4c04db61 to start an election
I20260709 13:07:59.857399 25318 raft_consensus.cc:2955] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:59.877270 25393 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "37bfef8566b143cf94971f26f417678c"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:60946
I20260709 13:07:59.877818 25393 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:59.878165 25393 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:59.906404 26745 raft_consensus.cc:1064] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:07:59.948971 25140 catalog_manager.cc:5697] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:59.909080 26745 consensus_queue.cc:237] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:07:59.982172 25393 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:07:59.985733 25387 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
I20260709 13:07:59.990015 25944 raft_consensus.cc:2955] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:59.990844 25239 raft_consensus.cc:2955] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:07:59.996790 25393 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:00.000491 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:00.007745 26737 raft_consensus.cc:1064] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:08:00.010455 26737 consensus_queue.cc:237] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:00.025530 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:00.026093 25318 raft_consensus.cc:3055] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:00.026388 25318 raft_consensus.cc:740] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:00.027202 25318 consensus_queue.cc:260] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:00.033236 25315 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:08:00.037485 26732 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:08:00.042193 25942 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 3 LEARNER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:08:00.044701 26671 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:08:00.045368 26772 consensus_queue.cc:1059] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:08:00.048328 25312 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:08:00.051033 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:08:00.051961 25318 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:00.054559 26671 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:08:00.075002 25942 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 1 LEARNER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:08:00.077804 25387 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:08:00.077009 25243 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:08:00.081794 26775 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:08:00.088020 26745 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:08:00.088440 25318 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
W20260709 13:08:00.109027 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 197277us
I20260709 13:08:00.117622 25387 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:08:00.119874 25391 consensus_queue.cc:237] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:00.123303 25393 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:08:00.173079 25358 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:00.109237 26771 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
W20260709 13:08:00.198062 26761 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
W20260709 13:08:00.241739 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 68786.2us
I20260709 13:08:00.263942 25940 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:08:00.217044 26778 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:00.276723 26778 raft_consensus.cc:697] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:00.305418 26778 consensus_queue.cc:237] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:00.315393 26777 raft_consensus.cc:2955] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:00.331122 25243 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260709 13:08:00.393402 26793 log.cc:927] Time spent T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
W20260709 13:08:00.342499 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 64104.3us
I20260709 13:08:00.398563 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:08:00.412873 26733 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260709 13:08:00.454398 25944 raft_consensus.cc:2955] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:08:00.513948 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 176086us
W20260709 13:08:00.518137 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 180572us
I20260709 13:08:00.858081 26786 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
W20260709 13:08:00.880295 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 359306us
I20260709 13:08:00.926290 26797 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.474s
I20260709 13:08:01.498982 26789 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.589s
W20260709 13:08:01.587255 25359 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.688s	user 0.007s	sys 0.018s
W20260709 13:08:01.589191 25358 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.005s	sys 0.021s
W20260709 13:08:01.589546 25358 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.005s	sys 0.021s
I20260709 13:08:01.397509 26758 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:08:01.590277 25359 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.008s	sys 0.018s
W20260709 13:08:01.591498 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 710008us
I20260709 13:08:01.486140 26743 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:08:01.600528 25910 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.004s	sys 0.019s
I20260709 13:08:01.592239 26803 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:08:01.609407 25910 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.004s	sys 0.020s
W20260709 13:08:01.616852 25360 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.015s	sys 0.007s
W20260709 13:08:01.617199 25909 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.055s	sys 0.066s
W20260709 13:08:03.630872 25909 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.755s	user 0.055s	sys 0.066s
I20260709 13:08:01.620363 26789 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:01.601464 26758 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:03.633210 26723 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:01.604702 26743 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:03.633917 26723 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.639272 26743 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:01.623082 26803 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:01.998239 26816 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:02.309590 25911 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.705s	user 0.000s	sys 0.010s
I20260709 13:08:02.046084 26815 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:02.047255 26818 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:02.039402 26768 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:03.072600 26821 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:03.073397 26814 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:08:03.076892 25910 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.003s	sys 0.006s
I20260709 13:08:03.120950 26822 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:08:03.124576 25908 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.000s	sys 0.010s
I20260709 13:08:03.649598 26818 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:03.622689 26830 raft_consensus.cc:2955] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:08:03.630520 25360 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.569s	user 0.030s	sys 0.021s
I20260709 13:08:03.645866 26833 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:03.642701 26816 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:03.642038 26789 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:03.639294 26723 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:03.644032 25911 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.043s	user 0.000s	sys 0.010s
W20260709 13:08:03.685938 25908 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.187s	user 0.000s	sys 0.011s
W20260709 13:08:03.124964 25210 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.454s	user 0.000s	sys 0.024s
I20260709 13:08:03.687266 26834 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:08:03.687825 25910 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.256s	user 0.003s	sys 0.006s
W20260709 13:08:03.696368 25210 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.075s	user 0.000s	sys 0.024s
I20260709 13:08:03.695513 26758 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.693403 26803 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.697316 26671 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:03.689005 26814 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.650843 26815 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.688215 26821 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.712153 26815 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:03.712532 26821 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.712713 26815 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:03.712656 26828 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:03.709160 26758 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:03.714663 26816 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:03.705251 26829 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:03.715095 26758 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:03.696249 26822 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.685838 26768 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.726835 26814 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.740311 26831 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:03.740880 26816 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:03.741084 26831 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.741643 26816 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:03.744184 26831 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:03.696798 26833 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.705513 26671 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.714825 26815 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.719115 26821 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:03.753978 26818 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.767127 26816 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:03.767957 26816 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:03.768579 26816 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.771222 26816 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:03.771867 26816 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:03.772449 26816 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:03.699968 26834 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.754719 26818 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:03.748502 26836 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:03.747291 26828 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:03.755357 26821 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.791105 26758 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.801721 26789 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:03.791819 26803 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:03.792392 26818 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.799433 26816 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:03.801698 26840 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:03.747833 26829 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:03.807770 26789 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:03.818589 26816 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:03.827209 26671 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:03.817737 26831 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:03.803953 26836 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.830713 26821 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.827544 26768 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.806855 26803 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.861850 26828 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.808131 26844 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:03.850690 26834 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.828089 26822 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:03.850965 26829 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.836771 26816 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.851392 26833 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.851536 26837 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:03.823047 26835 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:03.835876 26832 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:03.863754 26821 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:03.851374 26842 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:03.834076 26789 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.871101 26768 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:03.831743 26840 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:03.873185 26833 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:03.867517 26831 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:03.873338 26815 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:03.886370 26833 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:03.910822 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 92975.3us
I20260709 13:08:03.879868 26768 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.871820 26837 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.875142 26835 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:03.877334 26821 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.919719 26818 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:03.884418 26832 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:03.928432 26803 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:03.928612 26845 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:03.928779 26840 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.872040 26844 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.882308 26842 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.934080 26795 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:03.934544 26850 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:03.936203 26803 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:03.937409 26789 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:03.921727 26848 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:03.941413 26850 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.940632 26831 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.947341 26833 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.941818 26835 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.941440 26803 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.942631 26845 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:03.941793 26848 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:03.949154 26768 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.968863 26844 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.955241 26832 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.968760 26850 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.968628 26837 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:03.954546 26816 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:03.971628 26836 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.972198 26803 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.974378 26848 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:03.945015 26795 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.007762 26818 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:04.008981 26831 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:04.010079 26829 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:04.010780 26832 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:04.011533 26835 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:04.012310 26840 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:04.013842 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:04.021903 26843 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:04.023056 26843 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.024935 26835 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.027484 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:04.029500 26831 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.031097 26841 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:04.032186 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:04.032023 26841 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.034804 26839 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:04.035738 26852 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:04.034868 26818 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.015625 26822 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:04.036303 26839 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.042348 26840 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:04.044330 25387 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 1.
I20260709 13:08:04.045694 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:04.046712 25387 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:04.046136 26822 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:04.027532 26838 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:04.049937 26838 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:04.045706 26829 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:04.058441 26829 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.061479 26849 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:04.036617 26852 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.062666 26822 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.062211 26849 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:04.074476 25284 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 3.155s	user 0.017s	sys 0.044s
W20260709 13:08:04.074574 25285 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 3.187s	user 0.000s	sys 0.052s
W20260709 13:08:04.075218 25284 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.156s	user 0.018s	sys 0.044s
W20260709 13:08:04.075280 25285 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.188s	user 0.000s	sys 0.053s
I20260709 13:08:04.080401 26829 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:04.082871 26829 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.081238 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:04.071573 26832 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.096837 26843 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.119242 26831 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.115139 25944 raft_consensus.cc:2955] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:04.098733 26818 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.096632 26772 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:04.122560 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:04.120976 26840 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.123531 25942 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:04.126076 26838 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.094681 26822 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:04.094604 26830 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:04.130805 26841 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.123087 26772 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:04.133558 26772 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.117918 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:04.119190 26835 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.136745 26830 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.139765 26822 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:04.091324 25237 raft_consensus.cc:2955] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:04.145334 25387 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 1.
I20260709 13:08:04.146917 26856 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:04.147980 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:04.125866 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:04.150400 26854 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:04.150466 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:04.151506 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:04.134680 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:04.154253 26845 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:04.156154 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:04.163028 26855 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.116s	user 0.000s	sys 0.002s
I20260709 13:08:04.163271 25395 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
W20260709 13:08:04.164911 25286 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 3.277s	user 0.008s	sys 0.041s
W20260709 13:08:04.171257 25287 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 3.314s	user 0.024s	sys 0.021s
I20260709 13:08:04.166455 26854 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.172410 26803 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:04.173113 26803 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.174995 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:04.174889 26856 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.177994 25394 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 1.
I20260709 13:08:04.184362 26822 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.182981 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:04.187271 25387 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:08:04.195151 26795 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.197816 26830 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.197710 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:04.199189 26839 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.202736 26832 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.203322 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:04.204921 25911 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:04.206010 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:04.208443 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:04.208618 26854 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:04.211226 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:04.213052 26850 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:04.213619 26850 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:04.214725 26850 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:04.216396 26803 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:04.217308 25286 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.330s	user 0.008s	sys 0.042s
W20260709 13:08:04.225355 25287 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.368s	user 0.025s	sys 0.022s
I20260709 13:08:04.230561 26858 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:04.236964 26748 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:04.229785 25911 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:04.237807 26748 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.242650 26771 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:04.231740 26858 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.250993 26856 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.243620 26771 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:04.271812 26775 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:04.292289 25209 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:04.293588 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 72732.7us
I20260709 13:08:04.302227 26858 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.309509 26832 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:04.310002 26832 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:04.310453 26832 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:08:04.310932 26800 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 3.418s	user 0.000s	sys 0.029s
I20260709 13:08:04.303906 26802 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:04.305827 26748 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.319803 26771 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.314404 26802 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:04.324450 26802 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:04.322258 26775 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:04.328224 26861 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:04.326995 25911 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:04.326479 26856 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:04.332115 26856 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.335253 26856 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.335881 26775 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.338764 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:04.339529 26748 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:04.343811 25942 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:08:04.346766 26748 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.330255 26861 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.350829 26862 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:04.351754 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:04.352638 25240 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 1.
I20260709 13:08:04.350345 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:04.357271 26801 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:04.364899 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:04.365618 25945 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:08:04.360150 26807 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:04.366701 26807 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:04.367282 25358 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.767s	user 0.074s	sys 0.092s
I20260709 13:08:04.368594 26748 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.367170 26807 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:04.371666 26829 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.358714 26850 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.377077 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:04.377549 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:04.378760 25358 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.778s	user 0.075s	sys 0.092s
I20260709 13:08:04.362365 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:04.358919 26801 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.381117 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:04.351732 26862 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:04.382633 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 2.78215e+06us
W20260709 13:08:04.386749 25359 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.788s	user 0.043s	sys 0.095s
W20260709 13:08:04.387223 25359 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.788s	user 0.043s	sys 0.095s
I20260709 13:08:04.387493 26832 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:04.391341 25361 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.913s	user 0.033s	sys 0.042s
I20260709 13:08:04.387959 26861 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.375121 26858 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:04.602701 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:04.436952 26857 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:04.852787 25240 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:04.392995 26856 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:04.649374 26863 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:04.853924 26856 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.854831 26863 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:04.541862 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:04.857257 26856 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.538308 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:04.857957 26857 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:04.859076 26802 consensus_meta.cc:287] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.534s	user 0.007s	sys 0.005s
I20260709 13:08:04.863745 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:04.863008 26861 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:04.864662 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:04.836783 26801 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.865617 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:04.857061 26863 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.836356 26850 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.865908 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:04.834275 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:04.391566 26832 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.863274 26802 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.877349 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:04.878584 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:04.878719 26802 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.879611 26801 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:04.880694 26801 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.883558 26801 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:04.874290 25361 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.401s	user 0.052s	sys 0.056s
I20260709 13:08:04.866006 26821 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.875301 26861 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:04.876111 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:04.876304 26849 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.886214 26861 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.838227 26862 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.540679 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:04.879786 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:04.889055 25244 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:04.711664 26860 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:04.866247 26807 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:04.923640 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 308975us
I20260709 13:08:04.904975 26858 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.932401 26862 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:04.949990 26857 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.951711 26807 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.952760 26862 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:04.955307 26862 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.943756 26868 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:04.962330 26868 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:04.940724 26860 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.966032 26860 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.966825 26858 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.956794 25284 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:04.970556 25284 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:04.971292 26863 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:04.971853 26863 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.972926 26801 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:04.973553 26801 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:04.973693 26866 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:04.974004 26863 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:04.974637 26863 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:04.975173 26863 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:04.974589 26866 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:04.975528 26863 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:04.975584 26801 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.977450 26801 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:04.977830 26801 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:04.978191 26801 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:04.980013 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:04.980813 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:04.981256 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:04.981855 25317 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 1.
I20260709 13:08:04.982024 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:04.982631 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:04.984282 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:04.984855 25316 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:04.964982 26868 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:04.990967 26866 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:04.993095 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 44619.4us
I20260709 13:08:05.000473 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:04.993671 26867 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:05.002043 26867 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:05.005846 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.001365 25316 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 1.
I20260709 13:08:05.006969 25318 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:05.010324 26875 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:05.011152 26875 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:05.013278 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.013584 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.013866 25316 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 1.
I20260709 13:08:05.014266 26867 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:05.014750 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.015331 25321 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 1.
I20260709 13:08:05.016242 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.018054 26868 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:05.018143 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.018289 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.018944 26842 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:05.018699 26868 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:05.019784 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.024379 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.024613 26868 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:05.014765 26867 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:05.026630 26867 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:05.030177 26863 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:05.030864 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.032245 25313 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 2: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:05.032729 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.033586 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.033831 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.034725 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.035392 25319 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 3: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:05.035842 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.034970 25312 raft_consensus.cc:2955] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:05.036052 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.036501 25314 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 1.
I20260709 13:08:05.036576 25319 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 1.
I20260709 13:08:05.037853 26875 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:05.038132 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.043987 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.044955 26804 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:05.046262 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.046816 25314 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 1.
I20260709 13:08:05.045576 26804 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:05.050169 26874 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:05.052465 26874 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:05.042614 26871 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:05.054623 26871 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:05.056512 26868 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:05.057618 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.058029 26862 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:05.058694 26862 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:05.059006 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.060771 26862 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:05.057060 26868 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:05.061705 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.061872 26874 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:05.047509 26867 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:05.062465 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.095974 26871 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:05.115011 25314 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:05.095175 26863 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:05.116014 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 736412us
I20260709 13:08:05.075819 25318 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:08:05.117254 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.119947 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.092762 26872 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:05.121815 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.121693 26872 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:05.122248 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.122916 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.123193 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.124238 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.124358 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.124693 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.124912 25386 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:05.124363 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.125583 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.125761 25395 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:05.126080 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.126592 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.126744 25386 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:08:05.127219 25391 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:05.118294 26860 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:05.128361 25394 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:08:05.124963 25390 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:08:05.128664 25911 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:05.049054 26875 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:05.129758 25911 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:05.129904 26875 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:05.130503 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.131717 26842 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:05.128362 26860 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:05.132160 26842 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:05.132375 26802 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:05.132630 26842 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:05.132793 26875 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:05.132776 26802 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:05.136600 25359 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:05.134189 26802 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:05.141603 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.097946 26804 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:05.116163 26868 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:05.101716 26876 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:05.148793 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.156896 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.157718 25390 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 1.
I20260709 13:08:05.158216 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:05.158406 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.158679 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.160611 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.161262 25245 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:05.166538 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:05.167177 25391 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:05.147814 25139 catalog_manager.cc:5697] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:05.169555 25911 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:05.169661 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.120713 26801 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:05.048245 26873 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:05.147003 26871 raft_consensus.cc:2955] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:05.180393 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.155983 26875 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:05.140599 26877 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:05.174458 26807 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:08:05.186093 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 48070.6us
I20260709 13:08:05.187846 25320 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 2: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:05.189319 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.198228 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.149325 26876 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:05.200048 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.201081 25391 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:08:05.203333 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.188714 26877 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:05.203389 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.207000 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.207825 26872 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:05.208170 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.188220 26807 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:05.209781 26807 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:05.210701 26801 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:05.207194 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.184015 26875 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:05.195658 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.214529 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.215746 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.186167 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:05.219046 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.220389 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.221614 26876 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:05.223788 26802 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:05.206080 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.225970 25285 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:05.226779 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.226984 26801 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:05.227352 25285 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:05.227947 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.228785 26868 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:05.229010 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.231407 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.233790 25911 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:05.235324 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.236119 25321 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:05.237224 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.239260 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.192413 26873 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:05.233435 26868 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:05.243294 26868 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:05.243506 26802 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:05.217571 26875 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:05.237955 25316 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:05.246421 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.248064 25312 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:08:05.249567 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.250743 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:05.253049 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:05.674387 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 442628us
I20260709 13:08:05.674782 26807 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:05.479822 26827 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:05.675704 25359 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:05.675967 26827 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:05.676802 26794 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:05.677297 26794 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:05.677757 26794 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:05.382761 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.672927 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.230849 25386 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:05.678485 26827 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:05.713636 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:05.621840 25908 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.512s (SENT)
W20260709 13:08:05.738559 26871 consensus_meta.cc:287] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.529s	user 0.010s	sys 0.000s
I20260709 13:08:05.477454 26876 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:05.742697 26876 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:05.744812 26876 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:05.636200 26743 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:05.657512 26877 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:05.643821 26842 consensus_meta.cc:287] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.511s	user 0.008s	sys 0.001s
I20260709 13:08:05.526079 26873 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:05.754175 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:05.760490 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.775194 25394 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:08:05.755245 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 51136.3us
I20260709 13:08:05.253906 26802 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:08:05.679549 26807 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:05.780543 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 158832us
I20260709 13:08:05.784373 26802 raft_consensus.cc:697] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:05.785483 26873 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:05.786080 26873 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:05.787925 26873 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:05.789227 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.789528 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.790417 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.790978 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.791592 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.792358 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.792763 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:05.793831 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.788486 26802 consensus_queue.cc:237] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:05.794117 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.776593 26743 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:05.800691 26743 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:05.796896 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.803861 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:05.781296 26875 consensus_meta.cc:287] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.537s	user 0.007s	sys 0.003s
I20260709 13:08:05.806833 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:05.810508 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:05.814337 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.814917 25240 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:05.816088 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.816674 25316 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:05.775923 26842 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:05.831525 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.832439 25386 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:08:05.833210 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.833269 26871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:60924 (request call id 81) took 5709 ms. Trace:
I20260709 13:08:04.733990 26771 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:05.836759 25240 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:08:05.837616 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.844187 25394 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
W20260709 13:08:05.839090 26868 consensus_meta.cc:287] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.595s	user 0.007s	sys 0.003s
I20260709 13:08:05.809206 26875 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:05.837723 25911 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:05.841470 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 149341us
I20260709 13:08:05.849255 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:05.848963 25287 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:05.850579 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.845526 26868 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:05.838765 26771 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:05.852368 25318 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 3: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:05.853760 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.854640 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.848605 25911 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:05.857179 25390 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:05.858994 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:05.862437 26848 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:08:05.863137 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.863427 26875 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:05.863651 26874 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:05.864059 26874 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:05.864403 26874 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:05.864786 26771 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:05.835443 25416 debug-util.cc:398] Leaking SignalData structure 0x7b080014f380 after lost signal to thread 24116
I20260709 13:08:05.873562 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.853353 26868 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:05.876755 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.870841 26807 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:08:05.877735 26807 raft_consensus.cc:697] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:08:05.878211 25416 debug-util.cc:398] Leaking SignalData structure 0x7b08000dbca0 after lost signal to thread 25178
I20260709 13:08:05.878899 26807 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:05.879755 25416 debug-util.cc:398] Leaking SignalData structure 0x7b08003401c0 after lost signal to thread 25270
I20260709 13:08:05.885764 25240 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:05.885915 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.870728 26848 raft_consensus.cc:697] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:08:05.881090 25416 debug-util.cc:398] Leaking SignalData structure 0x7b080034e960 after lost signal to thread 25345
I20260709 13:08:05.891742 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.892402 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.888722 26848 consensus_queue.cc:237] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:05.893716 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.894587 25390 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:05.893026 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.896333 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.897682 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.898761 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.899981 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:05.899986 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 82574.6us
I20260709 13:08:05.900413 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.865888 26891 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:05.902168 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.902748 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.904193 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.905496 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.835038 26871 rpcz_store.cc:276] 0709 13:08:00.115794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:00.116228 (+   434us) service_pool.cc:224] Handling call
0709 13:08:00.123324 (+  7096us) log.cc:844] Serialized 501 byte log entry
0709 13:08:00.135279 (+ 11955us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b2c000e1828. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f29755aba11 00007f29755baa0a 00007f29755ba693 00007f29755ba631 00007f2976ceeb52 00007f2976ceecf5 00007f2976c2b691 00007f2976c0beaa 00007f2976c12d53 00007f2976c12cd9 00007f2976c12c69
0709 13:08:00.321556 (+186277us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b1800206ca0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976ceed01 00007f2976c2b691 00007f2976c0beaa 00007f2976c12d53 00007f2976c12cd9 00007f2976c12c69 00007f2976c12c31 00007f2976c11dfd 000000000038f354 000000000038f199
0709 13:08:05.819701 (+5498145us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3621,"raft.run_cpu_time_us":4228,"raft.run_wall_time_us":250100,"spinlock_wait_cycles":38349440,"thread_start_us":79054,"threads_started":1}
I20260709 13:08:05.907187 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.907210 25391 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:05.908739 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.909399 25240 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:05.914424 25131 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 17584a19690d4010851d65378cfcbafa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:08:05.914872 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.915114 25209 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:05.916007 26827 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:05.916236 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.916414 26827 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:05.916812 26827 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:08:05.902801 25416 debug-util.cc:398] Leaking SignalData structure 0x7b08001b55a0 after lost signal to thread 25393
I20260709 13:08:05.916966 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:05.918821 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.919358 25312 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 3: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:05.902108 26891 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:05.920660 26891 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:05.920578 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.921818 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.922133 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:05.927611 25242 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:05.926761 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.926828 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:05.930043 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.934051 25359 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:05.935230 25359 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:05.909272 26794 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:05.936448 25359 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:05.940925 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.942555 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:05.944813 25359 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:05.937047 25908 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:05.947304 26786 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:05.947665 26786 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:05.947991 26786 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:05.948958 26794 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:05.949784 25395 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:05.950001 25242 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:05.951156 25285 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:05.942330 26881 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:05.956909 26881 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:05.957082 26886 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:05.957382 26881 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:05.957594 26886 raft_consensus.cc:697] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:05.956868 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.944449 26871 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:05.951504 26878 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:05.978458 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:05.979713 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 120777us
I20260709 13:08:05.980463 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:05.980468 25908 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:05.983528 25287 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:05.990806 26890 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:05.992116 26890 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:05.992518 26890 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:04.756264 26772 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:05.990655 25287 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:05.967998 26871 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:05.977304 26878 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:05.995816 26878 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:05.995434 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:05.990945 26771 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:05.996294 26892 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:06.002946 26892 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:05.997498 26866 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:05.996773 26871 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:05.999246 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:05.998765 26772 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:06.002123 26868 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:08:06.017881 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 89881us
I20260709 13:08:06.012887 26772 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:06.002102 26771 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:06.019799 25287 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:06.022189 25908 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:06.019443 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:06.005810 26892 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:06.023669 26874 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:06.005856 26866 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:06.023264 25287 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:06.028273 25287 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:06.028294 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:06.029455 25908 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:06.022846 26894 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 13:08:06.021363 26802 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:06.027828 26866 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:06.021242 26771 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:06.038350 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:06.044309 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:06.045702 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:06.050807 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:06.053725 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:06.029948 26772 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:06.051436 26884 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:06.042207 26802 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:06.027308 26807 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:06.053321 25358 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:06.057685 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:06.063478 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:06.063992 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:06.064632 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:06.042207 26868 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:06.022202 26886 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:06.061770 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:06.066977 25908 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:06.057502 26807 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:06.055379 26884 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:06.070199 26807 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:06.070436 26884 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:06.070684 26887 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:06.031078 26827 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:06.078374 26899 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:04.806773 26775 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:06.057237 25388 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:06.079895 25312 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:06.057098 26802 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:06.084805 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:06.088400 25908 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:06.089308 26794 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:06.089664 26775 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:06.090660 26786 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:06.091666 26881 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:06.092811 26786 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:06.093411 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:06.094244 25319 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 1.
I20260709 13:08:06.095623 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:06.098906 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 45713.4us
I20260709 13:08:06.104498 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:06.106980 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:06.107764 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:06.097097 25240 raft_consensus.cc:2955] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:06.103541 25242 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:06.113086 25358 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:06.113706 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:06.116776 26846 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:06.117265 26846 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:06.117321 25358 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:06.117517 26881 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:06.117736 26846 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:06.118002 26794 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:06.118382 26794 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:06.120465 26794 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:06.111233 26786 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:06.126816 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:06.133073 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:06.136471 25314 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:06.125224 26874 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:06.128386 26881 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:08:06.143689 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 124315us
I20260709 13:08:06.141074 25358 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:06.138177 26778 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:08:06.148424 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 104705us
I20260709 13:08:06.151950 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:06.156339 25358 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:06.156435 26903 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:06.159552 26883 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:06.145742 26881 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:06.147470 26848 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:06.162518 25209 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:06.162309 26877 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:06.147353 26778 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:06.148917 26872 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:06.164260 26881 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:06.165020 26848 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:06.170445 26848 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:06.164695 25358 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:06.160853 26827 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:06.176767 26877 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:06.171833 26778 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:06.200042 26899 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:06.201683 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:06.200493 26899 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:06.203787 26899 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:06.210525 25209 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:06.214917 26814 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:06.215458 26814 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:06.219684 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:06.217999 26743 consensus_queue.cc:1059] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:08:06.204707 26877 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:06.225302 26827 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:06.233886 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:06.222043 26903 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:06.204952 26872 consensus_queue.cc:1059] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260709 13:08:06.233212 26860 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:06.285331 25386 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:06.253219 26903 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:08:06.293082 25393 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 5.858s	user 0.062s	sys 0.043s
I20260709 13:08:06.295264 26878 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:06.299924 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1564) took 1270 ms. Trace:
I20260709 13:08:06.300297 26878 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:06.282428 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:06.283344 26814 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:06.300419 25317 rpcz_store.cc:276] 0709 13:08:05.029972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:05.036368 (+  6396us) service_pool.cc:224] Handling call
0709 13:08:05.036723 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:06.013388 (+976665us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:06.013425 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:06.013492 (+    67us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:06.013532 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:06.298222 (+284690us) spinlock_profiling.cc:243] Waited 976 ms on lock 0x7b54000b22d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:06.299605 (+  1383us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:06.299797 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2245483904}
I20260709 13:08:06.257735 26802 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:06.291553 26903 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:06.326763 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:06.327642 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 337736us
I20260709 13:08:06.329875 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:06.335824 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 213032us
W20260709 13:08:06.383939 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 55299.9us
I20260709 13:08:06.397284 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:06.387702 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 299347us
I20260709 13:08:06.399976 25908 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:06.387342 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:06.405284 25317 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:08:06.407156 26827 consensus_queue.cc:1059] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:06.408883 26827 raft_consensus.cc:3055] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:06.335166 26890 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:06.412205 26827 raft_consensus.cc:740] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:06.409219 25140 catalog_manager.cc:5697] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 2 to 3, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:06.413519 26802 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:06.413204 26827 consensus_queue.cc:260] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:06.414005 26890 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:06.414482 26827 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:06.401880 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:06.421964 25393 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 6.024s	user 0.063s	sys 0.043s
I20260709 13:08:06.422415 25312 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:06.426843 25908 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:06.403266 26807 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:06.408810 26892 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:06.438467 26879 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:06.411901 26871 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:06.402037 26891 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:06.442718 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 971) took 6103 ms. Trace:
I20260709 13:08:06.431867 26802 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:06.439767 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:06.461072 26909 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:06.460886 26758 consensus_queue.cc:1059] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260709 13:08:06.468747 26778 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:06.469090 26906 consensus_queue.cc:1059] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260709 13:08:06.471431 26778 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:06.447636 26794 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:06.448421 25393 rpcz_store.cc:276] 0709 13:08:00.335802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:00.336187 (+   385us) service_pool.cc:224] Handling call
0709 13:08:00.336670 (+   483us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:00.336981 (+   311us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:00.337003 (+    22us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:00.343160 (+  6157us) log.cc:844] Serialized 564 byte log entry
0709 13:08:04.295712 (+3952552us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:06.429587 (+2133875us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:06.429638 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:06.433401 (+  3763us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:06.433439 (+    38us) raft_consensus.cc:1685] finished
0709 13:08:06.433452 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:06.439551 (+  6099us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":552111488,"thread_start_us":6024696,"threads_started":1,"wal-append.queue_time_us":481036,"wal-append.run_cpu_time_us":34446,"wal-append.run_wall_time_us":4451247}
I20260709 13:08:06.475522 25908 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:06.453203 26802 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:06.484071 26917 raft_consensus.cc:3055] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:06.484474 26917 raft_consensus.cc:740] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:06.485427 26917 consensus_queue.cc:260] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:06.450439 26866 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:06.492439 25908 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:06.499789 25908 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:06.480715 26802 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:06.488806 26887 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:06.473220 26879 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:06.492919 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:06.505336 26887 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:06.517747 26908 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:06.517534 26797 consensus_queue.cc:1059] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20260709 13:08:06.518618 26797 raft_consensus.cc:3055] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:06.518947 26797 raft_consensus.cc:740] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:06.507543 26821 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:06.474524 26884 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:06.519790 26906 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:08:06.531755 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 293193us
I20260709 13:08:06.544625 26892 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:06.548000 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:06.551604 26871 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:06.519431 26771 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:06.555002 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 979) took 1433 ms. Trace:
I20260709 13:08:06.556898 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:06.557696 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 57991.3us
I20260709 13:08:06.555568 25392 rpcz_store.cc:276] 0709 13:08:05.115761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:05.118982 (+  3221us) service_pool.cc:224] Handling call
0709 13:08:05.120133 (+  1151us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:05.131386 (+ 11253us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:05.131415 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:05.131450 (+    35us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:05.131477 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:06.549190 (+1417713us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b54001e3f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:06.549261 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:06.549427 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25428736}
I20260709 13:08:06.564028 25908 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:06.532459 26906 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:06.570492 25944 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 2 LEARNER]: Advancing to term 3
I20260709 13:08:06.553254 26807 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:06.524065 26846 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:06.571830 25313 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:06.554791 26794 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:06.584811 25944 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 3 LEARNER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20260709 13:08:06.599639 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 182520us
I20260709 13:08:06.579836 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1574) took 1514 ms. Trace:
I20260709 13:08:06.604774 25395 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:08:06.604986 25942 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:06.567600 26906 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:06.606441 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.606375 25212 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:06.566648 26892 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:08:06.612955 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:06.627969 26794 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:06.628825 26794 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:06.632042 26794 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:06.572304 26848 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:06.602015 26891 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:06.645735 25317 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
W20260709 13:08:06.646034 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 167969us
I20260709 13:08:06.604470 25313 rpcz_store.cc:276] 0709 13:08:05.065857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:05.113398 (+ 47541us) service_pool.cc:224] Handling call
0709 13:08:06.571450 (+1458052us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b58000e3498. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:06.571636 (+   186us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:06.579570 (+  7934us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:06.579778 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4591872}
I20260709 13:08:06.555217 26899 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:06.651664 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.652485 25944 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 3: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:06.609167 26903 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:06.655407 26887 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:06.609705 26866 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:06.664970 25943 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
W20260709 13:08:06.661194 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 54889.6us
I20260709 13:08:06.652630 26827 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:06.666148 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.668676 25942 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 2: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:06.670305 26914 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:08:06.671777 25212 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:06.688692 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.680756 26884 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:06.683192 26881 consensus_meta.cc:287] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.513s	user 0.007s	sys 0.005s
I20260709 13:08:06.696105 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.697489 25944 raft_consensus.cc:2393] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 2: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:06.694597 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.709707 25942 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:06.710995 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.715476 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.715938 26903 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:06.716568 26903 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:06.710080 26909 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:06.716094 25942 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:06.722258 26846 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:06.728965 25945 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:06.729528 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:06.730513 25241 raft_consensus.cc:2393] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 2: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:06.724086 26848 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:06.731423 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:06.732625 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:06.724915 26899 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:06.737592 25416 debug-util.cc:398] Leaking SignalData structure 0x7b080030eec0 after lost signal to thread 25419
W20260709 13:08:06.738557 25416 debug-util.cc:398] Leaking SignalData structure 0x7b0800320cc0 after lost signal to thread 25420
I20260709 13:08:06.737915 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 1644) took 1074 ms. Trace:
I20260709 13:08:06.740027 25387 rpcz_store.cc:276] 0709 13:08:05.663757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:05.663946 (+   189us) service_pool.cc:224] Handling call
0709 13:08:05.672992 (+  9046us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:05.808894 (+135902us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:05.808939 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:05.808996 (+    57us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:08:05.813626 (+  4630us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:06.737707 (+924081us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b54001f59d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:06.737749 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:06.737863 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":296377088}
I20260709 13:08:06.741235 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.741940 25944 raft_consensus.cc:2393] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 2: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:06.700711 26881 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:06.744699 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.746747 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.755383 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.761531 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.762821 25937 raft_consensus.cc:2393] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 2: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:06.725874 26797 consensus_queue.cc:260] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:06.763742 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:06.763878 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:06.765626 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:06.767094 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:06.768031 25317 raft_consensus.cc:2393] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 3: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:06.769474 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:06.771749 25945 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 2: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:06.776113 25316 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:06.775967 26802 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:06.776969 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:06.777626 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:06.777151 25943 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:06.726940 26903 consensus_queue.cc:1059] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260709 13:08:06.781431 26802 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:06.781497 26881 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:06.782824 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.743310 26857 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:06.787561 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.778820 25316 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:06.768953 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:06.796280 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 114358us
I20260709 13:08:06.797501 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:06.797914 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 116501us
I20260709 13:08:06.788381 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.796947 26926 raft_consensus.cc:3055] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:06.829608 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:06.831600 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:08:06.832027 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:06.771251 26743 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260709 13:08:06.835487 26912 consensus_queue.cc:1059] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 13:08:06.841427 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.842023 25944 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 3: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:06.842742 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:06.846508 26728 consensus_queue.cc:1059] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260709 13:08:06.846856 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.847611 25940 raft_consensus.cc:2393] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 2: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:06.848479 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.835068 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:06.843880 25908 connection.cc:256] RPC call timeout handler was delayed by -1.1015609099999892s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19310000000000002s for a response
I20260709 13:08:06.827317 26857 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:06.854110 26875 consensus_queue.cc:1059] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260709 13:08:06.863417 26870 raft_consensus.cc:3055] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:06.863847 26870 raft_consensus.cc:740] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:06.864075 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.864705 26870 consensus_queue.cc:260] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:06.865454 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.865824 26870 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:06.866679 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:06.867400 25318 raft_consensus.cc:2393] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 2: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:06.869248 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:06.839735 26926 raft_consensus.cc:740] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:06.865490 26814 raft_consensus.cc:3055] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:06.872897 26814 raft_consensus.cc:740] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:06.872222 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:06.879909 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:06.883775 26869 consensus_queue.cc:1059] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:06.884176 25358 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:06.884768 25945 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:06.884934 26869 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:06.885497 25358 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:06.866694 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.886345 26869 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:06.886689 25936 raft_consensus.cc:2393] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 3: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:06.887212 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.887290 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:06.845921 26871 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
W20260709 13:08:07.026270 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 81710.5us
I20260709 13:08:06.941850 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.033519 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.941922 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1052) took 1722 ms. Trace:
W20260709 13:08:07.034135 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 148712us
I20260709 13:08:07.035264 25358 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:07.035473 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 131306us
I20260709 13:08:07.036542 25908 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.037544 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:07.034768 25244 rpcz_store.cc:276] 0709 13:08:05.160813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:05.161375 (+   562us) service_pool.cc:224] Handling call
0709 13:08:06.879456 (+1718081us) spinlock_profiling.cc:243] Waited 9.03 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8de9c 00007f2973b8da5b 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:06.879619 (+   163us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:06.879771 (+   152us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:06.879792 (+    21us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:06.882624 (+  2832us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:06.882666 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:06.882727 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:06.882827 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20760320}
I20260709 13:08:07.038483 25243 raft_consensus.cc:2393] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 2: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:06.943835 26879 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:07.037855 26775 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:06.888015 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:07.074456 25243 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 1044) took 1902 ms (client timeout 1794 ms). Trace:
I20260709 13:08:07.077100 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.077662 25240 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:07.083571 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1407) took 1231 ms. Trace:
I20260709 13:08:07.077936 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.079769 25317 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 2: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:07.077104 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:07.084005 25240 rpcz_store.cc:276] 0709 13:08:05.852507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:07.048535 (+1196028us) service_pool.cc:224] Handling call
0709 13:08:07.077478 (+ 28943us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:07.083242 (+  5764us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:07.083501 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:07.086504 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:07.086992 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1076) took 1230 ms. Trace:
I20260709 13:08:07.087427 25244 rpcz_store.cc:276] 0709 13:08:05.856347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:07.055582 (+1199235us) service_pool.cc:224] Handling call
0709 13:08:07.086551 (+ 30969us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:07.086711 (+   160us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:07.086736 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:07.086775 (+    39us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:07.086802 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:07.086854 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:07.086945 (+    91us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:08:07.080119 25243 rpcz_store.cc:269] 0709 13:08:05.145106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:05.161685 (+ 16579us) service_pool.cc:224] Handling call
0709 13:08:07.037387 (+1875702us) spinlock_profiling.cc:243] Waited 1.14 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:07.047663 (+ 10276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2630912}
I20260709 13:08:06.890542 26852 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:06.941831 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:06.893226 25242 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:08:07.073859 26869 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:06.880729 26794 consensus_queue.cc:1059] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.212s
I20260709 13:08:07.102187 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.098193 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.103298 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1077) took 1246 ms. Trace:
I20260709 13:08:07.103534 25945 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:07.103821 25240 rpcz_store.cc:276] 0709 13:08:05.856643 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:07.086601 (+1229958us) service_pool.cc:224] Handling call
0709 13:08:07.102816 (+ 16215us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:07.102983 (+   167us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:07.103007 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:07.103046 (+    39us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:07.103073 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:07.103123 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:07.103244 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:07.077281 26723 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.081249 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.092475 26870 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:07.092057 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.113950 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1080) took 1180 ms. Trace:
W20260709 13:08:07.109944 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:06.882987 26795 consensus_queue.cc:1059] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.215s
I20260709 13:08:07.116457 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.117805 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:07.118008 25945 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:06.890043 26906 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:07.115320 25244 rpcz_store.cc:276] 0709 13:08:05.933528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:07.099659 (+1166131us) service_pool.cc:224] Handling call
0709 13:08:07.113523 (+ 13864us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:07.113682 (+   159us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:07.113706 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:07.113763 (+    57us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:07.113793 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:07.113836 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:07.113909 (+    73us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:07.125669 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.125842 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:07.127501 25319 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 152) took 2091 ms (client timeout 1931 ms). Trace:
I20260709 13:08:07.131057 25936 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 2: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:07.157557 26906 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:07.131385 25416 debug-util.cc:398] Leaking SignalData structure 0x7b0800355080 after lost signal to thread 25969
I20260709 13:08:07.158102 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.132627 26857 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:07.135360 25237 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:07.136292 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1557) took 2164 ms. Trace:
I20260709 13:08:07.164852 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:07.165513 25936 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:08:07.166448 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:06.945169 26926 consensus_queue.cc:260] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:07.123399 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.139979 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:07.144366 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:07.149688 25319 rpcz_store.cc:269] 0709 13:08:05.035901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:05.037866 (+  1965us) service_pool.cc:224] Handling call
0709 13:08:07.127341 (+2089475us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400044f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:07.127429 (+    88us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2413751168}
I20260709 13:08:07.147934 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:07.153470 25359 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:07.155112 26869 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:07.156044 25938 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 3: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:07.156765 25940 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:07.131454 26875 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:07.163141 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:07.161798 25416 debug-util.cc:398] Leaking SignalData structure 0x7b080034cfe0 after lost signal to thread 25970
W20260709 13:08:07.172573 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 49276.8us
I20260709 13:08:07.180966 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:07.180984 25285 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.182942 25359 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:07.186645 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:07.184451 25321 raft_consensus.cc:2393] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 3: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:07.185599 25943 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:07.188988 25359 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:07.191907 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:07.192289 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.192528 25319 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 1.
I20260709 13:08:07.192667 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:07.164335 25315 rpcz_store.cc:276] 0709 13:08:04.972211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:04.977606 (+  5395us) service_pool.cc:224] Handling call
0709 13:08:04.977992 (+   386us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:05.852911 (+874919us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:08:05.885817 (+ 32906us) log.cc:844] Serialized 14 byte log entry
0709 13:08:06.010589 (+124772us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:06.010656 (+    67us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:06.010711 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:07.135987 (+1125276us) spinlock_profiling.cc:243] Waited 875 ms on lock 0x7b54000889d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:07.136024 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:07.136119 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":46071,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":95,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":1314,"spinlock_wait_cycles":2011677952,"thread_start_us":138956,"threads_started":2,"wal-append.queue_time_us":167563,"wal-append.run_cpu_time_us":1569,"wal-append.run_wall_time_us":1038666}
I20260709 13:08:07.193559 25319 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:07.198769 25945 raft_consensus.cc:2393] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 2: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:07.184520 26743 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.184800 25937 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.205261 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:07.207443 25321 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.211177 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:07.212334 26826 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:07.198446 26869 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:07.212715 26826 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:07.213165 26826 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.190035 26870 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:08:07.199061 25416 debug-util.cc:398] Leaking SignalData structure 0x7b080025fde0 after lost signal to thread 26768
I20260709 13:08:07.217909 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:07.198772 26926 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.192009 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:08:07.220891 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:07.222112 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 124029us
I20260709 13:08:07.192049 26857 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.225324 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.227734 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.228204 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.228343 25387 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:07.230645 25387 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:07.231072 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.231564 25237 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:08:07.232239 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:07.232802 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:07.232339 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.231491 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.234536 25387 raft_consensus.cc:2393] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:08:07.225025 25416 debug-util.cc:398] Leaking SignalData structure 0x7b0800067780 after lost signal to thread 26789
I20260709 13:08:07.234902 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.235237 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.235842 25390 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:07.212715 26728 consensus_queue.cc:1059] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260709 13:08:07.201200 26912 consensus_queue.cc:1059] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 13:08:07.242424 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.244006 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.244611 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.245585 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.245868 26873 raft_consensus.cc:3055] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:07.246279 26873 raft_consensus.cc:740] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:07.246407 26912 raft_consensus.cc:3055] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:07.246984 26912 raft_consensus.cc:740] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:07.247396 26873 consensus_queue.cc:260] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:07.247855 26912 consensus_queue.cc:260] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:07.248646 26873 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.249064 26912 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.243124 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.241847 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.255635 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:07.257023 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.258960 25387 raft_consensus.cc:2393] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 3: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:07.259822 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:07.260131 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.259838 26834 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.260799 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.261364 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:07.261641 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:07.262050 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:07.262903 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:07.263175 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:07.243168 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.270342 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.263643 26858 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20260709 13:08:07.270916 25944 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:07.271723 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.272230 25944 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:07.273411 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:07.273716 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:07.273838 25287 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.274904 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:07.275420 26801 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:07.276001 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:07.276780 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.246253 25936 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:07.274009 25246 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
W20260709 13:08:07.280570 25246 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 1049) took 2120 ms (client timeout 2163 ms). Trace:
W20260709 13:08:07.280913 25246 rpcz_store.cc:269] 0709 13:08:05.159883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:05.166540 (+  6657us) service_pool.cc:224] Handling call
0709 13:08:07.273275 (+2106735us) spinlock_profiling.cc:243] Waited 1.48 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:07.280495 (+  7220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3400448}
I20260709 13:08:07.263407 26846 consensus_queue.cc:1059] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260709 13:08:07.281929 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:07.282009 25937 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:07.282761 26846 raft_consensus.cc:3055] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:07.283008 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:07.283581 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.284122 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:07.284284 25392 raft_consensus.cc:2393] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:07.284222 25937 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:07.283123 26846 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:07.273000 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.285585 26826 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:07.286012 26846 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:07.287202 26846 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:07.288587 26876 raft_consensus.cc:3055] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:07.288975 26876 raft_consensus.cc:740] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:07.289798 26876 consensus_queue.cc:260] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:07.290711 25321 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:08:07.290920 26876 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.287443 25284 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.294751 25285 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.295053 25284 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.296164 25284 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:07.287660 26826 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:07.264078 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:07.263513 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.296409 25909 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:07.298686 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:07.298830 25942 raft_consensus.cc:1240] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
W20260709 13:08:07.299440 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:07.307482 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.308118 25387 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:07.302628 26748 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.308627 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.309414 25389 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:07.310063 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:07.310631 25389 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:08:07.310886 25940 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:07.302812 26775 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:07.311666 26775 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:07.312000 26775 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:07.312112 25940 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:07.312844 25391 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:08:07.313457 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:07.312136 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:07.314373 25936 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:07.315268 25942 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:07.315600 25936 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:07.302811 25911 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.305923 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:07.316237 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:07.317417 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:07.317819 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:07.318574 25284 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.318905 25911 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.319290 25909 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:07.321496 25390 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:07.321867 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.323302 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.323670 26861 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:07.324085 25392 raft_consensus.cc:2393] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 2: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:07.324167 26861 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:07.328055 26861 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:07.324976 26866 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:07.325304 26723 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.325533 26801 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.338852 26723 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.325652 26908 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:07.339985 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:07.321008 26834 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.326186 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:07.327124 26802 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.344460 26856 consensus_queue.cc:1059] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:08:07.316862 26858 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:07.333636 25139 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53264 (request call id 202) took 6205 ms. Trace:
I20260709 13:08:07.323006 25911 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.336077 26772 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.347437 25942 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:07.343289 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:07.348453 25909 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:07.329613 25284 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.317224 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:07.321938 26857 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.367780 25394 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:07.396247 25359 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.365423 25909 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:07.334463 26866 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:07.416778 26848 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:08:07.418411 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:07.365078 26802 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.366315 25940 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:07.427403 25359 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:07.423704 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 94129.6us
I20260709 13:08:07.431455 25284 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.404035 26723 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.370885 26930 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:07.433065 25284 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.469672 25359 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:07.422003 26866 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:07.472230 25359 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:07.367514 26908 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.102s
I20260709 13:08:07.475924 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:07.357916 26834 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20260709 13:08:07.478910 26937 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.479259 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:07.468637 26873 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.431576 26795 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 13:08:07.482297 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 90477us
I20260709 13:08:07.483100 26941 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.493283 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.494127 25359 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:07.483625 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:07.489209 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.496142 25287 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.489249 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:07.477594 26930 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:07.490265 26842 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:07.476241 26778 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:07.500564 25358 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.502082 25393 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 2. Current term is 3. Ops: []
W20260709 13:08:07.497000 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 131625us
I20260709 13:08:07.474993 26931 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:07.492178 25942 raft_consensus.cc:2955] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:08:07.565191 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 71958.3us
I20260709 13:08:07.567786 25212 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.573014 25395 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:07.575795 25240 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
W20260709 13:08:07.575937 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 75436.8us
I20260709 13:08:07.501490 26794 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:07.346899 25139 rpcz_store.cc:276] 0709 13:08:01.123048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:04.010027 (+2886979us) service_pool.cc:224] Handling call
0709 13:08:07.328824 (+3318797us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:08:05.527902 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:08:05.845459 (+317557us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b5400470d28. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297d06e9e1 00007f297d06d64c 00007f297a6fcd5d 00007f297a6e80a6 00007f297a6e7d58 00007f297a6ec9f1 00007f297a6ec9a9 00007f297a6ec939 00007f297a6ec901 00007f297a6ebb2d 000000000038f354
0709 13:08:05.845590 (+   131us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:08:05.845637 (+    47us) write_op.cc:435] Acquired schema lock
0709 13:08:05.846403 (+   766us) tablet.cc:662] Decoding operations
0709 13:08:05.864368 (+ 17965us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:08:05.864478 (+   110us) write_op.cc:641] Partition lock acquired for write op
0709 13:08:05.864504 (+    26us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:08:05.864815 (+   311us) tablet.cc:701] Row locks acquired
0709 13:08:05.864856 (+    41us) write_op.cc:260] PREPARE: finished
0709 13:08:05.865030 (+   174us) write_op.cc:270] Start()
0709 13:08:05.865280 (+   250us) write_op.cc:276] Timestamp: P: 1783602485864972 usec, L: 0
0709 13:08:05.865322 (+    42us) op_driver.cc:348] REPLICATION: starting
0709 13:08:05.882867 (+ 17545us) log.cc:844] Serialized 599 byte log entry
0709 13:08:06.520834 (+637967us) op_driver.cc:464] REPLICATION: finished
0709 13:08:06.711960 (+191126us) write_op.cc:301] APPLY: starting
0709 13:08:06.712068 (+   108us) tablet.cc:1367] starting BulkCheckPresence
0709 13:08:06.712178 (+   110us) tablet.cc:1370] finished BulkCheckPresence
0709 13:08:06.712215 (+    37us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:08:06.712607 (+   392us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:08:06.712855 (+   248us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:08:06.712911 (+    56us) write_op.cc:312] APPLY: finished
0709 13:08:06.715934 (+  3023us) log.cc:844] Serialized 23 byte log entry
0709 13:08:06.716126 (+   192us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:06.716206 (+    80us) write_op.cc:454] Released schema lock
0709 13:08:06.716261 (+    55us) write_op.cc:341] FINISH: Updating metrics
0709 13:08:06.716612 (+   351us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:06.716644 (+    32us) write_op.cc:454] Released schema lock
0709 13:08:07.140455 (+423811us) spinlock_profiling.cc:243] Waited 3.6 ms on lock 0x7b5400388c20. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976c27dbe 00007f2976c32930 00007f2976c328d8 00007f2976c32858 00007f2976c32813 00007f2976c31a9c 00007f297cb1d626 00007f297cb17dd8 00007f2976c6e7cb 00007f2976c712c0
Metrics: {"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":190839,"apply.run_cpu_time_us":2106,"apply.run_wall_time_us":4625,"num_ops":1,"prepare.queue_time_us":336060,"prepare.run_cpu_time_us":17892,"prepare.run_wall_time_us":371441,"raft.queue_time_us":347099,"raft.run_cpu_time_us":43674,"raft.run_wall_time_us":187466,"replication_time_us":636391,"spinlock_wait_cycles":62230528,"thread_start_us":487351,"threads_started":4,"wal-append.queue_time_us":15250}]]}
I20260709 13:08:07.560205 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:07.586583 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 90514.9us
I20260709 13:08:07.572290 25388 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:07.594965 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:07.587296 26938 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:07.575413 26795 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.505923 26826 consensus_queue.cc:1059] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.081s
W20260709 13:08:07.596036 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 163067us
I20260709 13:08:07.614283 25284 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.615434 25284 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.616523 25284 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.598157 25287 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.579298 26857 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:07.586192 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:07.495182 26875 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:07.601102 26778 consensus_queue.cc:1059] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:08:07.603626 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:07.604050 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:07.604633 26929 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:07.607755 25943 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:07.606999 26930 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:07.583280 26903 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.582458 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.598421 25212 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.631933 25284 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.639667 25212 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.609306 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:07.608008 26936 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:08:07.641355 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 59017.6us
I20260709 13:08:07.647910 25209 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.614954 26928 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:07.651546 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.652197 25943 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:07.652462 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.662854 26801 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.665170 25285 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:07.641269 26943 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:07.663585 26801 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:08:07.668010 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 69938.8us
I20260709 13:08:07.676618 25287 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.650451 26925 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.655508 26826 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.655757 26928 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:07.646451 26927 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:07.664410 26930 raft_consensus.cc:3055] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:07.668020 26857 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:07.692976 25316 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:07.656736 26743 consensus_queue.cc:1059] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260709 13:08:07.682356 26936 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:07.714784 26856 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:07.714802 26947 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.715484 26856 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:07.682847 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:07.689292 26930 raft_consensus.cc:740] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:07.716871 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:07.673617 26946 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.683595 26928 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:08:07.724231 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 71813.3us
I20260709 13:08:07.725060 25209 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:07.724655 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 59580.8us
W20260709 13:08:07.726445 26940 log.cc:927] Time spent T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260709 13:08:07.726500 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:07.707083 26792 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:07.714270 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:07.729278 25209 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:07.717694 26936 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:08:07.730939 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:07.735827 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:07.739535 26926 consensus_meta.cc:287] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.508s	user 0.007s	sys 0.002s
I20260709 13:08:07.745092 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:07.733017 26936 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.741603 26748 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.731544 26925 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.732183 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:07.741743 26950 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:07.748546 25315 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:07.750283 25237 raft_consensus.cc:3055] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:07.753791 25237 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:07.750705 26758 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:07.771149 26758 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:07.785295 26758 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.786870 26861 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:07.788287 25316 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:07.771270 26902 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:07.771554 26748 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:07.788954 26748 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:07.751681 26950 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:07.789268 26930 consensus_queue.cc:260] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:07.789990 26950 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.790484 26930 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.775043 25242 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:07.754585 25237 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:07.752828 26879 raft_consensus.cc:3055] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:07.795120 26879 raft_consensus.cc:740] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:07.730106 26792 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:07.760963 26926 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:07.797960 26834 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:08:07.799796 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 160203us
I20260709 13:08:07.811574 25212 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.801465 26856 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:07.804118 25237 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:08:07.813983 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:07.814595 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.813869 25212 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.815613 26954 consensus_queue.cc:1059] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:08:07.792680 26748 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:07.817721 26948 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:07.820596 26948 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:07.822005 26917 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.819649 25212 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.831524 26797 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.821187 26948 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:07.804812 26926 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:07.813026 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:07.846839 26936 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:07.859236 25212 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:07.850253 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 218434us
I20260709 13:08:07.861649 25284 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.863029 26745 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:08:07.863503 26745 raft_consensus.cc:697] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:07.864404 26745 consensus_queue.cc:237] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:07.865865 26827 raft_consensus.cc:3055] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:07.866314 26827 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:07.853709 26748 consensus_queue.cc:1059] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
W20260709 13:08:07.867019 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:07.874445 25284 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.875556 25284 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.848287 26926 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.881297 26953 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.881570 26748 raft_consensus.cc:3055] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:07.882514 25284 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.825636 26902 consensus_queue.cc:1059] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 13:08:07.849937 26876 consensus_queue.cc:1059] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260709 13:08:07.887738 25943 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
W20260709 13:08:07.888970 26846 consensus_meta.cc:287] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.600s	user 0.008s	sys 0.005s
I20260709 13:08:07.866920 26925 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 13:08:07.886365 26952 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.146s	user 0.001s	sys 0.000s
I20260709 13:08:07.902272 26827 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:07.886888 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:07.849820 26758 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:07.909658 26938 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.911149 26795 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:07.892386 26945 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:08:07.912628 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 64933.6us
I20260709 13:08:07.915087 25319 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: []
W20260709 13:08:07.918244 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 59091.3us
I20260709 13:08:07.895170 26748 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:07.916808 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:07.917165 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:07.880332 26954 consensus_queue.cc:1059] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 13:08:07.900838 26879 consensus_queue.cc:260] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:07.908522 26743 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:07.924786 25314 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:07.922752 26925 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:07.927722 26827 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:07.911383 26943 consensus_queue.cc:1059] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
W20260709 13:08:07.933374 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:07.919462 26801 consensus_queue.cc:1059] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
W20260709 13:08:07.958058 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:07.929498 25937 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:08:07.932056 25212 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.953990 25242 raft_consensus.cc:2393] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 2: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:07.923813 26947 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:07.956928 26879 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:07.945415 26937 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:08:07.933336 26775 consensus_meta.cc:287] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.619s	user 0.012s	sys 0.000s
I20260709 13:08:07.936482 26883 raft_consensus.cc:3055] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:07.964407 25212 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:07.966078 26861 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:07.966864 25314 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:07.968616 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:07.971251 26955 log.cc:927] Time spent T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 13:08:07.942289 26876 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:07.936834 26866 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:07.980614 25942 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:07.945600 26743 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:07.986688 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:08.013465 26758 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:08.016255 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:07.973472 26925 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:07.992959 25212 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:07.979074 26883 raft_consensus.cc:740] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:08:08.026229 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:08.029292 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 70381.1us
I20260709 13:08:08.010169 25317 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
W20260709 13:08:08.028646 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:08.029016 25316 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:07.982362 26929 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:08.028244 25212 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:08.024811 26814 consensus_queue.cc:260] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.034072 26814 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:08.021101 26958 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:08.032505 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:08.030928 26758 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:08.052772 26758 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:08.034830 25237 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:08:08.055133 25314 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:08.020115 26876 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.042403 26866 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:08.043910 26905 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:08.046494 26743 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:08.047237 26883 consensus_queue.cc:260] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.053951 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:08.055790 25140 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39596 (request call id 281) took 3665 ms. Trace:
I20260709 13:08:07.977914 26775 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:08.057621 25212 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:08.021157 26928 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.053256 26758 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:08.062229 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:08.060636 26957 raft_consensus.cc:3055] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:08.066746 25314 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:08.036247 26902 consensus_queue.cc:1059] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 13:08:08.057669 26929 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:07.986506 26947 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.082607 25320 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20260709 13:08:08.040804 26950 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.062305 26826 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:08.086999 26961 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:08.088624 26928 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:08.090518 26950 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:08.095820 25390 raft_consensus.cc:2955] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:08.096825 26771 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:08.076272 26876 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:08.077396 26883 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:08.103091 25391 raft_consensus.cc:1240] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:08.100536 25359 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:08.101362 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:08.103149 25389 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:08.104064 26926 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.104552 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:08.118881 25392 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:08.119897 25359 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:08.058816 25140 rpcz_store.cc:276] 0709 13:08:04.390267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:05.375980 (+985713us) service_pool.cc:224] Handling call
0709 13:08:08.055727 (+2679747us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:08:06.695678 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:08:06.706836 (+ 11158us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:08:06.706871 (+    35us) write_op.cc:435] Acquired schema lock
0709 13:08:06.706891 (+    20us) tablet.cc:662] Decoding operations
0709 13:08:06.707211 (+   320us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:08:06.707272 (+    61us) write_op.cc:641] Partition lock acquired for write op
0709 13:08:06.707302 (+    30us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:08:06.707508 (+   206us) tablet.cc:701] Row locks acquired
0709 13:08:06.707525 (+    17us) write_op.cc:260] PREPARE: finished
0709 13:08:06.707662 (+   137us) write_op.cc:270] Start()
0709 13:08:06.707770 (+   108us) write_op.cc:276] Timestamp: P: 1783602486707615 usec, L: 0
0709 13:08:06.707790 (+    20us) op_driver.cc:348] REPLICATION: starting
0709 13:08:06.758800 (+ 51010us) log.cc:844] Serialized 599 byte log entry
0709 13:08:07.142010 (+383210us) op_driver.cc:464] REPLICATION: finished
0709 13:08:07.142360 (+   350us) write_op.cc:301] APPLY: starting
0709 13:08:07.142444 (+    84us) tablet.cc:1367] starting BulkCheckPresence
0709 13:08:07.142544 (+   100us) tablet.cc:1370] finished BulkCheckPresence
0709 13:08:07.142659 (+   115us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:08:07.142917 (+   258us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:08:07.143068 (+   151us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:08:07.143118 (+    50us) write_op.cc:312] APPLY: finished
0709 13:08:07.143320 (+   202us) log.cc:844] Serialized 23 byte log entry
0709 13:08:07.143474 (+   154us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:07.143550 (+    76us) write_op.cc:454] Released schema lock
0709 13:08:07.143592 (+    42us) write_op.cc:341] FINISH: Updating metrics
0709 13:08:07.143933 (+   341us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:07.143963 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":184558,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":118,"apply.run_cpu_time_us":1520,"apply.run_wall_time_us":1519,"num_ops":1,"prepare.queue_time_us":280331,"prepare.run_cpu_time_us":3609,"prepare.run_wall_time_us":88027,"replication_time_us":433832,"thread_start_us":237705,"threads_started":1}]]}
I20260709 13:08:08.082962 26957 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:08.105259 25395 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:08.082628 26948 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:08.106833 26946 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:08.127425 25321 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:08.128321 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:08.129911 25359 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:08.131585 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:08.125922 26867 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 13:08:08.132215 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 67795.3us
I20260709 13:08:08.083794 26827 consensus_queue.cc:1059] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260709 13:08:08.116344 26936 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:08.107018 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:08.134812 25393 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:08.135680 25212 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:08.132988 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:08.140741 25395 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:08.141510 25359 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:08.142418 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:08.126127 26876 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:08:08.145422 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 61858us
I20260709 13:08:08.126439 26925 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:08.147326 25359 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:08.150128 26963 consensus_queue.cc:1059] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:08:08.144161 26826 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:08.119007 26929 consensus_queue.cc:1059] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260709 13:08:08.134814 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:08.132683 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:08.134472 26827 raft_consensus.cc:3055] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:08.131484 26927 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:08.144140 26928 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:08.153623 26867 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:08.162732 26775 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:08.154840 26941 consensus_meta.cc:287] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.593s	user 0.008s	sys 0.004s
I20260709 13:08:08.166721 25241 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
W20260709 13:08:08.166618 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 45602.8us
I20260709 13:08:08.144876 26758 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.172119 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:08.156414 26827 raft_consensus.cc:740] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:08.165150 26826 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:08.166386 26948 consensus_queue.cc:1059] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260709 13:08:08.173924 26826 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.174324 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:08.159720 26927 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.175446 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:08.176080 26826 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:08.174242 26827 consensus_queue.cc:260] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.176450 26927 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:08.172632 26743 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:08.177275 26810 raft_consensus.cc:3055] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:08.177373 26827 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:08.177739 26810 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:08.176484 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:08.181986 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:08.184087 25314 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:08.169960 26941 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:08.159344 25238 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:08.185978 25209 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:08.165925 26926 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:08.152982 26957 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.181659 25237 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:08.188194 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:08.190472 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:08.190708 25238 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:08.191431 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 507) took 1318 ms. Trace:
I20260709 13:08:08.191955 26826 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:08.191840 25941 rpcz_store.cc:276] 0709 13:08:06.877478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:06.886613 (+  9135us) service_pool.cc:224] Handling call
0709 13:08:08.190313 (+1303700us) spinlock_profiling.cc:243] Waited 1.04 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:08.191337 (+  1024us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2390784}
I20260709 13:08:08.192807 25238 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:08.193969 26826 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:08.196426 26826 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:08.196653 26914 raft_consensus.cc:3055] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:08.195335 26944 consensus_queue.cc:1059] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:08:08.197109 26914 raft_consensus.cc:740] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:08.197703 26944 raft_consensus.cc:3055] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:08.198076 26944 raft_consensus.cc:740] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:08.198110 26914 consensus_queue.cc:260] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.199002 26944 consensus_queue.cc:260] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.199620 26914 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:08.200555 26944 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:08.204128 25319 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:08.205866 26881 consensus_queue.cc:1059] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:08.206909 26795 raft_consensus.cc:3055] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:08.207175 26881 raft_consensus.cc:1064] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:08:08.207309 26795 raft_consensus.cc:740] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:08:08.207696 26881 raft_consensus.cc:1068] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: Unable to promote non-voter f646dfd532974dceadfe00005d2bbf59: Illegal state: Leader has not yet committed an operation in its own term
I20260709 13:08:08.208294 26795 consensus_queue.cc:260] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.209777 26795 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:08.211552 25392 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:08.212100 26872 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:08.213860 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:08.195235 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:08.178696 26810 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.215348 26810 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:08.215430 26814 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:08.215894 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 346) took 1371 ms. Trace:
I20260709 13:08:08.216359 25937 rpcz_store.cc:276] 0709 13:08:06.845888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:07.302492 (+456604us) service_pool.cc:224] Handling call
0709 13:08:07.315879 (+ 13387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:07.963332 (+647453us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:08.215605 (+252273us) spinlock_profiling.cc:243] Waited 600 ms on lock 0x7b54001b2bd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:08.215816 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1379011840}
I20260709 13:08:08.214425 26953 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:08.219115 26748 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.219825 25312 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:08.220361 26748 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:08.221670 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:08.221871 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:08.195998 25238 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:08.223083 25243 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:08.223959 25238 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:08.215687 26957 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:08.214586 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 255) took 1004 ms. Trace:
I20260709 13:08:08.229272 25936 raft_consensus.cc:1240] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:08.229571 26775 consensus_queue.cc:1059] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:08:08.229909 26941 consensus_queue.cc:1059] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:08.230700 26896 consensus_queue.cc:1059] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:08.231769 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:08.231842 25284 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:08.232627 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:08.233346 25240 rpcz_store.cc:276] 0709 13:08:07.210139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:07.584302 (+374163us) service_pool.cc:224] Handling call
0709 13:08:08.213698 (+629396us) spinlock_profiling.cc:243] Waited 2.31 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:08.214517 (+   819us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5303040}
I20260709 13:08:08.206358 26905 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:08.219678 26958 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:08.240005 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:08.240737 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 45551.3us
W20260709 13:08:08.241825 26916 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 13:08:08.252328 26772 raft_consensus.cc:3055] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:08.252712 26772 raft_consensus.cc:740] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:08.233363 25940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 519) took 1341 ms. Trace:
I20260709 13:08:08.237715 26848 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:08.253598 26772 consensus_queue.cc:260] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.254638 26772 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:08.243647 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 344) took 1454 ms. Trace:
I20260709 13:08:08.255297 25938 rpcz_store.cc:276] 0709 13:08:06.792817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:06.853341 (+ 60524us) service_pool.cc:224] Handling call
0709 13:08:08.243407 (+1390066us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b54004d7954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:08.243579 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":616151936}
I20260709 13:08:08.244249 25944 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:08.252760 25908 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:08.258141 25908 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:08.258533 25944 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 481) took 1422 ms. Trace:
I20260709 13:08:08.259537 25908 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:08.244997 26852 raft_consensus.cc:3055] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:08.261814 26910 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:08.262115 26852 raft_consensus.cc:740] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:08.262799 26775 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:08.215442 26947 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:08.263231 26852 consensus_queue.cc:260] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.264369 25237 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:08.245476 26896 raft_consensus.cc:3055] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:08.246256 26941 raft_consensus.cc:3055] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:08.265352 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:08.265772 26941 raft_consensus.cc:740] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:08.265887 26792 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:08.267009 25238 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:08.266784 26941 consensus_queue.cc:260] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.268141 26941 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:08.268474 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:08.247006 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:08.247170 26758 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:08.247669 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:08.253265 25940 rpcz_store.cc:276] 0709 13:08:06.891852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:07.313003 (+421151us) service_pool.cc:224] Handling call
0709 13:08:07.336418 (+ 23415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:07.468450 (+132032us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:08.233068 (+764618us) spinlock_profiling.cc:243] Waited 22.8 ms on lock 0x7b540065eda0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976c29dc0 00007f2976cb5a02 00007f2976cb1dd6 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2
0709 13:08:08.233300 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":52368128}
I20260709 13:08:08.261669 26906 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:08.268159 25240 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:08.270459 25359 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:08.265878 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:08.278230 25240 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
W20260709 13:08:08.278599 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:08.279172 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:08.279798 25243 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:08.275920 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:08.281404 26792 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:08.281699 25939 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:08.212694 26826 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.259171 25944 rpcz_store.cc:276] 0709 13:08:06.836005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:07.286575 (+450570us) service_pool.cc:224] Handling call
0709 13:08:08.244073 (+957498us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:08.256774 (+ 12701us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:08.258275 (+  1501us) spinlock_profiling.cc:243] Waited 957 ms on lock 0x7b54004d7950. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:08.258456 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2201289600}
I20260709 13:08:08.281855 26792 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:08.236776 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:08.288276 26943 raft_consensus.cc:3055] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:08.303886 25209 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:08.288511 26879 consensus_queue.cc:1059] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:08:08.292634 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:08.306152 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:08.293649 25242 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:08:08.293475 26743 consensus_queue.cc:1059] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:08:08.274770 26947 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260709 13:08:08.316830 26775 consensus_queue.cc:1059] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:08:08.317724 26743 raft_consensus.cc:3055] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:08.294363 26875 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:08.318173 26743 raft_consensus.cc:740] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:08.316823 26879 raft_consensus.cc:3055] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:08.318512 26875 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.318959 26879 raft_consensus.cc:740] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:08.319212 26743 consensus_queue.cc:260] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.320547 26743 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:08.320009 26879 consensus_queue.cc:260] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.303108 26943 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:08.265480 26896 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:08.282500 26634 consensus_queue.cc:1059] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260709 13:08:08.308960 25141 catalog_manager.cc:5697] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index 3 to 4, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:08:08.298749 26856 raft_consensus.cc:3055] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:08.273895 26939 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:08.299823 26797 raft_consensus.cc:3055] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:08.300382 25359 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:08.300662 25143 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39582 (request call id 304) took 2373 ms. Trace:
I20260709 13:08:08.301899 25908 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:08.309042 26758 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 13:08:08.282323 25939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 324) took 1585 ms (client timeout 2046 ms). Trace:
I20260709 13:08:08.295394 26792 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:08:08.326814 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:08.327138 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:08.327437 25943 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
W20260709 13:08:08.298691 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:08.320919 26875 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:08.303426 26852 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:08.340977 26797 raft_consensus.cc:740] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:08.341395 25143 rpcz_store.cc:276] 0709 13:08:05.927689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:05.927889 (+   200us) service_pool.cc:224] Handling call
0709 13:08:08.300514 (+2372625us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":751637,"reported_tablets":1}
W20260709 13:08:08.345840 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:08.353039 25944 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 3: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:08.353991 25240 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:08:08.336239 26856 raft_consensus.cc:740] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:08.339592 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:08.343868 26810 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:08.364802 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:08.367058 26758 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:08.357916 26923 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:08.358937 26879 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:08:08.390625 25394 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 1663) took 2489 ms (client timeout 1863 ms). Trace:
I20260709 13:08:08.388132 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:08:08.344285 25939 rpcz_store.cc:269] 0709 13:08:06.697103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:06.709195 (+ 12092us) service_pool.cc:224] Handling call
0709 13:08:07.076143 (+366948us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:08.273808 (+1197665us) spinlock_profiling.cc:243] Waited 6.72 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8de9c 00007f2973b8da5b 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:08.282259 (+  8451us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18246784}
W20260709 13:08:08.391050 25394 rpcz_store.cc:269] 0709 13:08:05.892788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:07.260551 (+1367763us) service_pool.cc:224] Handling call
0709 13:08:08.382244 (+1121693us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b54000dd9d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:08.382373 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":244577152}
W20260709 13:08:08.396720 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:08.371178 26943 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.398524 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:08.399458 25321 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:08.399474 26943 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:08.413353 26972 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:08:08.413991 26972 raft_consensus.cc:697] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:08.414912 26972 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.418119 25244 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:08.418356 25321 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
W20260709 13:08:08.420202 26966 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 13:08:08.423514 26848 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:08:08.427866 25241 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 230) took 1997 ms (client timeout 2109 ms). Trace:
I20260709 13:08:08.374711 26958 consensus_queue.cc:1059] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260709 13:08:08.431248 25245 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 13:08:08.428275 25241 rpcz_store.cc:269] 0709 13:08:06.430976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:06.755573 (+324597us) service_pool.cc:224] Handling call
0709 13:08:08.427697 (+1672124us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b5400456854. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:08.427814 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3189725568}
I20260709 13:08:08.438880 26748 consensus_queue.cc:1059] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 13:08:08.441326 26947 raft_consensus.cc:3055] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:08.441718 26947 raft_consensus.cc:740] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:08.442626 26947 consensus_queue.cc:260] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.443902 26947 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:08.417508 26875 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:08.419102 26939 consensus_queue.cc:1059] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 13:08:08.449792 26842 raft_consensus.cc:3055] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:08.450191 26842 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:08.451329 26953 raft_consensus.cc:3055] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:08.451053 26842 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.451862 26953 raft_consensus.cc:740] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:08.452576 26842 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:08.389746 26797 consensus_queue.cc:260] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:08.458630 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 147533us
I20260709 13:08:08.473526 25392 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:08.475267 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:08.476706 25392 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:08:08.478696 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:08.479478 25387 raft_consensus.cc:2393] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 2: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:08.480548 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:08.481256 25389 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:08:08.481621 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:08.480715 25395 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:08.482506 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 240) took 2036 ms. Trace:
I20260709 13:08:08.482916 25245 rpcz_store.cc:276] 0709 13:08:06.446424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:07.135448 (+689024us) service_pool.cc:224] Handling call
0709 13:08:08.428995 (+1293547us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:08.431787 (+  2792us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:08.482350 (+ 50563us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400456850. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:08.482456 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2974225152}
I20260709 13:08:08.485489 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:08.379748 26896 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.490969 25287 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:08.467696 26826 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:08.493207 25318 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:08.476673 25941 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:08:08.459048 26875 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.448166 26856 consensus_queue.cc:260] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.503561 26856 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:08:08.502699 26968 log.cc:927] Time spent T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 13:08:08.504977 25287 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:08.505640 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:08.500365 26797 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:08.506999 26771 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:08.517990 26875 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:08.518242 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1101) took 1217 ms. Trace:
I20260709 13:08:08.518692 25239 rpcz_store.cc:276] 0709 13:08:07.300571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:08.177008 (+876437us) service_pool.cc:224] Handling call
0709 13:08:08.177429 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:08.184586 (+  7157us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:08.185954 (+  1368us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:08.193576 (+  7622us) log.cc:844] Serialized 26 byte log entry
0709 13:08:08.349346 (+155770us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:08.358064 (+  8718us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:08.365516 (+  7452us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:08.518000 (+152484us) raft_consensus.cc:1685] finished
0709 13:08:08.518031 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:08.518178 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4968,"raft.queue_time_us":6476,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":2301,"thread_start_us":127926,"threads_started":1,"wal-append.queue_time_us":149911}
I20260709 13:08:08.540200 26865 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:08.566762 25246 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:08.547843 26748 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:08.575331 26865 consensus_queue.cc:1059] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:08:08.576804 26949 raft_consensus.cc:3055] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:08.577251 26949 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:08:08.578024 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:08.453012 26953 consensus_queue.cc:260] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.584275 26949 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.590126 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1412) took 1314 ms. Trace:
I20260709 13:08:08.593163 25239 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
mode: GRACEFUL
new_leader_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:37596
I20260709 13:08:08.593961 25239 raft_consensus.cc:606] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Received request to transfer leadership to TS 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:08.594321 25239 raft_consensus.cc:614] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:08:08.399540 26634 consensus_queue.cc:1175] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f646dfd532974dceadfe00005d2bbf59" responder_term: 3 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 13:08:08.504365 26946 consensus_queue.cc:1175] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" responder_term: 3 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:08:08.600210 25388 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:52318
I20260709 13:08:08.590622 25242 rpcz_store.cc:276] 0709 13:08:07.275635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:08.047803 (+772168us) service_pool.cc:224] Handling call
0709 13:08:08.057304 (+  9501us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:08.309442 (+252138us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:08.589855 (+280413us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b5400422554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:08.590059 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":522383744}
I20260709 13:08:08.600836 25388 raft_consensus.cc:606] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:08.607035 25287 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:08.610040 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 308191us
W20260709 13:08:08.611624 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:08.616137 25287 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:08.618515 25287 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:08.619503 26772 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:08.620321 26772 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:08.620632 25937 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:08.620714 26772 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:08.618599 26944 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260709 13:08:08.624466 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:08.604436 26949 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:08.592772 26941 consensus_queue.cc:1059] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
W20260709 13:08:08.630877 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 251898us
I20260709 13:08:08.632552 25287 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:08.604736 26743 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260709 13:08:08.626457 26947 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:08.616943 26748 consensus_queue.cc:1059] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260709 13:08:08.629084 26944 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:08.636970 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:08:08.642226 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:08:08.637995 26946 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:08.658912 26792 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.638711 25936 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:08.638974 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:08.661422 25394 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:08.660214 26772 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.661099 26792 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:08.672498 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:08.669358 26908 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260709 13:08:08.664996 25941 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:08.664615 26879 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:08.681879 25242 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:08.685561 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:08.683701 26772 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:08.676774 26745 raft_consensus.cc:3055] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:08.675323 26797 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:08.690115 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 253) took 1480 ms. Trace:
I20260709 13:08:08.690579 25246 rpcz_store.cc:276] 0709 13:08:07.209400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:07.316481 (+107081us) service_pool.cc:224] Handling call
0709 13:08:08.247539 (+931058us) spinlock_profiling.cc:243] Waited 1.06 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:08.689921 (+442382us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x7b54003a8ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:08.690048 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":677277184}
I20260709 13:08:08.671376 26748 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260709 13:08:08.689204 26745 raft_consensus.cc:740] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:08.692530 26917 raft_consensus.cc:3055] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:08.686630 26879 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.724786 26745 consensus_queue.cc:260] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.726322 26745 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:08.729058 26879 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:08.733139 25209 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:08.702826 26772 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:08:08.746470 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 70494.4us
W20260709 13:08:08.749598 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 111277us
I20260709 13:08:08.751310 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:08.751859 25321 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
W20260709 13:08:08.752244 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 332289us
I20260709 13:08:08.694418 26797 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.714201 26917 raft_consensus.cc:740] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:08.755105 25321 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 2. Current term is 3. Ops: []
W20260709 13:08:08.757184 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:08.758455 25909 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:08.759441 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:08.748843 26896 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:08:08.761346 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:08.761884 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:08.762063 26914 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:08.764061 25387 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:08.748274 26875 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:08.765581 26906 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:08:08.769418 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:08.772433 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:08.756965 26950 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:08:08.778355 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:08.774928 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:08.779460 25316 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:08.779845 25284 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:08.784094 25390 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:08.777041 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:08.785290 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:08.791549 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:08.751083 26772 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.795289 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:08.796402 26946 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:08.797339 26910 consensus_queue.cc:1059] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:08:08.799999 25242 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:08.772928 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:08.801501 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:08.804128 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:08.840479 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:08.832469 26936 raft_consensus.cc:3055] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:08.841184 26936 raft_consensus.cc:740] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:08.842024 26936 consensus_queue.cc:260] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.843282 26936 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:08.805250 26917 consensus_queue.cc:260] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.844504 26917 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:08.846266 26772 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:08.807015 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:08.807684 26797 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:08.777895 26875 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.763300 26970 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 13:08:08.848088 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:08.849371 26875 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:08.807723 26743 consensus_queue.cc:1059] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 13:08:08.850342 26743 raft_consensus.cc:3055] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:08.850701 26743 raft_consensus.cc:740] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:08.812073 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:08.778153 26943 consensus_queue.cc:1059] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260709 13:08:08.786053 26923 raft_consensus.cc:3055] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:08.792755 26950 raft_consensus.cc:697] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:08.820502 25315 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:08.770051 26906 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:08.833457 25240 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 13:08:08.834659 25313 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:08.836686 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:08.836906 26771 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:08.829101 25358 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:08.884948 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 55971.7us
I20260709 13:08:08.854770 25359 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:08.847148 26914 consensus_queue.cc:1175] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "2329e428ebbf484fa3ea9452133df567" responder_term: 3 status { last_received { term: 2 index: 2 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:08:08.855860 26743 consensus_queue.cc:260] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.857271 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:08.857898 26946 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:08.828126 26795 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:08.862021 25387 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:08.882424 26950 consensus_queue.cc:237] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.883324 26846 raft_consensus.cc:3055] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:08.880365 26906 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:08.877715 25386 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:08.877506 26923 raft_consensus.cc:740] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:08.880810 25388 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: []
W20260709 13:08:08.903388 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 48733.2us
I20260709 13:08:08.908313 25389 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:08.908741 26970 consensus_queue.cc:1059] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:08.902994 26875 consensus_queue.cc:1059] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260709 13:08:08.923087 26923 consensus_queue.cc:260] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:08.923799 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:08.924114 26923 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:08.905999 26846 raft_consensus.cc:740] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:08.929322 26883 consensus_queue.cc:1059] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:08:08.902382 26795 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:08.910115 26743 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:08.903138 26792 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:08.905576 26879 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:08.929999 26846 consensus_queue.cc:260] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:08.906782 26810 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:08:08.935519 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:08.932718 26795 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:08.937662 26879 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:08.979414 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 199610us
I20260709 13:08:08.947086 26846 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:08:09.065761 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:09.065977 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 142248us
I20260709 13:08:08.987177 26910 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.007568 25318 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:09.077486 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:09.072520 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 66696.2us
W20260709 13:08:09.070468 26968 log.cc:927] Time spent T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
W20260709 13:08:09.074172 26977 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 13:08:09.000638 26954 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20260709 13:08:09.069507 26810 raft_consensus.cc:697] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:09.075685 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.075850 26943 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.073880 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:09.108561 26943 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.114643 26879 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:09.115235 26879 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:09.115696 26879 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.116199 26943 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:09.117654 26879 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:09.117596 26810 consensus_queue.cc:237] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.118350 26879 raft_consensus.cc:3055] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:09.120165 25284 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:09.081209 26910 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.121626 26943 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:09.118712 26879 raft_consensus.cc:740] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:09.122160 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:09.122766 25240 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:09.122830 26910 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:09.122936 26879 consensus_queue.cc:260] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:09.124107 25238 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:08:09.124505 26879 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:09.125332 26998 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:09.126724 26998 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:09.129158 26908 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:09.129618 25285 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:09.131016 26771 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:09.131337 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:09.131418 26771 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:09.132102 25238 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:09.132802 26999 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.133281 26999 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.108970 25313 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:09.135967 25909 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:09.136760 26978 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:09.137164 26978 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:09.137508 26978 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:09.137538 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:09.138458 25388 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:09.138999 25209 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:08:09.140241 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:09.132095 26771 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:09.139894 26954 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:09.171267 26906 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.172953 26999 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:09.177677 26795 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.191196 25386 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:09.157562 26908 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:09.195545 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 118177us
I20260709 13:08:09.197574 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:09.201053 25212 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:09.194933 26954 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:09.195133 26975 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.205549 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.206178 25314 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:09.207132 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:09.207540 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.208256 25315 raft_consensus.cc:2393] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 2: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:09.208977 26989 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.209612 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:09.209892 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.210096 26743 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:09.209458 26989 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.210702 26743 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.209451 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.213361 25318 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:09.207188 25317 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:09.212707 26989 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:09.215708 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:09.216457 25243 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:09.210191 25387 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:09.217541 25285 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:09.218189 25313 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:09.215154 26923 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:09.215000 26848 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.221376 26848 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.223408 26848 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:09.225346 25313 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:09.226152 26946 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:09.226538 26946 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:09.226878 26946 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:09.227002 26906 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:09.227761 26923 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:08:09.228499 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.228204 26975 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.229092 25317 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:08:09.229599 25941 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:09.230811 26975 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:09.200449 26908 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.232213 25908 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:09.233771 26908 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:09.234057 26910 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:09.234182 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:09.215831 26954 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:09.234807 25245 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:08:09.236690 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:09.200623 26999 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:09.238159 26999 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:09.238667 26999 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:09.243737 26795 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:09.234480 26910 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:09.247303 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.250857 26910 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:09.251262 25318 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:09.249169 25315 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:08:09.252665 26883 raft_consensus.cc:3055] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:09.253072 26883 raft_consensus.cc:740] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:09.249524 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:09.249222 26996 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.253939 26883 consensus_queue.cc:260] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.254259 26996 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.255017 26883 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:09.250335 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.257689 25317 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
W20260709 13:08:09.260743 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 45951.6us
I20260709 13:08:09.253947 25313 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:09.261984 25358 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:09.262946 25908 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:09.262951 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:09.264489 26846 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:09.264575 26914 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:09.265246 26914 raft_consensus.cc:697] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:09.266146 26914 consensus_queue.cc:237] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.268261 25908 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:09.271281 26976 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:09.271477 26972 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:09.256809 26996 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:09.272012 26972 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:09.272431 26972 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:09.272987 26996 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:09.274833 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:09.280409 26996 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:09.280611 25941 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:09.280865 26996 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:09.268373 26743 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:09.274942 25358 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:09.279245 26917 raft_consensus.cc:3055] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:09.283946 26917 raft_consensus.cc:740] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:09.285056 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.284909 26917 consensus_queue.cc:260] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.285768 25315 raft_consensus.cc:2368] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 2. Current term is 3.
I20260709 13:08:09.286479 26917 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:09.287794 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:09.288576 25315 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:09.291226 26846 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:09.291678 26846 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:09.294710 27002 log.cc:927] Time spent T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.132s	user 0.002s	sys 0.000s
I20260709 13:08:09.283593 26953 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:09.298578 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.299469 25313 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:09.305959 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:09.307834 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:09.310545 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:09.312633 26743 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.313342 26743 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.317919 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:09.318682 25389 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:09.320132 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:09.320788 25945 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
W20260709 13:08:09.328593 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 53720us
I20260709 13:08:09.330020 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:09.330654 25388 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:09.333293 25358 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:09.334962 26810 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.335242 26743 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:09.337654 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:09.337520 26743 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:09.339076 25244 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:09.332813 26795 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:09.343312 26795 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:09.343681 26795 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:09.335470 26810 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.345701 26810 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:09.343886 25359 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:09.347615 26810 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:09.346890 25358 leader_election.cc:400] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 2. Current term is 3.
I20260709 13:08:09.348448 25358 leader_election.cc:403] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:08:09.348357 25315 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:09.348039 26810 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:09.362538 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:09.362179 25287 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:09.364840 26792 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.361352 26989 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.412748 26976 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.374275 26944 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:08:09.414722 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:09.368480 26810 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:09.367924 27000 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:08:09.367950 26925 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.385778 26875 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:09.425777 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:09.431273 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 217683us
I20260709 13:08:09.385635 26997 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:09.428304 25143 catalog_manager.cc:5697] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: config changed from index 3 to 4, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New cstate: current_term: 1 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:08:09.435949 26997 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:09.438978 25143 catalog_manager.cc:5697] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 1 to 2, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 2 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:09.423242 26998 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.448028 25143 catalog_manager.cc:5697] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 1 to 2, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 2 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:09.433400 26775 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:09.447283 26997 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:09.475350 25941 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:09.476676 26996 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.479506 26996 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:09.480152 26996 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:09.480564 26996 raft_consensus.cc:697] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:09.483021 26972 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.423914 26989 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.436636 26970 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:09.473438 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:09.421137 26792 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.425199 26976 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.493557 25141 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39572 (request call id 286) took 4483 ms. Trace:
I20260709 13:08:09.441529 27004 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.437518 26925 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.468228 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:09.498821 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 73160.9us
I20260709 13:08:09.503530 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:09.504101 25314 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:09.439903 26875 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.509423 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.510027 25318 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:09.511041 26846 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.512137 25944 raft_consensus.cc:2393] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:09.515901 25245 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:09.491402 26944 raft_consensus.cc:697] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:09.499624 25388 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:09.492869 26775 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.516880 26852 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.553649 26879 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.557770 25314 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:09.498003 25141 rpcz_store.cc:276] 0709 13:08:05.002276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:05.002461 (+   185us) service_pool.cc:224] Handling call
0709 13:08:09.485622 (+4483161us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:08:08.625901 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:08:08.652168 (+ 26267us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:08:08.652202 (+    34us) write_op.cc:435] Acquired schema lock
0709 13:08:08.652221 (+    19us) tablet.cc:662] Decoding operations
0709 13:08:08.652568 (+   347us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:08:08.652620 (+    52us) write_op.cc:641] Partition lock acquired for write op
0709 13:08:08.652650 (+    30us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:08:08.652814 (+   164us) tablet.cc:701] Row locks acquired
0709 13:08:08.652831 (+    17us) write_op.cc:260] PREPARE: finished
0709 13:08:08.652960 (+   129us) write_op.cc:270] Start()
0709 13:08:08.653080 (+   120us) write_op.cc:276] Timestamp: P: 1783602488652922 usec, L: 0
0709 13:08:08.653098 (+    18us) op_driver.cc:348] REPLICATION: starting
0709 13:08:08.673623 (+ 20525us) log.cc:844] Serialized 599 byte log entry
0709 13:08:08.847713 (+174090us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b5400470ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cade28 00007f297a6e8581 00007f297a6e7d58 00007f297a6ec9f1 00007f297a6ec9a9 00007f297a6ec939 00007f297a6ec901 00007f297a6ebb2d 000000000038f354 000000000038f199
0709 13:08:09.228946 (+381233us) op_driver.cc:464] REPLICATION: finished
0709 13:08:09.413018 (+184072us) write_op.cc:301] APPLY: starting
0709 13:08:09.413098 (+    80us) tablet.cc:1367] starting BulkCheckPresence
0709 13:08:09.413226 (+   128us) tablet.cc:1370] finished BulkCheckPresence
0709 13:08:09.413245 (+    19us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:08:09.413470 (+   225us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:08:09.413611 (+   141us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:08:09.413657 (+    46us) write_op.cc:312] APPLY: finished
0709 13:08:09.413840 (+   183us) log.cc:844] Serialized 23 byte log entry
0709 13:08:09.413993 (+   153us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:09.414073 (+    80us) write_op.cc:454] Released schema lock
0709 13:08:09.414108 (+    35us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":92842,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":183823,"apply.run_cpu_time_us":1399,"apply.run_wall_time_us":1403,"num_ops":1,"prepare.queue_time_us":270255,"prepare.run_cpu_time_us":20307,"prepare.run_wall_time_us":228921,"raft.queue_time_us":251294,"raft.run_cpu_time_us":9782,"raft.run_wall_time_us":210355,"replication_time_us":547173,"spinlock_wait_cycles":26414336,"thread_start_us":450151,"threads_started":4,"wal-append.queue_time_us":231911}]]}
I20260709 13:08:09.554001 25389 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:09.565032 25318 raft_consensus.cc:1275] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 13:08:09.559520 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 72638.8us
I20260709 13:08:09.537396 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.567909 25313 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:09.568861 26906 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.569396 26906 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.570394 26976 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:09.571372 26906 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:09.563421 25358 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:09.505708 26771 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.578002 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:09.578712 25245 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:08:09.496428 26970 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.500686 27004 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.582337 27018 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:09.584188 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:09.559618 26852 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.587025 26775 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:09.567934 26846 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:09.520434 26978 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.600536 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:09.601198 25393 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:08:09.469729 26998 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.603484 26998 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:09.604480 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:09.565073 26879 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.611447 26910 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.611950 26879 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:09.560925 26944 consensus_queue.cc:237] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.581086 25909 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:09.594000 26917 consensus_queue.cc:1048] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:08:09.622066 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:09.622839 25243 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 3: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:09.625530 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:09.595121 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:09.508056 26996 consensus_queue.cc:237] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.627297 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.627923 25318 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:09.594852 26936 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:09.620044 27018 raft_consensus.cc:697] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:09.629658 26936 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.591791 25911 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:09.617239 26925 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:09.619047 26875 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:09.626362 25245 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:09.630580 27018 consensus_queue.cc:237] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.596101 25391 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:08:09.631841 26846 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:09.632802 25285 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:09.633832 26775 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:09.634227 26775 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:09.590134 26743 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:09.634577 26775 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:08:09.636457 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 55462.2us
I20260709 13:08:09.638180 25911 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:09.637099 25209 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:09.596164 26883 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.594707 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:09.600559 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:09.613481 26989 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:09.665335 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.667202 26953 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.667609 25314 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 1.
I20260709 13:08:09.670200 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:09.670804 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:08:09.671020 25314 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:09.604893 26999 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.672200 26953 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.677721 26906 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.678005 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:09.693171 25944 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:09.658195 27005 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:09.658386 26998 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:09.595786 26795 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.677778 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:09.703213 25321 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:09.681453 25245 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:08:09.659933 26925 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:09.663829 27014 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:09.719885 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 124831us
W20260709 13:08:09.708177 25129 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet eda233ace2a24b379fef320a5f610c7a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 46de6919f85341c6a9b574dff112e35a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:09.660964 27019 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:09.722451 27019 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:09.722842 27019 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:09.723179 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:09.723275 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.723910 25938 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:09.723941 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:09.724687 25945 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:09.687844 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:09.710770 26953 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:09.723978 25315 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:09.726400 25314 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:08:09.727999 26917 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:08:09.729458 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:09.729739 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:09.730121 25941 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:09.730834 26848 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:09.729686 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.736742 25314 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:08:09.710580 26771 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:09.715184 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:09.715847 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:09.774453 25245 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:09.778075 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:09.778532 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:09.778975 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:09.779314 25395 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:09.716331 26810 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.700579 26792 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:09.662177 26883 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.702754 26998 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:09.662089 26954 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.719041 26925 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:09.701082 27005 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:09.731415 26848 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.711773 26852 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:09.774097 25239 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:09.789741 25937 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:08:09.791417 26775 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.792973 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:09.793783 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:09.794003 26810 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:09.784080 26748 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.785231 26875 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:09.794626 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:09.795694 26883 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:09.796271 25240 raft_consensus.cc:2368] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 2. Current term is 3.
I20260709 13:08:09.796692 26954 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:09.794524 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.799405 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:09.787278 25321 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:09.783708 26743 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:09.800875 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:09.801265 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:09.801318 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:09.805444 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:09.776731 26846 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:09.793872 27010 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.256s	user 0.002s	sys 0.000s
I20260709 13:08:09.773864 26996 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.814666 25312 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:09.774091 27018 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.806622 25285 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:09.801894 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.700860 26906 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.776700 27014 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:09.955648 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 149127us
I20260709 13:08:09.956291 26953 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:09.956835 26953 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:09.808894 26775 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:09.957295 26953 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:09.884742 26914 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.959290 26914 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.961484 26914 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:09.812933 25244 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:08:09.962234 25314 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
W20260709 13:08:09.808109 26946 consensus_meta.cc:287] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.581s	user 0.003s	sys 0.004s
I20260709 13:08:09.963362 26946 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:09.964445 25358 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:09.811650 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:09.811576 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:09.813587 25316 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:09.828310 25245 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:09.815629 25395 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:08:09.934262 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:09.973155 25239 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:09.975903 25321 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:09.965071 27021 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:10.002123 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:09.982632 27018 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.003844 25244 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.004987 26906 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.005434 27018 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:09.805245 26917 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:09.977031 25285 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:09.805332 27005 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:09.788416 26998 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:09.814299 26743 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:10.008155 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:10.008746 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.806798 26848 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:09.809342 26999 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.016130 25321 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:10.017660 25358 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:09.977466 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:09.809286 26988 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.965274 27024 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.882216 25386 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:09.810343 26875 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:09.810660 25238 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:09.813398 26925 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:09.937515 26771 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:09.811391 26795 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:09.981787 26883 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:10.015630 25393 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:09.810209 26970 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:09.939406 27019 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.016028 25314 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:09.884529 26852 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:10.018676 25319 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:09.965521 26946 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:09.809875 25940 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:08:09.812645 25313 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:10.006634 26775 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:10.021939 25284 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:10.002269 26996 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:09.805117 27000 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 2. Current term is 3.
I20260709 13:08:09.810179 26748 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.026155 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:10.053356 25245 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:10.031533 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:10.031639 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:10.030393 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:10.055644 25944 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:10.010133 27021 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.038194 25212 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:10.024986 27024 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.040663 26797 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:10.019490 26743 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.019997 26999 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:10.021941 27013 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:09.939203 26997 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.005604 26846 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.026633 26875 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.047341 25358 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:10.013837 26917 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.036679 26953 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.041913 26883 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:10.058219 25312 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:10.074265 25244 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:10.055737 26772 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:10.041136 26925 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.060608 25937 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.083560 25936 raft_consensus.cc:2393] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 3: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:10.084204 27021 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:10.047340 26852 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.087185 27024 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:10.042811 26771 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.088490 25239 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:10.062217 26970 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:10.088757 26999 raft_consensus.cc:697] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:10.091785 26846 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.094059 27013 raft_consensus.cc:697] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:10.096163 25316 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
W20260709 13:08:10.098344 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 51083.9us
I20260709 13:08:10.099684 25395 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:10.132720 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:10.133463 25212 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:10.109982 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 89282.7us
I20260709 13:08:10.134001 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:10.134627 25316 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:08:10.022956 26988 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.114023 25285 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:10.055377 26775 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.110201 27018 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:10.071838 26797 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.114189 26883 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:10.129077 26748 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.129992 26996 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.133576 25358 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:10.111985 25245 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:10.134450 25909 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:10.137487 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:10.162261 25244 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 2: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:10.146288 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:10.162554 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:10.162700 25212 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:10.146687 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:10.146948 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:10.162724 25314 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.163386 25315 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:08:10.163479 25316 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:08:10.147629 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:10.164381 25321 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:08:10.164960 27013 consensus_queue.cc:237] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.165453 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:10.166023 25386 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:10.166497 27004 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.167608 26976 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:10.168116 26976 raft_consensus.cc:697] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:10.168562 27000 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:10.168190 26999 consensus_queue.cc:237] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.169092 27000 raft_consensus.cc:697] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:10.169052 26976 consensus_queue.cc:237] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.170504 26775 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.170156 27000 consensus_queue.cc:237] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.143334 27018 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:10.171921 27018 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:10.148366 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:10.149466 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:10.173854 25312 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:10.149618 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:10.175019 25239 raft_consensus.cc:2368] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3.
I20260709 13:08:10.149492 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:10.175894 25944 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:08:10.143824 26989 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:10.177585 26944 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:10.177685 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:10.178323 25944 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:08:10.178175 26944 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:10.175050 26883 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.180286 26944 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.168972 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:10.125206 26970 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:10.153311 26852 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.154717 26917 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.155643 27031 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:10.156790 26797 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.158418 26953 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.181600 26883 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:10.181668 26989 raft_consensus.cc:697] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:10.194692 27014 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.191259 27031 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:10.195597 27031 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.195294 26875 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.197340 26875 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.197438 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:10.197886 27030 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:10.198047 25245 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:10.198395 27030 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.199891 26989 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.200493 27030 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.201521 26914 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:10.201910 26914 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:10.202235 26914 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.202986 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:10.197445 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:10.176765 26998 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.136886 27024 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:10.183216 25358 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:10.184538 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:10.187165 25314 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:10.264516 25239 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:10.187937 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:10.265005 25244 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:10.265393 25240 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:10.148841 25908 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:10.266343 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 117639us
I20260709 13:08:10.190552 25285 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:10.192916 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:10.266964 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:10.267395 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 76959us
I20260709 13:08:10.267483 25241 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:10.267479 25245 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.268987 25285 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:10.270115 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:10.270889 25243 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:10.200619 25238 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:10.263504 27005 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.273497 26925 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.275113 27005 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.276638 25908 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:10.277521 25908 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:10.278242 26975 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:10.278450 26988 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.278538 27004 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:10.280378 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:10.280937 25243 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:08:10.190050 26970 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.290856 25245 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:10.296945 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:10.297346 25388 raft_consensus.cc:2393] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:10.279031 27004 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:10.298341 27004 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.298856 25937 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:10.300654 25389 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:08:10.301339 26775 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:10.301424 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:10.301879 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:10.302544 25238 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:10.280153 26842 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:10.181982 26743 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.265805 27024 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:10.307155 25240 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:10.309929 27015 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:10.313508 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:10.318596 25314 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.321460 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:10.278625 26975 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:10.322100 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 157455us
I20260709 13:08:10.329397 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:10.330165 25391 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:10.325302 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:10.326465 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:10.331218 25392 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:10.333735 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 129825us
I20260709 13:08:10.335685 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:10.336273 25394 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:10.336882 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:10.337618 25394 raft_consensus.cc:2410] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260709 13:08:10.338689 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:10.339608 25316 raft_consensus.cc:2368] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4.
I20260709 13:08:10.339763 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:10.340453 25317 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:10.340551 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:10.301971 26775 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:10.308589 26797 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:10.304942 27014 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:10.304746 26842 raft_consensus.cc:697] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 4 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:10.357340 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:10.342537 26775 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.358047 26842 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.297008 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:10.360911 25237 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:10.341002 25242 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:08:10.341979 25909 leader_election.cc:400] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3.
I20260709 13:08:10.361830 25909 leader_election.cc:403] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:08:10.362568 27018 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.363061 25909 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:10.364030 25395 raft_consensus.cc:2393] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:10.364637 27018 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:10.365309 26917 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:10.365617 27005 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3.
I20260709 13:08:10.365830 26917 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.366442 27005 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:10.366937 27005 raft_consensus.cc:697] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 5 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:10.367806 26917 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.368654 27014 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:10.317518 27024 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.349677 25392 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:10.344617 25359 leader_election.cc:400] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 2. Current term is 3.
I20260709 13:08:10.347100 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:10.343293 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:10.382933 25359 leader_election.cc:403] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:08:10.337357 27000 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:10.352229 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:10.322393 27019 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.353529 26997 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.354377 26743 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.354730 26998 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.356541 27031 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.335122 26975 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:10.341257 25315 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:10.324023 26883 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:10.369800 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:10.391454 25240 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:10.371243 25285 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:10.328717 27015 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.373306 25209 leader_election.cc:400] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4.
I20260709 13:08:10.379547 25314 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:10.379683 25391 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:10.394130 25209 leader_election.cc:403] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:08:10.383236 25317 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.341365 26834 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:10.384927 25321 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:10.383791 25358 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:10.396057 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:10.386811 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 224157us
I20260709 13:08:10.373986 26797 raft_consensus.cc:697] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:10.382838 25936 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:10.372692 25241 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:08:10.392769 25285 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:10.369408 26996 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 13:08:10.367843 27005 consensus_queue.cc:237] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.404161 27014 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.432101 27024 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.405076 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:10.433596 25945 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.405499 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:10.405800 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:10.406498 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:10.407658 25939 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:08:10.408488 26743 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:10.447355 26743 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:10.447781 26743 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.454085 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:10.409693 25285 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:10.410856 26953 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:10.411578 26997 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:10.464121 26953 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:10.464637 26953 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.384703 27000 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:10.413447 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:10.413801 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:10.465540 27000 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.465819 25391 raft_consensus.cc:2368] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4.
I20260709 13:08:10.414919 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:10.466794 25388 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:10.427104 26998 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:10.469619 26998 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:10.470094 26998 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.473336 26748 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:10.473920 26748 raft_consensus.cc:697] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:10.473062 27000 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.474972 26748 consensus_queue.cc:237] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.477312 27030 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:10.477941 27030 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.482595 26997 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 13:08:10.484419 27036 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 13:08:10.486123 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:10.486887 25939 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.412621 26834 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.418200 26910 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:10.402454 26857 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:10.418394 26797 consensus_queue.cc:237] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.423313 26775 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.416945 25242 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:10.395642 26883 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.434857 25390 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
W20260709 13:08:10.435803 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 89973.1us
I20260709 13:08:10.436019 25317 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:10.388067 26995 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 2. Current term is 3.
I20260709 13:08:10.436869 25243 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:10.439285 26771 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.440413 26917 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:10.440934 26795 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:08:10.441617 27024 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.443563 25945 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:10.444762 25212 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:10.456634 25941 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.457091 25938 raft_consensus.cc:2393] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 3: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:08:10.461436 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 51833.1us
I20260709 13:08:10.461941 26743 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.538688 26857 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:10.489929 25940 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:08:10.465785 25392 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.581838 26857 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.580996 25941 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:10.446146 27004 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.490912 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:10.608683 25393 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.537918 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:10.558460 25908 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:10.610287 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 51906.9us
W20260709 13:08:10.554818 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 163376us
W20260709 13:08:10.431257 27012 log.cc:927] Time spent T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.953s	user 0.013s	sys 0.000s
I20260709 13:08:10.621483 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:10.622303 25243 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:10.624722 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:10.566079 25209 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:08:10.627296 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 61292.6us
I20260709 13:08:10.628871 25209 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:08:10.581302 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 136615us
I20260709 13:08:10.387264 27018 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:10.600272 25392 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:10.568061 27041 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4.
W20260709 13:08:10.632642 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 54544.6us
I20260709 13:08:10.633423 25285 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:10.582561 26999 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:10.583142 27000 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.642495 25284 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:10.643427 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:10.630384 25209 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:10.617125 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:10.463503 27015 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:10.634905 25212 leader_election.cc:400] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4.
I20260709 13:08:10.583547 26923 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:10.644546 25244 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:10.571240 26775 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.647075 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:10.583384 26743 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.655956 25943 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 3: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:10.656699 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:10.620029 26946 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:10.658915 25243 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:10.632215 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:10.642359 26846 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:10.633912 26925 raft_consensus.cc:2764] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 13:08:10.645828 26997 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:10.645601 26988 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:10.656097 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:10.632119 26852 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260709 13:08:10.649435 26923 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:10.703372 25243 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:08:10.643627 26914 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.650192 27024 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:10.571899 26917 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:10.669116 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:10.655135 27015 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:10.663762 27032 raft_consensus.cc:2764] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 13:08:10.662532 26936 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.782822 25243 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:10.669118 27000 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:10.645641 26944 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260709 13:08:10.652617 26995 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:10.669337 26775 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:10.679405 27030 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:10.710072 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 63101.3us
I20260709 13:08:10.649488 26999 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.821707 26775 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.706032 26923 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.708078 26797 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:10.737602 27015 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.698662 26846 raft_consensus.cc:697] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:10.701794 26997 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.733620 25237 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 3: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:10.701336 26743 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:10.831995 26743 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.825941 26797 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:10.833384 26797 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:10.759657 25393 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:10.760987 26834 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.673290 26857 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.836702 26857 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.761682 27005 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:10.837409 27018 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:10.838090 27018 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.838363 26846 consensus_queue.cc:237] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.762635 26917 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.842991 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:10.843616 26972 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.844020 26995 raft_consensus.cc:697] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:10.763360 27004 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.764909 25238 raft_consensus.cc:2393] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 3: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:10.697818 26988 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:10.764900 26914 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.770834 26970 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.706491 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:10.773046 26883 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.707294 26925 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:10.673594 26946 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.698815 26772 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:10.776794 25939 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:10.762180 27031 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.869033 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:10.869679 25392 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.850658 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:10.870584 25316 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.852926 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:10.871407 25317 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:10.768906 26975 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.854525 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:10.823478 26852 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:10.837404 27005 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.855301 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:10.852052 26988 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:10.844924 26995 consensus_queue.cc:237] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.826030 27030 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:10.899210 26857 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:10.898981 27030 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.899691 26857 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.817610 27000 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:10.859843 26772 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:10.860446 26743 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.845901 26944 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:10.781081 26953 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.867920 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:10.869854 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:10.927887 25393 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:10.873925 25241 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.878619 26914 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:10.875424 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:10.878783 25392 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:10.881498 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:10.882161 25319 raft_consensus.cc:2393] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 3: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:10.883855 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:10.881453 26852 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.901573 26999 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.877573 26925 raft_consensus.cc:697] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:10.885924 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:10.886451 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:10.924654 26772 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.888656 25359 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:11.039281 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 150934us
I20260709 13:08:11.040153 25359 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:11.040982 25359 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:11.041496 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:10.889477 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:10.891289 25129 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet eda233ace2a24b379fef320a5f610c7a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 46de6919f85341c6a9b574dff112e35a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:10.896735 26997 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:11.042850 25314 raft_consensus.cc:2393] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:11.042714 26995 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:11.043093 26997 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:11.043576 26997 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:11.043603 26743 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:11.043895 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:11.042768 25129 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 151521us
I20260709 13:08:10.897812 25386 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:08:11.044886 26995 raft_consensus.cc:697] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:11.045778 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:11.045895 26995 consensus_queue.cc:237] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:11.046765 25386 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 3: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:10.916072 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:11.047336 25387 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:11.049185 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:10.919523 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:10.925323 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:11.050366 25321 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:08:11.050367 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 130935us
I20260709 13:08:10.877398 26917 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:11.053424 25212 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:10.931638 25320 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.877369 25941 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:10.884439 26998 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.942533 25315 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:10.887075 25312 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:08:10.933631 26914 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:10.948036 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:10.944276 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:10.906735 27030 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.932185 26944 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.941843 25388 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:10.916424 26771 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260709 13:08:10.907428 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:10.977469 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:10.977669 25241 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:10.978612 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:10.978878 26953 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.906524 27051 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260709 13:08:10.980566 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:10.980957 26775 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.981747 26978 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.981859 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:10.982342 25389 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:08:10.983916 26970 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.966706 25318 raft_consensus.cc:2368] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3.
I20260709 13:08:10.983453 26797 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.984375 27031 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:10.985317 27005 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:10.985745 27000 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:10.987021 26946 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.988340 26975 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:10.989513 27032 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:10.990048 26923 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:10.990675 27052 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:10.991079 26852 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.965461 26925 consensus_queue.cc:237] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:11.134454 27030 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:11.001881 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:11.008071 26857 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:10.980592 27015 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:11.009475 27055 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.135s	user 0.000s	sys 0.003s
I20260709 13:08:11.010848 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:11.014505 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:11.016188 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:11.022933 27018 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:11.028465 25143 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39596 (request call id 282) took 1825 ms. Trace:
I20260709 13:08:11.138913 27024 raft_consensus.cc:697] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:10.943604 25313 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:11.038265 25316 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:11.177294 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:11.042196 25242 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:11.044466 25314 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:11.044379 26743 raft_consensus.cc:697] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:11.046608 25244 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:08:10.917222 25944 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:10.905578 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:11.179382 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:11.077805 25212 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:11.180068 25244 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
W20260709 13:08:11.180307 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 102631us
I20260709 13:08:11.093010 25387 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:11.103662 25393 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 3: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:11.112040 25321 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:11.116657 25315 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:11.181540 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:11.078713 26997 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:11.120344 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 220579us
I20260709 13:08:11.123143 25942 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:08:11.123834 25358 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:11.079170 26795 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:11.190250 25911 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:11.191294 25287 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:11.191730 26852 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:11.192212 26852 raft_consensus.cc:697] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:11.126816 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:11.193006 26852 consensus_queue.cc:237] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.085801 27041 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:11.194232 27041 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:11.194604 27041 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:11.194871 26857 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:11.131444 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:11.195884 25312 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 3: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:11.174998 26988 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:11.134677 25246 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:11.134513 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:11.197319 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:11.126309 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:11.197983 25319 raft_consensus.cc:2393] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:11.198124 26797 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:11.199018 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:11.199762 25393 raft_consensus.cc:2393] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 3: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:11.117594 27053 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:11.142671 25285 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:11.125371 25939 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:11.136646 25237 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:11.205678 27051 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:08:11.205960 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:11.134721 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:11.206779 25313 raft_consensus.cc:2368] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3.
I20260709 13:08:11.206984 25938 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:08:11.207407 27051 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:11.207549 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:11.207937 27051 raft_consensus.cc:697] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:11.208271 25313 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:11.109234 26917 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.208817 27051 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.208843 26743 consensus_queue.cc:237] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:11.210867 26917 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:11.130358 25936 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
W20260709 13:08:11.001099 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 370787us
W20260709 13:08:11.207047 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 64467.1us
I20260709 13:08:11.174294 26998 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:11.220912 25942 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:11.168781 25239 raft_consensus.cc:2368] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3.
I20260709 13:08:11.030248 25141 catalog_manager.cc:5697] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 2 to 3, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 3 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:11.222056 25284 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:11.222289 25141 catalog_manager.cc:5697] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:11.222970 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:11.171563 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:11.225028 25314 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:11.225636 25392 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 3: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:11.172690 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:11.173419 25310 tablet_service.cc:1558] Processing DeleteTablet for tablet 17584a19690d4010851d65378cfcbafa with delete_type TABLET_DATA_TOMBSTONED (TS 2329e428ebbf484fa3ea9452133df567 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36848
I20260709 13:08:11.226042 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:11.145952 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:11.170193 25390 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 1.
I20260709 13:08:11.177947 25316 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:11.179502 25394 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:11.230490 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:11.197934 26988 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:11.237741 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:11.169926 27030 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.182822 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:11.242496 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:11.243052 25315 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:11.247871 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:11.248955 25316 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:11.185554 25321 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:11.185932 27039 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:11.251967 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:11.171037 25143 rpcz_store.cc:276] 0709 13:08:09.203497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:09.205363 (+  1866us) service_pool.cc:224] Handling call
0709 13:08:09.389493 (+184130us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b5400470ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1c090 00007f2976cbaf3a 00007f297ca711cc 00007f297cb6f37e 00007f297b0302a4 00007f297b030232 00007f297b030161 00007f297b0300dc 00007f297b02f392 00007f2974fed48c 00007f2974fec8c6
0709 13:08:11.028349 (+1638856us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:08:09.615020 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:08:09.667636 (+ 52616us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:08:09.667675 (+    39us) write_op.cc:435] Acquired schema lock
0709 13:08:09.674125 (+  6450us) tablet.cc:662] Decoding operations
0709 13:08:09.929397 (+255272us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:08:09.973632 (+ 44235us) write_op.cc:641] Partition lock acquired for write op
0709 13:08:09.981705 (+  8073us) tablet.cc:685] Acquiring locks for 3 operations
0709 13:08:09.982124 (+   419us) tablet.cc:701] Row locks acquired
0709 13:08:10.001685 (+ 19561us) write_op.cc:260] PREPARE: finished
0709 13:08:10.005448 (+  3763us) write_op.cc:270] Start()
0709 13:08:10.005580 (+   132us) write_op.cc:276] Timestamp: P: 1783602490005399 usec, L: 0
0709 13:08:10.005609 (+    29us) op_driver.cc:348] REPLICATION: starting
0709 13:08:10.041287 (+ 35678us) log.cc:844] Serialized 1298 byte log entry
0709 13:08:10.345527 (+304240us) op_driver.cc:464] REPLICATION: finished
0709 13:08:10.595522 (+249995us) write_op.cc:301] APPLY: starting
0709 13:08:10.638196 (+ 42674us) tablet.cc:1367] starting BulkCheckPresence
0709 13:08:10.658644 (+ 20448us) tablet.cc:1370] finished BulkCheckPresence
0709 13:08:10.680235 (+ 21591us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:08:10.737593 (+ 57358us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:08:10.748437 (+ 10844us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0709 13:08:10.748492 (+    55us) write_op.cc:312] APPLY: finished
0709 13:08:10.760436 (+ 11944us) log.cc:844] Serialized 35 byte log entry
0709 13:08:10.760656 (+   220us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:10.760768 (+   112us) write_op.cc:454] Released schema lock
0709 13:08:10.760833 (+    65us) write_op.cc:341] FINISH: Updating metrics
0709 13:08:10.761277 (+   444us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:10.761309 (+    32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":11458,"reported_tablets":7,"spinlock_wait_cycles":396601344,"child_traces":[["op",{"apply.queue_time_us":215287,"apply.run_cpu_time_us":3611,"apply.run_wall_time_us":196515,"num_ops":3,"prepare.queue_time_us":62663,"prepare.run_cpu_time_us":5143,"prepare.run_wall_time_us":454408,"replication_time_us":334761,"thread_start_us":46702,"threads_started":2}]]}
I20260709 13:08:11.197412 25241 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:11.149225 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:11.198040 25246 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 2: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:11.199476 26914 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:11.141477 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:11.203925 25320 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:11.163689 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:11.257790 25395 raft_consensus.cc:2368] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3.
I20260709 13:08:11.258140 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:11.257894 25318 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:11.219481 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:11.232604 26989 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:11.147063 27031 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.221424 25944 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:11.149466 27032 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.156647 27052 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.226949 25314 raft_consensus.cc:2393] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 2: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:11.224356 25141 catalog_manager.cc:5697] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 1 to 2, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 2 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:11.063916 25317 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:11.284593 27052 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:11.286089 25317 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:08:11.284637 25141 catalog_manager.cc:5697] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 2 to 3, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 3 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:11.288159 25317 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:11.231104 25394 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:11.231688 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:11.290369 25393 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:11.182133 25387 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:11.203579 26970 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:11.238656 25312 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:11.180989 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:11.239926 25908 leader_election.cc:400] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3.
I20260709 13:08:11.297844 25908 leader_election.cc:403] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:08:11.223759 26946 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:11.256889 25389 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 3: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:11.261060 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:11.261276 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:11.302809 25243 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:11.237741 26748 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:11.312134 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:11.312462 25393 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:11.312309 26748 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.313474 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:11.314014 25241 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:11.212872 27053 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.257475 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:11.232498 25937 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:11.263458 25386 raft_consensus.cc:2393] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 3: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:11.225000 26772 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.230316 26771 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260709 13:08:11.277263 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:11.324035 25316 raft_consensus.cc:2393] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 3: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:11.253755 26988 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:11.258919 27024 consensus_queue.cc:237] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.278434 25237 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:11.278882 26944 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:11.279460 26989 raft_consensus.cc:697] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:11.252341 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:11.280224 26997 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:11.282078 25388 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:11.298380 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:11.341009 25244 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:11.306556 27052 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:11.304571 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:08:11.305662 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 68649.2us
I20260709 13:08:11.306133 25387 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:11.319052 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:11.347285 25908 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:11.347841 25318 raft_consensus.cc:2368] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 2. Current term is 3.
I20260709 13:08:11.348326 25908 leader_election.cc:400] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3.
I20260709 13:08:11.348658 25908 leader_election.cc:403] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 13:08:11.322077 25320 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:11.322808 25245 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:11.327055 25285 leader_election.cc:400] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3.
I20260709 13:08:11.349336 25908 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:11.349547 25285 leader_election.cc:403] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:08:11.334303 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 152327us
W20260709 13:08:11.335101 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 211366us
I20260709 13:08:11.350674 25359 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:11.352607 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:11.340265 25241 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:11.316601 26975 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3.
I20260709 13:08:11.298889 25391 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:08:11.345263 25239 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:11.359392 25392 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:08:11.257208 27039 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.361353 25285 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:11.364455 25237 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:11.365566 26772 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:11.366053 27073 log.cc:927] Time spent T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20260709 13:08:11.366171 26944 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:08:11.366849 26936 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:11.367236 26989 consensus_queue.cc:237] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.368465 27032 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:11.371498 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:11.372037 25945 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:08:11.372323 25359 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:11.375797 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:11.376125 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:11.376946 25391 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:11.376787 25387 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:11.364856 26748 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:11.337893 26795 raft_consensus.cc:697] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:11.383306 25212 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:11.384239 26917 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:11.384076 26795 consensus_queue.cc:237] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.384516 25212 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:11.385430 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:11.313874 26946 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:11.386052 25386 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:11.387125 26996 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:11.387493 26946 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:11.367261 26936 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:11.382632 26923 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:11.360860 26848 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3.
I20260709 13:08:11.360648 26834 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:11.384620 26917 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:11.379789 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:11.369634 26910 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:11.360669 25242 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 2: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:11.389182 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:11.392093 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:11.407810 25393 raft_consensus.cc:2393] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:11.393745 25942 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:11.390533 25212 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:11.378850 26999 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:11.384086 27031 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:11.387522 26996 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:11.398581 27015 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:11.398797 27075 tablet_replica.cc:333] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:08:11.403211 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:11.403440 25945 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:11.404582 25244 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:11.360713 26883 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:11.401861 26914 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:11.395792 25936 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 3: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:11.335078 26988 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.414098 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:11.365568 27053 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:11.421626 25909 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:11.425577 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:11.436861 25285 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:11.406111 26917 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:11.435438 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:11.402485 26936 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:11.437547 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:11.369602 27059 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.086s
I20260709 13:08:11.358803 27052 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.405910 26910 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:11.450939 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:11.413655 26999 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:11.433598 25287 leader_election.cc:400] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3.
I20260709 13:08:11.453557 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:11.474848 25387 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:11.416690 27058 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260709 13:08:11.475828 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:11.413774 26995 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:11.458869 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:11.460184 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:11.416500 26923 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:11.440691 25209 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:11.456753 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:11.434177 26996 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:11.416103 26834 raft_consensus.cc:697] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:11.401526 27041 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:11.405823 26953 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
W20260709 13:08:11.470379 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 56389.1us
I20260709 13:08:11.456164 26914 raft_consensus.cc:697] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:11.504102 27088 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:11.459671 26978 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260709 13:08:11.494685 25239 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:11.496351 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:11.460155 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:11.460939 26748 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:11.460031 26848 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:11.466641 26883 raft_consensus.cc:697] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:08:11.494009 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 53375.8us
I20260709 13:08:11.474639 25388 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:11.556366 25244 raft_consensus.cc:2368] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3.
I20260709 13:08:11.474366 26910 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:11.531369 25358 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:11.527521 25387 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:08:11.562914 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:11.563450 25909 leader_election.cc:400] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3.
I20260709 13:08:11.476472 26999 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:11.473950 27030 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:11.566484 25358 leader_election.cc:400] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 2. Current term is 3.
I20260709 13:08:11.530124 27088 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:11.577522 25358 leader_election.cc:403] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:08:11.578313 25240 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:11.581452 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:11.472913 27053 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:11.527192 25389 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:11.475760 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:11.571077 27018 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260709 13:08:11.584154 27088 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:11.587002 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:11.587395 27084 log.cc:927] Time spent T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:08:11.498000 26923 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:11.527522 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:11.590268 25358 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:11.567727 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:11.520761 26795 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260709 13:08:11.606999 26795 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:11.607105 25237 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:08:11.607429 26795 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:11.607869 26795 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:11.608471 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:11.609227 25390 raft_consensus.cc:2393] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 3: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:11.566470 27000 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:08:11.592873 27069 log.cc:927] Time spent T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.391s	user 0.000s	sys 0.005s
I20260709 13:08:11.562310 26748 raft_consensus.cc:697] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:11.735963 27054 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:11.567999 27015 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:11.736832 27015 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:11.736721 27054 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.737303 27015 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:11.740679 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:11.562038 26848 raft_consensus.cc:697] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:08:11.599229 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 172284us
I20260709 13:08:11.741811 25287 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:11.741951 26848 consensus_queue.cc:237] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.742909 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:11.743943 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:11.589355 26998 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:08:11.604539 27039 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:11.594492 26834 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.670733 26988 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:11.669690 27052 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:11.674263 27041 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:11.679248 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:11.642732 26883 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.569654 26995 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260709 13:08:11.597651 25242 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:11.597805 26923 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:11.690843 27077 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:11.708734 25388 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:08:11.787871 27077 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:11.790907 27077 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:11.713802 26999 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:11.782437 26988 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:11.792688 26988 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:11.793613 26999 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:11.794610 26988 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:11.706722 26914 consensus_queue.cc:237] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:11.756125 27063 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:11.797348 27063 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:11.797708 27063 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:11.719568 26944 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3.
W20260709 13:08:11.727599 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 116841us
I20260709 13:08:11.584352 26997 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:11.730124 25237 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:11.753315 27014 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:11.609985 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:11.730710 25387 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
W20260709 13:08:11.639043 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 48830.5us
I20260709 13:08:11.808347 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:11.810767 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:11.810967 25287 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:11.813895 26944 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:11.814404 26944 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:11.814792 26944 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:11.815788 25390 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:11.760351 26946 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:11.747383 26748 consensus_queue.cc:237] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:11.757718 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:11.764490 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:11.835131 25239 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:11.778810 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:11.779083 27015 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.733856 26917 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:11.724879 26996 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.734741 27088 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.786779 26910 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.730854 26936 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:11.818871 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:11.820647 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:11.821019 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:11.843850 25395 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:11.821295 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:11.822258 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:11.823141 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:11.844762 25319 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:11.824203 25359 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:11.824700 27059 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:08:11.747933 26795 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.825295 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:11.761660 26989 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:11.833469 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:11.830140 25285 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:11.836781 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:11.838477 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:11.845923 25391 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:11.825826 27054 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:11.834052 26946 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:11.853804 27060 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:11.856997 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:11.859671 25241 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:11.861725 25317 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:11.860906 25358 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:11.860289 25943 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:11.862416 25315 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:11.863601 25315 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:11.866797 25315 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:08:11.825032 27063 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.848521 25937 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:11.849669 26775 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:11.872843 25317 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:11.839787 27041 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260709 13:08:11.861434 27039 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:08:11.870955 26946 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:11.871299 27060 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:11.873554 27018 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:11.877350 27013 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:11.900238 25241 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:11.873625 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:11.883023 25237 raft_consensus.cc:2393] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:11.902171 25358 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:11.908097 26998 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 2. Current term is 3.
I20260709 13:08:11.873023 26797 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
W20260709 13:08:11.921979 27099 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 13:08:11.858425 27077 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260709 13:08:11.861582 26988 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260709 13:08:11.885787 26772 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:11.903529 27060 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:11.862185 26989 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:11.878293 26846 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 13:08:11.900815 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:11.881328 27053 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 13:08:11.992867 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:12.016911 25391 raft_consensus.cc:2368] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5.
I20260709 13:08:11.986362 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:11.925182 27018 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:12.027642 25395 raft_consensus.cc:1275] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:08:11.989809 25317 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:08:11.910655 27013 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:11.911375 27039 raft_consensus.cc:697] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 2 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:11.995814 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:11.992967 25911 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:12.053450 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 60624us
I20260709 13:08:12.054405 25911 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:11.923185 25388 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:12.047004 25393 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:12.027426 27063 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:12.047377 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:12.030198 25317 raft_consensus.cc:1275] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:08:12.060792 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:12.062712 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:11.991321 26917 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:12.063388 25317 raft_consensus.cc:2368] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5.
I20260709 13:08:12.070686 26925 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:12.025748 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:12.052699 27013 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:12.054811 27005 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:12.009449 26772 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:12.071281 26925 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:08:12.082002 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 208613us
I20260709 13:08:12.068713 26999 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:12.054939 26975 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 13:08:12.102380 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:12.080157 27015 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:12.071609 27058 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260709 13:08:12.082508 26936 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:12.089838 26996 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:12.090068 26910 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:12.092314 26848 raft_consensus.cc:2764] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
W20260709 13:08:12.120863 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 66517.8us
I20260709 13:08:12.125347 25911 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:12.093663 26925 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:12.092597 26978 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
W20260709 13:08:12.143428 27094 log.cc:927] Time spent T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.189s	user 0.000s	sys 0.001s
I20260709 13:08:12.121853 27056 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:12.099890 26999 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:12.167136 26999 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:12.173074 27005 consensus_queue.cc:1059] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:08:12.174294 27005 raft_consensus.cc:3055] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:12.174651 27005 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:12.175452 27005 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:12.176740 27005 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:12.177091 27056 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:12.179297 27056 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:12.179351 26944 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:12.137290 26834 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:12.184118 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:12.184749 25395 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:12.164567 27088 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:12.200911 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:12.203411 26944 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:12.227713 25386 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:12.227569 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:12.187930 26846 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260709 13:08:12.205878 26772 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:12.228497 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 543) took 1010 ms. Trace:
I20260709 13:08:12.181689 26910 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260709 13:08:12.228871 25938 rpcz_store.cc:276] 0709 13:08:11.217991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:11.224264 (+  6273us) service_pool.cc:224] Handling call
0709 13:08:12.227407 (+1003143us) spinlock_profiling.cc:243] Waited 4.35 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:12.228434 (+  1027us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10013568}
I20260709 13:08:12.123260 27104 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.080s
I20260709 13:08:12.216984 27056 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:12.239923 26925 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:12.245759 25209 leader_election.cc:400] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5.
I20260709 13:08:12.246306 25209 leader_election.cc:403] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 13:08:12.226625 26989 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:12.225862 25388 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:12.182184 27039 consensus_queue.cc:237] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:12.313150 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 223) took 1126 ms. Trace:
I20260709 13:08:12.313668 25392 rpcz_store.cc:276] 0709 13:08:11.187223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:11.289533 (+102310us) service_pool.cc:224] Handling call
0709 13:08:11.359102 (+ 69569us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:11.359724 (+   622us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:12.312864 (+953140us) spinlock_profiling.cc:243] Waited 69.3 ms on lock 0x7b54001216d0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:12.313059 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":159397248}
I20260709 13:08:12.292119 26954 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5.
I20260709 13:08:12.289525 26914 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260709 13:08:12.253396 27056 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:12.292771 26917 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260709 13:08:12.297973 26946 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:12.355511 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 237107us
I20260709 13:08:12.376322 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:12.379768 25320 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:12.330909 27060 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:12.389706 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 326471us
I20260709 13:08:12.346307 27005 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 13:08:12.392406 25287 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:12.397301 25358 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:12.371876 26875 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:12.370339 27063 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260709 13:08:12.398111 26995 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:12.417941 26995 raft_consensus.cc:697] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:12.418361 27063 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:12.418881 27063 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:12.418915 26995 consensus_queue.cc:237] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:12.404136 26875 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:12.411432 25388 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:12.415149 25314 raft_consensus.cc:1275] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:08:12.402254 26989 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:12.426450 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:12.434351 25315 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:08:12.416549 25242 consensus_meta.cc:287] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.587s	user 0.001s	sys 0.007s
W20260709 13:08:12.421207 26923 consensus_meta.cc:287] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.637s	user 0.011s	sys 0.000s
I20260709 13:08:12.430384 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:12.428236 25285 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:12.436678 25242 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:12.442124 26972 consensus_queue.cc:1059] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:08:12.442721 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 1180) took 1070 ms. Trace:
I20260709 13:08:12.443452 26972 raft_consensus.cc:3055] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:12.472488 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:12.428879 26875 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:12.445410 25314 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:12.483906 25240 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:12.488179 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:12.421095 27063 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:12.433758 27015 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:12.432348 26989 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:12.474987 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:12.443182 25242 rpcz_store.cc:276] 0709 13:08:11.372510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:11.451298 (+ 78788us) service_pool.cc:224] Handling call
0709 13:08:12.442642 (+991344us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:12.461511 26925 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:12.423470 27013 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:12.443950 26972 raft_consensus.cc:740] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:12.462697 26748 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:12.445006 26914 consensus_queue.cc:1048] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.122s
I20260709 13:08:12.558594 27111 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:12.568039 26970 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:12.576174 25237 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:12.617529 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:12.618423 25239 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:12.452242 26923 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:12.596446 26989 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:12.614660 25319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1660) took 1193 ms. Trace:
I20260709 13:08:12.629309 25319 rpcz_store.cc:276] 0709 13:08:11.420958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:11.421146 (+   188us) service_pool.cc:224] Handling call
0709 13:08:12.614447 (+1193301us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x7b5400044a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:12.614587 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":701691264}
I20260709 13:08:12.622253 25314 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:12.572963 27015 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:12.571141 27030 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:12.631132 25909 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.008s	sys 0.081s
I20260709 13:08:12.574373 26772 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260709 13:08:12.649564 25320 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 13:08:12.650274 25909 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.008s	sys 0.081s
W20260709 13:08:12.650725 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 597186us
I20260709 13:08:12.659888 25284 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:12.615902 27056 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:12.684890 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:12.685606 25944 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:12.671589 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:12.688802 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:12.691325 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:12.726606 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:12.616456 27111 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:12.746646 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:12.651866 26925 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:12.618841 26970 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:12.757326 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:12.766232 25392 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:12.664633 27109 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:12.770038 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:12.716081 27014 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:12.674500 25942 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:12.793960 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:12.737437 26883 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:12.738869 27121 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:12.743559 27060 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:12.738118 27058 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:12.795127 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:12.663260 27030 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:12.941962 25945 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 548) took 1487 ms (client timeout 1568 ms). Trace:
W20260709 13:08:12.942517 25945 rpcz_store.cc:269] 0709 13:08:11.454926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:12.794768 (+1339842us) service_pool.cc:224] Handling call
0709 13:08:12.941873 (+147105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:12.943904 27030 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:12.765554 27114 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 13:08:12.943806 27058 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:12.947257 27058 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:12.796869 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:12.887457 27090 log.cc:927] Time spent T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 13:08:12.798158 26954 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:12.954342 26954 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:12.955032 25388 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:12.760752 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:12.957993 25940 raft_consensus.cc:2410] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:08:12.715624 27063 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.069s
I20260709 13:08:12.959697 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:12.716459 26972 consensus_queue.cc:260] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:12.804001 27014 raft_consensus.cc:697] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:12.957015 26954 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:12.770978 26925 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:12.936360 25392 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:12.799835 25941 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:12.931062 25944 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
W20260709 13:08:12.925217 27115 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.275s	user 0.001s	sys 0.000s
I20260709 13:08:12.798290 27109 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:12.927322 26748 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:12.976909 26748 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:12.962520 27014 consensus_queue.cc:237] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:12.978039 27109 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:12.979094 26748 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:12.980448 27122 log.cc:927] Time spent T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.219s	user 0.000s	sys 0.000s
W20260709 13:08:12.981930 27124 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 13:08:12.983127 26946 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:12.983480 27119 log.cc:927] Time spent T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.255s	user 0.000s	sys 0.001s
W20260709 13:08:12.985139 27120 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 13:08:12.985445 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:12.986666 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:12.986783 27013 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:13.001019 25241 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:13.003338 25209 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:12.969192 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:12.877832 26883 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:12.963766 27058 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:13.118415 27058 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:13.118993 26995 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:13.120035 25359 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:13.120869 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 117626us
I20260709 13:08:12.960769 25388 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:13.130839 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:13.126747 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:12.965186 26925 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:13.122928 26914 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:13.138782 26995 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:12.961565 26972 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:12.877575 27121 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:12.986802 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:13.184015 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:13.245085 25241 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:08:13.148548 25209 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:13.193903 26797 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:13.200560 26914 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:13.202711 26923 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:13.211387 26883 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:13.214241 27058 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:13.215368 27015 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:13.224907 26970 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:13.226083 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 1751) took 1369 ms. Trace:
I20260709 13:08:13.228850 25242 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:13.229457 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:12.971321 27060 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:08:13.247485 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 99017.9us
I20260709 13:08:13.253895 26995 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:13.250099 26925 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:13.263787 25312 rpcz_store.cc:276] 0709 13:08:11.856354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:11.856528 (+   174us) service_pool.cc:224] Handling call
0709 13:08:11.877681 (+ 21153us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:12.079765 (+202084us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:12.079810 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:12.080346 (+   536us) log.cc:844] Serialized 26 byte log entry
0709 13:08:12.367097 (+286751us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:12.367150 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:12.367254 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:13.225788 (+858534us) raft_consensus.cc:1685] finished
0709 13:08:13.225818 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:13.226004 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":47039,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":3305,"thread_start_us":459695,"threads_started":2,"wal-append.queue_time_us":482730}
I20260709 13:08:13.273160 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:13.281072 26797 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:13.268446 27015 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:13.281792 25209 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:13.282612 26797 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:13.283573 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:13.284353 26797 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:13.284977 25209 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:13.285279 26914 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:13.285770 26797 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:13.286240 26797 raft_consensus.cc:697] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:13.287114 26797 consensus_queue.cc:237] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:13.288681 25209 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.68:39211 timed out after 1.568s (SENT) [suppressed 56 similar messages]
I20260709 13:08:13.290969 26946 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:13.291635 26946 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:13.293820 27121 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:13.294411 26946 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:13.294399 27060 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:13.296559 27060 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:13.297021 26944 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:13.297566 26944 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:13.299692 26944 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:13.300686 25141 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39572 (request call id 287) took 3683 ms. Trace:
I20260709 13:08:13.306262 25287 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:13.269981 25237 consensus_meta.cc:287] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.635s	user 0.006s	sys 0.005s
W20260709 13:08:13.320374 25284 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.659s	user 0.015s	sys 0.059s
W20260709 13:08:13.340440 25284 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.017s	sys 0.060s
W20260709 13:08:13.341106 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 679932us
I20260709 13:08:13.352505 25284 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:13.301077 25141 rpcz_store.cc:276] 0709 13:08:09.617296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:09.617632 (+   336us) service_pool.cc:224] Handling call
0709 13:08:13.300622 (+3682990us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:08:11.449322 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:08:11.587819 (+138497us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:08:11.587854 (+    35us) write_op.cc:435] Acquired schema lock
0709 13:08:11.587872 (+    18us) tablet.cc:662] Decoding operations
0709 13:08:11.673659 (+ 85787us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:08:11.673720 (+    61us) write_op.cc:641] Partition lock acquired for write op
0709 13:08:11.673747 (+    27us) tablet.cc:685] Acquiring locks for 4 operations
0709 13:08:11.674143 (+   396us) tablet.cc:701] Row locks acquired
0709 13:08:11.674161 (+    18us) write_op.cc:260] PREPARE: finished
0709 13:08:11.674308 (+   147us) write_op.cc:270] Start()
0709 13:08:11.674424 (+   116us) write_op.cc:276] Timestamp: P: 1783602491674265 usec, L: 0
0709 13:08:11.674447 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 13:08:11.677594 (+  3147us) log.cc:844] Serialized 1692 byte log entry
0709 13:08:11.736143 (+ 58549us) op_driver.cc:464] REPLICATION: finished
0709 13:08:11.954733 (+218590us) write_op.cc:301] APPLY: starting
0709 13:08:11.958478 (+  3745us) tablet.cc:1367] starting BulkCheckPresence
0709 13:08:12.050359 (+ 91881us) tablet.cc:1370] finished BulkCheckPresence
0709 13:08:12.063678 (+ 13319us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:08:12.064310 (+   632us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:08:12.064532 (+   222us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0709 13:08:12.064601 (+    69us) write_op.cc:312] APPLY: finished
0709 13:08:12.064887 (+   286us) log.cc:844] Serialized 41 byte log entry
0709 13:08:12.065111 (+   224us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:12.065257 (+   146us) write_op.cc:454] Released schema lock
0709 13:08:12.065312 (+    55us) write_op.cc:341] FINISH: Updating metrics
0709 13:08:12.065652 (+   340us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:12.065679 (+    27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1139,"reported_tablets":10,"spinlock_wait_cycles":9472,"child_traces":[["op",{"apply.queue_time_us":174137,"apply.run_cpu_time_us":2639,"apply.run_wall_time_us":141771,"num_ops":4,"prepare.queue_time_us":123634,"prepare.run_cpu_time_us":4345,"prepare.run_wall_time_us":254164,"replication_time_us":61533,"thread_start_us":174886,"threads_started":2}]]}
I20260709 13:08:13.352751 25287 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:13.319025 26989 consensus_meta.cc:287] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.665s	user 0.003s	sys 0.007s
I20260709 13:08:13.313490 26953 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:13.315389 27116 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:08:13.379348 25285 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.047s	sys 0.091s
I20260709 13:08:13.356366 26936 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:13.361985 26772 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:08:13.383380 25285 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.047s	sys 0.091s
I20260709 13:08:13.327841 25237 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:13.390614 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:13.370092 27108 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:13.391431 25241 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:13.369289 26989 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:13.396782 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:13.379544 26995 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:13.364140 25284 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:13.376993 26953 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:13.413722 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:13.381402 27116 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:13.418001 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:13.416074 27109 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:13.423144 25285 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:08:13.424786 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 60764us
I20260709 13:08:13.420401 26936 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:13.441211 25241 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:13.424566 25317 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:08:13.444944 26772 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:13.445737 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:13.447377 26989 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:13.414534 27108 raft_consensus.cc:697] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:08:13.472389 25941 consensus_meta.cc:287] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.503s	user 0.005s	sys 0.005s
I20260709 13:08:13.417770 26995 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:13.449448 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:13.474628 25241 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:13.476318 25241 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:13.422735 26944 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:13.447719 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1674) took 1024 ms. Trace:
I20260709 13:08:13.469365 26772 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:08:13.470813 27134 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260709 13:08:13.437876 27013 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:13.481341 27013 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:13.483757 27013 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:13.472810 25941 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:13.486742 26953 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:13.488147 25237 raft_consensus.cc:2393] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 3: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:13.478128 25313 rpcz_store.cc:276] 0709 13:08:12.423292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:12.429433 (+  6141us) service_pool.cc:224] Handling call
0709 13:08:12.439728 (+ 10295us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:12.440121 (+   393us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:12.440168 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:12.440703 (+   535us) log.cc:844] Serialized 26 byte log entry
0709 13:08:12.973365 (+532662us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:08:12.973418 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:12.976583 (+  3165us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:13.447455 (+470872us) raft_consensus.cc:1685] finished
0709 13:08:13.447479 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:13.447646 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1480,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":69,"thread_start_us":494034,"threads_started":1,"wal-append.queue_time_us":912665}
I20260709 13:08:13.474351 26936 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:13.495105 26936 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:13.495472 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:13.474160 27109 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:13.496739 25388 raft_consensus.cc:2393] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 3: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:13.498567 27109 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:13.468899 26988 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:13.503988 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:13.501099 27116 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:13.510303 25388 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:13.541400 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1196) took 1061 ms. Trace:
I20260709 13:08:13.541934 25244 rpcz_store.cc:276] 0709 13:08:12.480229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:12.482049 (+  1820us) service_pool.cc:224] Handling call
0709 13:08:12.482390 (+   341us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:12.482775 (+   385us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:12.482814 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:12.562212 (+ 79398us) log.cc:844] Serialized 26 byte log entry
0709 13:08:12.880140 (+317928us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:12.880210 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:12.880311 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:13.541176 (+660865us) raft_consensus.cc:1685] finished
0709 13:08:13.541196 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:13.541341 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7384,"raft.run_cpu_time_us":353,"raft.run_wall_time_us":41635,"spinlock_wait_cycles":1170304,"thread_start_us":277650,"threads_started":1,"wal-append.queue_time_us":733333}
I20260709 13:08:13.541298 25393 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:13.549620 26988 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:13.551935 26988 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:13.510848 26925 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:13.555075 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:13.480813 26944 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:13.555512 26925 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:13.473434 27108 consensus_queue.cc:237] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:13.552752 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 1749) took 1703 ms. Trace:
I20260709 13:08:13.574554 26795 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:13.579092 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1675) took 1155 ms. Trace:
I20260709 13:08:13.579519 25317 rpcz_store.cc:276] 0709 13:08:12.425201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:12.425536 (+   335us) service_pool.cc:224] Handling call
0709 13:08:12.425884 (+   348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:13.448916 (+1023032us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:13.578838 (+129922us) spinlock_profiling.cc:243] Waited 998 ms on lock 0x7b54000636d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:13.579029 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2295981952}
I20260709 13:08:13.581955 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:13.582841 25937 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:13.569434 27075 raft_consensus.cc:2243] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 13:08:13.587038 25129 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet eda233ace2a24b379fef320a5f610c7a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 46de6919f85341c6a9b574dff112e35a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:08:13.563717 27137 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.187s	user 0.000s	sys 0.000s
I20260709 13:08:13.588625 27041 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:13.589536 25938 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:13.589319 27041 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:13.590023 25212 leader_election.cc:400] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5.
W20260709 13:08:13.562517 27133 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
I20260709 13:08:13.591952 27056 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:08:13.593418 25212 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:13.570406 25143 catalog_manager.cc:5697] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 2, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:08:13.594492 25212 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:13.595595 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:13.596477 25317 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:13.597007 25315 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:13.598155 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:13.598588 25143 catalog_manager.cc:5697] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:13.607379 25143 catalog_manager.cc:5697] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 2, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
W20260709 13:08:13.561051 27131 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.278s	user 0.000s	sys 0.002s
I20260709 13:08:13.570008 25316 rpcz_store.cc:276] 0709 13:08:11.854975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:11.855528 (+   553us) service_pool.cc:224] Handling call
0709 13:08:11.855809 (+   281us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:12.019703 (+163894us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:12.019731 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:12.054437 (+ 34706us) log.cc:844] Serialized 26 byte log entry
0709 13:08:12.346346 (+291909us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:12.631222 (+284876us) log.cc:844] Serialized 14 byte log entry
0709 13:08:12.638391 (+  7169us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:13.552199 (+913808us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b540005ee54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:13.552297 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:13.552366 (+    69us) raft_consensus.cc:1685] finished
0709 13:08:13.552490 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:13.552677 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15402,"prepare.queue_time_us":70965,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":31,"raft.queue_time_us":45357,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":10747,"spinlock_wait_cycles":376231168,"thread_start_us":530383,"threads_started":2,"wal-append.queue_time_us":327414}
W20260709 13:08:13.625070 25909 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.899s	user 0.057s	sys 0.069s
W20260709 13:08:13.628840 25909 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.903s	user 0.058s	sys 0.069s
W20260709 13:08:13.630025 25391 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.866s	user 0.020s	sys 0.022s
W20260709 13:08:13.630393 25391 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.866s	user 0.020s	sys 0.022s
I20260709 13:08:13.630445 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:13.631295 25320 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:13.634188 25314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 233) took 1064 ms. Trace:
I20260709 13:08:13.634621 25314 rpcz_store.cc:276] 0709 13:08:12.569893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:12.570051 (+   158us) service_pool.cc:224] Handling call
0709 13:08:12.570457 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:12.630473 (+ 60016us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:13.633959 (+1003486us) spinlock_profiling.cc:243] Waited 51.5 ms on lock 0x7b5400045954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:13.634132 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":118494080}
I20260709 13:08:13.637557 25312 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:08:13.639484 25358 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:13.632143 25143 catalog_manager.cc:5697] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 3, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 3 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:13.618263 27059 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:13.636616 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:13.623507 26997 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.005s	sys 0.015s
I20260709 13:08:13.623953 26914 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:13.649215 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 1750) took 1794 ms. Trace:
I20260709 13:08:13.649912 25321 rpcz_store.cc:276] 0709 13:08:11.855303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:11.855411 (+   108us) service_pool.cc:224] Handling call
0709 13:08:11.880091 (+ 24680us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:12.297350 (+417259us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:12.297380 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:12.432650 (+135270us) log.cc:844] Serialized 26 byte log entry
0709 13:08:12.800601 (+367951us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:08:12.800644 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:12.800711 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:13.561811 (+761100us) spinlock_profiling.cc:243] Waited 25.5 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:13.645864 (+ 84053us) raft_consensus.cc:1685] finished
0709 13:08:13.645895 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:13.649153 (+  3258us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24182,"raft.queue_time_us":108570,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":24860,"spinlock_wait_cycles":58621696,"thread_start_us":659792,"threads_started":2,"wal-append.queue_time_us":852815}
W20260709 13:08:13.641791 27051 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.760s	user 0.022s	sys 0.020s
I20260709 13:08:13.648169 27041 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:13.648103 27053 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:08:13.655375 26997 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.005s	sys 0.015s
I20260709 13:08:13.665961 27075 raft_consensus.cc:2272] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:08:13.666628 25143 catalog_manager.cc:5697] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 2 to 4, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 4 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:13.679230 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1191) took 1213 ms. Trace:
I20260709 13:08:13.681622 25243 rpcz_store.cc:276] 0709 13:08:12.466295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:12.468879 (+  2584us) service_pool.cc:224] Handling call
0709 13:08:12.475781 (+  6902us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:12.476224 (+   443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:12.476267 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:12.476845 (+   578us) log.cc:844] Serialized 26 byte log entry
0709 13:08:12.890362 (+413517us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:12.890412 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:12.890492 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:13.560299 (+669807us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:13.667227 (+106928us) raft_consensus.cc:1685] finished
0709 13:08:13.667256 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:13.679136 (+ 11880us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":932,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":86858,"spinlock_wait_cycles":245508864,"thread_start_us":311351,"threads_started":1,"wal-append.queue_time_us":721423}
I20260709 13:08:13.656103 26914 raft_consensus.cc:697] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:13.669157 27053 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:13.649168 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:13.689028 25321 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 13:08:13.689589 25908 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.758s	user 0.007s	sys 0.035s
W20260709 13:08:13.684980 25394 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.937s	user 0.017s	sys 0.028s
W20260709 13:08:13.696550 25908 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.766s	user 0.007s	sys 0.035s
I20260709 13:08:13.649427 27059 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:13.702703 25361 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.012s	sys 0.034s
W20260709 13:08:13.656292 27051 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.855s	user 0.022s	sys 0.021s
I20260709 13:08:13.665289 26944 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:13.703207 25361 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.012s	sys 0.035s
W20260709 13:08:13.696720 25394 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.949s	user 0.017s	sys 0.028s
W20260709 13:08:13.683147 25239 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.012s	sys 0.017s
I20260709 13:08:13.655853 26998 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260709 13:08:13.688324 27053 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:13.721698 25911 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.810s	user 0.008s	sys 0.022s
W20260709 13:08:13.697929 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 767430us
W20260709 13:08:13.711658 25239 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.012s	sys 0.017s
W20260709 13:08:13.727818 25245 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.014s	sys 0.034s
I20260709 13:08:13.705421 26846 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:13.731791 25245 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.014s	sys 0.034s
W20260709 13:08:13.717677 27014 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.007s	sys 0.025s
W20260709 13:08:13.729650 25911 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.818s	user 0.009s	sys 0.022s
I20260709 13:08:13.732666 26846 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:13.733083 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:13.741010 25908 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:13.746753 27135 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 13:08:13.737774 26923 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:13.725796 26970 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260709 13:08:13.778486 27111 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:13.781805 27127 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.500s	user 0.003s	sys 0.000s
I20260709 13:08:13.765219 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:13.745332 26997 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:08:13.734668 27014 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.007s	sys 0.025s
I20260709 13:08:13.742733 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:13.751066 25910 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.898s	user 0.047s	sys 0.118s
I20260709 13:08:13.750084 26925 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:13.858465 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:13.858721 27075 ts_tablet_manager.cc:1916] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:08:13.861486 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 230) took 1257 ms. Trace:
I20260709 13:08:13.861881 25387 rpcz_store.cc:276] 0709 13:08:12.604073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:12.604263 (+   190us) service_pool.cc:224] Handling call
0709 13:08:12.604714 (+   451us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:12.605151 (+   437us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:12.605192 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:12.605767 (+   575us) log.cc:844] Serialized 26 byte log entry
0709 13:08:12.928439 (+322672us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:12.928497 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:12.928596 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:13.861284 (+932688us) raft_consensus.cc:1685] finished
0709 13:08:13.861304 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:13.861434 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":877,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":10259,"thread_start_us":277855,"threads_started":1,"wal-append.queue_time_us":1023962}
W20260709 13:08:13.792183 25246 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.020s	user 0.013s	sys 0.032s
I20260709 13:08:13.792277 25391 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
W20260709 13:08:13.821019 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 80155.9us
I20260709 13:08:13.835387 26972 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 13:08:13.789848 27039 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.866s	user 0.007s	sys 0.025s
I20260709 13:08:13.873814 26970 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:13.881165 27039 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.966s	user 0.008s	sys 0.025s
I20260709 13:08:13.854154 26944 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:13.874186 27111 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:13.881583 26970 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:13.882141 26944 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:13.882339 27111 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:13.884166 26970 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:13.885398 27132 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:08:13.856531 26997 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:13.888092 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:13.893285 25940 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:13.895620 26846 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:13.898073 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:13.898942 25937 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:13.857280 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:13.901821 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:13.857410 25936 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:13.750746 26978 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.090s
I20260709 13:08:13.742342 26914 consensus_queue.cc:237] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:13.869869 25246 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.097s	user 0.013s	sys 0.032s
I20260709 13:08:13.874287 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:13.884292 26944 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:13.855935 27041 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:13.898967 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:13.941890 25320 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:13.942750 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:13.943477 25317 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:13.943897 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:13.944509 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:13.911113 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:13.918382 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:13.946434 25314 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
W20260709 13:08:13.858763 25910 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.117s	user 0.047s	sys 0.118s
I20260709 13:08:13.936211 25395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1102) took 1049 ms. Trace:
I20260709 13:08:13.940523 25945 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:13.950263 25395 rpcz_store.cc:276] 0709 13:08:12.887230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:13.267321 (+380091us) service_pool.cc:224] Handling call
0709 13:08:13.935321 (+668000us) spinlock_profiling.cc:243] Waited 1.92 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:13.935585 (+   264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:13.935802 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:13.935839 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:13.935895 (+    56us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:13.935937 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:13.936010 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:13.936141 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4405120}
I20260709 13:08:13.951838 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:13.904384 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:13.954164 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:13.956909 25944 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:13.945209 25315 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:13.884299 27111 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:08:13.936893 26972 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:13.953168 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:08:13.961802 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 85505us
I20260709 13:08:13.963393 26998 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:13.958482 26999 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260709 13:08:13.966655 26997 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:13.967365 26995 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:13.941154 26936 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:13.948828 26875 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
W20260709 13:08:13.990783 25209 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.007s	sys 0.032s
W20260709 13:08:13.995406 25359 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.849s	user 0.026s	sys 0.028s
W20260709 13:08:13.996778 25359 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.850s	user 0.026s	sys 0.028s
W20260709 13:08:13.996476 25209 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.007s	sys 0.032s
W20260709 13:08:14.000571 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 711843us
W20260709 13:08:13.999142 25390 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.009s	user 0.026s	sys 0.041s
W20260709 13:08:14.004145 25390 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.015s	user 0.026s	sys 0.041s
W20260709 13:08:14.006093 25284 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.009s	sys 0.021s
W20260709 13:08:14.006531 25284 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.009s	sys 0.021s
W20260709 13:08:14.006863 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 563927us
I20260709 13:08:14.032831 25209 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:13.953805 27041 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:14.034041 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:14.034960 25945 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:14.037509 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:14.037992 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:13.995976 26797 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.023s	sys 0.040s
W20260709 13:08:13.984434 27106 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.007s	sys 0.032s
W20260709 13:08:14.054046 25285 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.022s	sys 0.030s
I20260709 13:08:13.952920 27121 consensus_queue.cc:1059] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 13:08:14.048391 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:14.009861 26996 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.029s	sys 0.034s
I20260709 13:08:13.976541 26972 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:14.072225 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:14.073092 25238 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:14.073442 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:14.073918 26972 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:14.074541 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:14.075595 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:14.076570 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:14.097905 27121 raft_consensus.cc:3055] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:14.098352 27121 raft_consensus.cc:740] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:14.099255 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:14.099180 27121 consensus_queue.cc:260] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:14.099949 27039 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:14.100047 25244 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:14.100375 27121 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:14.102227 27041 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:14.102574 25244 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:13.984364 26998 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:14.069065 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:14.108644 25243 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:14.092422 27053 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:14.071941 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:14.072090 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:14.122350 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:14.123081 25939 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:14.124073 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:14.079442 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:14.069401 26996 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.773s	user 0.030s	sys 0.034s
W20260709 13:08:14.046072 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 50190us
I20260709 13:08:14.069059 27063 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:08:14.053691 26797 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.023s	sys 0.040s
I20260709 13:08:14.081421 25942 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:14.132609 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:14.133909 25392 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.943s	user 0.033s	sys 0.038s
I20260709 13:08:13.993054 26936 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:14.087729 26988 consensus_meta.cc:287] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.534s	user 0.008s	sys 0.003s
I20260709 13:08:14.089804 25938 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
W20260709 13:08:14.054577 27106 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.809s	user 0.007s	sys 0.033s
I20260709 13:08:14.093058 27030 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:14.049458 26748 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:14.106365 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:14.161738 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 55467us
I20260709 13:08:14.162609 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:14.163086 25388 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:14.163683 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:14.164441 26997 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:14.164813 26997 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:14.070504 25284 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:14.165216 26997 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:14.165618 26936 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:14.135452 26995 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:08:14.166555 27147 log.cc:927] Time spent T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.078s	user 0.002s	sys 0.000s
I20260709 13:08:14.166617 26995 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:14.168431 27118 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:14.067080 25285 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.022s	sys 0.030s
W20260709 13:08:14.169663 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 746532us
I20260709 13:08:14.124709 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:14.170791 27138 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:14.144285 26988 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:14.171468 27138 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:14.129611 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:14.129734 25395 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:52318
I20260709 13:08:14.047010 26925 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260709 13:08:14.141793 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:14.134395 25392 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.943s	user 0.033s	sys 0.038s
I20260709 13:08:14.144575 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:14.145000 25391 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:08:14.198643 25942 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:14.173309 25321 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:14.147178 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:14.200067 25387 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:08:14.200742 25908 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:14.138839 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:14.175074 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 105672us
W20260709 13:08:14.206753 25211 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.015s	sys 0.059s
W20260709 13:08:14.207086 25211 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.015s	sys 0.059s
I20260709 13:08:14.207201 25395 raft_consensus.cc:606] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:14.206615 25284 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:14.207573 25395 raft_consensus.cc:614] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:08:14.208966 25285 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:14.210742 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:14.215324 25284 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:14.173077 27146 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 13:08:14.157629 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:14.168913 26995 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:14.222052 25212 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.002s	sys 0.022s
I20260709 13:08:14.217319 25285 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:08:14.229630 25212 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.002s	sys 0.022s
I20260709 13:08:14.160948 26946 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:08:14.134343 25242 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.002s	sys 0.025s
I20260709 13:08:14.202683 26944 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:14.139380 27063 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:14.173089 26988 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:14.198798 27030 raft_consensus.cc:697] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:14.205591 26875 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:14.170667 26998 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:14.232020 25284 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:14.233714 25318 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.000s	sys 0.022s
I20260709 13:08:14.230501 26797 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:14.159793 26970 consensus_queue.cc:1059] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.076s
W20260709 13:08:14.283275 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 66037.2us
I20260709 13:08:14.249769 25359 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:14.198662 26748 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:14.168956 27118 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:14.332185 25318 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.000s	sys 0.023s
I20260709 13:08:14.416731 27160 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.189s
I20260709 13:08:14.480141 27118 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:14.481815 26834 raft_consensus.cc:3055] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:14.216645 25937 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 3 LEARNER]: Advancing to term 5
I20260709 13:08:14.231340 26975 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:14.230324 26936 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:08:14.463431 27005 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.090s	sys 0.082s
I20260709 13:08:14.282248 26999 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:14.501889 27138 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:14.282526 26946 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:14.520439 27156 log.cc:927] Time spent T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260709 13:08:14.518787 27053 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:14.513242 27063 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:14.232540 27106 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:14.216512 27150 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:08:14.550733 27005 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.091s	sys 0.082s
I20260709 13:08:14.532513 26834 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:08:14.550487 27108 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.981s	user 0.024s	sys 0.045s
I20260709 13:08:14.551131 26936 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:14.332813 27014 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:14.550076 26975 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:14.234942 27136 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 13:08:14.593274 27108 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.023s	user 0.024s	sys 0.049s
I20260709 13:08:14.587354 27150 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:14.284336 27154 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.372s
W20260709 13:08:14.610067 27145 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.023s	sys 0.017s
W20260709 13:08:14.623665 27145 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.023s	sys 0.017s
I20260709 13:08:14.606793 27136 raft_consensus.cc:697] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:14.626264 27150 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:14.641230 27162 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 13:08:14.652524 27150 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:14.657531 27150 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:14.657800 27153 log.cc:927] Time spent T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
W20260709 13:08:14.658632 27167 log.cc:927] Time spent T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 13:08:14.229993 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 612270us
W20260709 13:08:14.697484 25240 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.001s	sys 0.000s
W20260709 13:08:14.698338 25240 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.001s	sys 0.000s
W20260709 13:08:14.282840 25242 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.809s	user 0.003s	sys 0.025s
W20260709 13:08:14.703027 25246 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.001s	sys 0.012s
W20260709 13:08:14.703504 25246 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.766s	user 0.001s	sys 0.012s
W20260709 13:08:14.705072 25287 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.004s	sys 0.010s
W20260709 13:08:14.705317 25312 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.801s	user 0.008s	sys 0.004s
W20260709 13:08:14.705665 25287 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.004s	sys 0.010s
W20260709 13:08:14.705895 25312 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.802s	user 0.008s	sys 0.004s
I20260709 13:08:14.706043 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:14.707206 25209 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.006s	sys 0.002s
W20260709 13:08:14.707682 25209 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.006s	sys 0.002s
W20260709 13:08:14.708263 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 659894us
I20260709 13:08:14.702468 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1206) took 1313 ms. Trace:
I20260709 13:08:14.709493 25316 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:14.711558 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 1759) took 1115 ms. Trace:
I20260709 13:08:14.522914 25394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1094) took 1792 ms. Trace:
I20260709 13:08:14.711467 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:14.710546 25212 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:14.546716 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:14.284693 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:14.706409 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 601220us
W20260709 13:08:14.714179 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 569964us
I20260709 13:08:14.731256 26772 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:08:14.709223 25242 rpcz_store.cc:276] 0709 13:08:13.388751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:13.388951 (+   200us) service_pool.cc:224] Handling call
0709 13:08:13.395347 (+  6396us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:13.414216 (+ 18869us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:13.417632 (+  3416us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:13.473645 (+ 56013us) log.cc:844] Serialized 26 byte log entry
0709 13:08:14.701758 (+1228113us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:14.701869 (+   111us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:14.702000 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:14.702073 (+    73us) raft_consensus.cc:1685] finished
0709 13:08:14.702092 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:14.702297 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5132,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":4827,"thread_start_us":1227430,"threads_started":1,"wal-append.queue_time_us":1039054}
I20260709 13:08:14.735086 26743 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:14.735765 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:14.736450 26743 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:14.732812 25359 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:14.714327 25394 rpcz_store.cc:276] 0709 13:08:12.730777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:12.746884 (+ 16107us) service_pool.cc:224] Handling call
0709 13:08:12.747286 (+   402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:13.710727 (+963441us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:13.710787 (+    60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:13.723044 (+ 12257us) log.cc:844] Serialized 26 byte log entry
0709 13:08:14.218091 (+495047us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:14.218132 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:14.218206 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:14.522690 (+304484us) raft_consensus.cc:1685] finished
0709 13:08:14.522710 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:14.522853 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":168119,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":11680,"spinlock_wait_cycles":1408,"thread_start_us":1430259,"threads_started":2,"wal-append.queue_time_us":789443}
I20260709 13:08:14.740018 26743 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:14.736799 25242 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:14.746371 26748 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:14.747789 25359 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:14.722102 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 490010us
I20260709 13:08:14.712170 25318 rpcz_store.cc:276] 0709 13:08:13.595648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:13.595809 (+   161us) service_pool.cc:224] Handling call
0709 13:08:13.633571 (+ 37762us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:13.639902 (+  6331us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:13.639936 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:13.645088 (+  5152us) log.cc:844] Serialized 26 byte log entry
0709 13:08:14.464710 (+819622us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:14.464770 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:14.464869 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:14.710912 (+246043us) raft_consensus.cc:1685] finished
0709 13:08:14.710957 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:14.711273 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":99,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":8779,"thread_start_us":794802,"threads_started":1,"wal-append.queue_time_us":968152}
I20260709 13:08:14.751556 26748 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:14.751518 25284 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:14.752089 26748 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:14.752547 26748 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:14.723860 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 260130us
I20260709 13:08:14.753718 26795 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:14.754793 26795 consensus_queue.cc:1059] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 13:08:14.729156 25943 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.017s	sys 0.010s
I20260709 13:08:14.757428 26795 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:08:14.757958 25943 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.018s	sys 0.010s
I20260709 13:08:14.714337 25390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1096) took 1958 ms. Trace:
I20260709 13:08:14.716109 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1473) took 2028 ms. Trace:
I20260709 13:08:14.759702 26795 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:14.760563 26795 raft_consensus.cc:3055] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:14.761111 26795 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:14.759997 25390 rpcz_store.cc:276] 0709 13:08:12.755373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:12.987560 (+232187us) service_pool.cc:224] Handling call
0709 13:08:12.987876 (+   316us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:12.988191 (+   315us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:12.988218 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:12.988668 (+   450us) log.cc:844] Serialized 26 byte log entry
0709 13:08:14.005796 (+1017128us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:08:14.005873 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:14.005978 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:14.713610 (+707632us) raft_consensus.cc:1685] finished
0709 13:08:14.713669 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:14.714018 (+   349us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":266472,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":11482,"thread_start_us":1015775,"threads_started":1,"wal-append.queue_time_us":1469062}
I20260709 13:08:14.762530 26795 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:14.760097 25239 rpcz_store.cc:276] 0709 13:08:12.687120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:12.687264 (+   144us) service_pool.cc:224] Handling call
0709 13:08:12.687659 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:12.965633 (+277974us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:12.965675 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:12.966248 (+   573us) log.cc:844] Serialized 26 byte log entry
0709 13:08:13.725647 (+759399us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:13.725711 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:13.771884 (+ 46173us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b5400363b54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:13.771946 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:14.172413 (+400467us) spinlock_profiling.cc:243] Waited 36.5 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:14.715627 (+543214us) raft_consensus.cc:1685] finished
0709 13:08:14.715668 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:14.715921 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2611,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":133,"spinlock_wait_cycles":722054272,"thread_start_us":758939,"threads_started":1,"wal-append.queue_time_us":1066886}
I20260709 13:08:14.574765 26797 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:14.723958 25237 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:14.764878 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:14.765496 25359 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:14.765787 26914 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.687s	user 0.012s	sys 0.020s
W20260709 13:08:14.780634 26914 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.012s	sys 0.020s
I20260709 13:08:14.764321 26795 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:14.783514 27166 consensus_queue.cc:1059] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:08:14.665498 27136 consensus_queue.cc:237] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:14.786878 25242 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
W20260709 13:08:14.790599 26996 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.011s	sys 0.013s
W20260709 13:08:14.791111 26996 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.011s	sys 0.013s
I20260709 13:08:14.594635 26936 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:14.792616 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:14.793745 25242 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:14.599973 26975 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:14.798246 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:14.798368 26834 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:14.800177 26834 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:14.800705 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:14.610471 27030 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:14.802088 26797 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:14.784946 27063 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:08:14.846040 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 135914us
I20260709 13:08:14.838644 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:14.850318 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 107751us
I20260709 13:08:14.839087 25937 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 5 LEARNER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20260709 13:08:14.855659 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 99442.9us
I20260709 13:08:14.856349 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:14.860136 27173 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:14.865252 26914 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:14.865798 26914 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:08:14.866221 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 100947us
W20260709 13:08:14.866622 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 113466us
I20260709 13:08:14.867720 25284 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:14.868021 25359 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:14.868484 27132 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.728s	user 0.005s	sys 0.015s
I20260709 13:08:14.869436 26995 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:08:14.855141 25944 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.705s	user 0.002s	sys 0.018s
W20260709 13:08:14.871021 25944 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.003s	sys 0.018s
I20260709 13:08:14.870604 26995 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:08:14.874101 27149 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:08:14.868984 27132 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.792s	user 0.005s	sys 0.015s
I20260709 13:08:14.599785 27014 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:14.872823 26995 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:14.866210 26914 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:14.885159 25285 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:14.883846 27191 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:14.892858 25358 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:14.901561 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:14.901811 25908 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.009s	sys 0.011s
W20260709 13:08:14.902202 25908 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.010s	sys 0.011s
W20260709 13:08:14.903342 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 702620us
I20260709 13:08:14.904151 27165 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:14.905272 27160 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:14.905115 27165 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:14.905985 27160 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:14.907701 27165 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:14.908525 27160 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:14.911203 27166 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:14.911969 27166 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:14.912676 27056 raft_consensus.cc:3055] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:14.913163 27056 raft_consensus.cc:740] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:14.914515 27166 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:14.914415 27056 consensus_queue.cc:260] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:14.861006 27173 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:14.915987 27056 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:14.916025 25911 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:14.917328 26910 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:08:14.917652 26972 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:08:14.921228 27063 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:14.921782 27063 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:14.922273 27063 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:14.922403 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:14.925273 27161 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:14.929328 27014 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:14.924588 27111 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:14.931496 25239 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 1.
I20260709 13:08:14.927125 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:14.942127 25945 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:14.935554 27039 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:08:14.945801 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 56340.7us
W20260709 13:08:14.944581 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 45428.7us
I20260709 13:08:14.923426 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:14.953853 27059 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:14.954988 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 187441us
W20260709 13:08:14.955461 26997 consensus_meta.cc:287] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.790s	user 0.014s	sys 0.000s
I20260709 13:08:14.955901 26997 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:14.956856 25285 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:14.970409 26997 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:14.973433 27032 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:14.974469 27059 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:08:14.974746 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 85839.8us
I20260709 13:08:14.977545 26995 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:08:14.885255 27149 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:14.931173 25237 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
W20260709 13:08:14.972609 27121 consensus_meta.cc:287] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.872s	user 0.018s	sys 0.000s
I20260709 13:08:14.926148 27161 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:14.961683 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 93846.9us
I20260709 13:08:14.947670 27111 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:15.028131 25359 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:15.020397 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1478) took 2259 ms. Trace:
I20260709 13:08:15.029771 25358 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:15.050664 26925 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:15.051640 26925 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:15.053280 27151 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.943s	user 0.005s	sys 0.005s
I20260709 13:08:15.061085 27159 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:15.061688 27159 raft_consensus.cc:697] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:15.062057 25390 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:15.062351 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 320) took 2062 ms. Trace:
I20260709 13:08:15.062909 27159 consensus_queue.cc:237] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:15.062824 25245 rpcz_store.cc:276] 0709 13:08:12.999623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:13.004216 (+  4593us) service_pool.cc:224] Handling call
0709 13:08:13.004624 (+   408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:13.005039 (+   415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:13.005075 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:13.118894 (+113819us) log.cc:844] Serialized 26 byte log entry
0709 13:08:13.732337 (+613443us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:13.732393 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:13.732565 (+   172us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:14.993892 (+1261327us) spinlock_profiling.cc:243] Waited 26.4 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:15.061863 (+ 67971us) raft_consensus.cc:1685] finished
0709 13:08:15.061888 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:15.062148 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":49900,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":20001,"spinlock_wait_cycles":60645632,"thread_start_us":612960,"threads_started":1,"wal-append.queue_time_us":951741}
I20260709 13:08:15.064797 25944 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 547) took 1079 ms. Trace:
I20260709 13:08:15.065179 25944 rpcz_store.cc:276] 0709 13:08:13.985332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:13.986018 (+   686us) service_pool.cc:224] Handling call
0709 13:08:14.036251 (+ 50233us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:14.103143 (+ 66892us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:14.103170 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:14.104376 (+  1206us) log.cc:844] Serialized 26 byte log entry
0709 13:08:14.871689 (+767313us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:14.871780 (+    91us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:14.871894 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:15.064586 (+192692us) raft_consensus.cc:1685] finished
0709 13:08:15.064609 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:15.064726 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":80061,"raft.queue_time_us":72425,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":13222,"thread_start_us":728009,"threads_started":1,"wal-append.queue_time_us":929107}
I20260709 13:08:15.033802 27111 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:15.075325 26936 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:15.077267 25313 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:08:15.082237 27166 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:15.013794 27165 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:14.991403 27117 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:14.990727 27014 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:14.993347 26946 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:14.981303 27191 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:15.042091 25242 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:15.090735 25238 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:15.091188 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:15.093942 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:15.117803 27191 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:15.118453 27191 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:14.994496 26748 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:15.120514 26748 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:15.120571 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1100) took 2364 ms. Trace:
I20260709 13:08:15.121444 27191 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:15.030929 25246 rpcz_store.cc:276] 0709 13:08:12.725436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:12.741347 (+ 15911us) service_pool.cc:224] Handling call
0709 13:08:12.741703 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:12.741891 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:08:13.937799 (+1195908us) log.cc:844] Serialized 14 byte log entry
0709 13:08:14.881259 (+943460us) log.cc:844] Serialized 14 byte log entry
0709 13:08:14.943144 (+ 61885us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:14.951293 (+  8149us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:14.962527 (+ 11234us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:14.962989 (+   462us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:14.984871 (+ 21882us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61933,"prepare.queue_time_us":467353,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":85,"raft.queue_time_us":105756,"raft.run_cpu_time_us":594,"raft.run_wall_time_us":62442,"thread_start_us":2083525,"threads_started":3,"wal-append.queue_time_us":717298}
I20260709 13:08:15.111971 26997 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:15.112197 27165 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:15.114825 27158 log.cc:927] Time spent T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.781s	user 0.005s	sys 0.012s
I20260709 13:08:15.118405 25395 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:15.124351 25909 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:15.094848 25314 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 13:08:15.104746 27111 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:15.112810 27145 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:08:15.151297 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 174134us
I20260709 13:08:15.088387 27117 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:08:15.160820 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 133711us
I20260709 13:08:15.169829 27145 raft_consensus.cc:697] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:15.171257 26923 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:15.171953 26923 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:15.164753 27117 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:15.173554 26875 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:15.174045 26875 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:15.170513 26772 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:15.176092 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:15.150668 26997 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:15.176798 26997 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:15.174894 26772 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:15.179714 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:15.180428 27173 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:15.086212 26914 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:15.180791 27075 ts_tablet_manager.cc:1929] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:08:15.202306 25944 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
W20260709 13:08:15.197940 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 73709.8us
I20260709 13:08:15.181156 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:15.181414 25359 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:15.181805 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 1769) took 1504 ms. Trace:
I20260709 13:08:15.171157 27145 consensus_queue.cc:237] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:15.189334 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 109674us
I20260709 13:08:15.126212 25392 rpcz_store.cc:276] 0709 13:08:12.756494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:13.054455 (+297961us) service_pool.cc:224] Handling call
0709 13:08:13.122155 (+ 67700us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:14.201496 (+1079341us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:14.201589 (+    93us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:14.218647 (+ 17058us) log.cc:844] Serialized 26 byte log entry
0709 13:08:14.713407 (+494760us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:14.713534 (+   127us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:15.119934 (+406400us) spinlock_profiling.cc:243] Waited 26.9 ms on lock 0x7b5400481054. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:15.119993 (+    59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:15.120053 (+    60us) raft_consensus.cc:1685] finished
0709 13:08:15.120079 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:15.120365 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":134138,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":197,"spinlock_wait_cycles":61823360,"thread_start_us":1391889,"threads_started":2,"wal-append.queue_time_us":370396}
I20260709 13:08:15.101482 27166 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:15.169891 26936 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:15.205698 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:15.202154 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:15.161777 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:15.206871 25321 rpcz_store.cc:276] 0709 13:08:13.621573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:13.690101 (+ 68528us) service_pool.cc:224] Handling call
0709 13:08:13.694327 (+  4226us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:14.199785 (+505458us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:15.125487 (+925702us) spinlock_profiling.cc:243] Waited 479 ms on lock 0x7b54000f8a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:15.125790 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1101036288}
I20260709 13:08:15.191045 26834 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:15.230018 27075 log.cc:1199] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-1-root/wals/17584a19690d4010851d65378cfcbafa
I20260709 13:08:15.247854 25284 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:15.263329 25246 raft_consensus.cc:2410] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:08:15.242681 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:15.274861 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 44315us
I20260709 13:08:15.257849 26936 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:15.261765 26996 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:08:15.285516 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 104246us
I20260709 13:08:15.242055 26972 consensus_queue.cc:1059] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.045s
I20260709 13:08:15.294235 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:15.297577 25246 raft_consensus.cc:1813] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Handling vote request from an unknown peer 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:15.298305 25246 raft_consensus.cc:2410] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 13:08:15.335109 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:15.286435 26936 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:15.273118 26925 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:15.289458 25936 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:08:15.259742 26834 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:15.295795 26996 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:15.274210 26975 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:15.345410 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:15.555423 27201 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.352s	user 0.002s	sys 0.000s
W20260709 13:08:15.556181 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:15.557235 26846 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:15.557660 27154 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:15.557792 27217 consensus_queue.cc:1059] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 13:08:15.556915 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:15.334357 25313 raft_consensus.cc:2393] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:08:15.393244 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:15.613029 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 327875us
W20260709 13:08:15.413199 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 117050us
I20260709 13:08:15.597066 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:15.383769 25237 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
W20260709 13:08:15.567126 27204 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.278s	user 0.000s	sys 0.002s
I20260709 13:08:15.557684 26772 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:15.358103 27014 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:15.463447 27059 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:15.562472 26954 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:15.379040 26978 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:15.417834 27111 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:15.565789 27215 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.276s	user 0.001s	sys 0.000s
I20260709 13:08:15.462814 25285 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:15.417366 27136 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:15.562325 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:15.558991 27165 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:15.492223 27039 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:15.509505 27191 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:15.348971 26936 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:15.386011 27161 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:15.630578 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 210347us
I20260709 13:08:15.510036 27150 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:15.601725 26996 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:15.684725 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:15.493458 26995 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:15.558296 27154 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:15.491596 26743 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:15.688721 26954 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:15.742081 26743 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:15.744295 26954 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:15.744571 26743 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:15.557833 26846 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:15.684973 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:15.574427 27166 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:15.688985 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 226278us
I20260709 13:08:15.689370 27111 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:15.689592 27014 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:15.667944 26978 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:15.691237 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1488) took 1596 ms. Trace:
I20260709 13:08:15.691720 27136 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:15.692683 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:15.693617 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:15.695628 25240 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
W20260709 13:08:15.695888 27211 log.cc:927] Time spent T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.119s	user 0.001s	sys 0.001s
I20260709 13:08:15.696059 26975 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:15.752874 25244 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:15.737690 27144 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:15.420315 27117 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:15.718945 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:15.691740 27216 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 13:08:15.697412 27039 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:15.455261 26914 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:15.693631 26834 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:15.739825 27217 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:15.714552 27165 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:15.741269 27154 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:15.757061 27014 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:15.696609 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:15.745303 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:15.745807 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 408936us
I20260709 13:08:15.807853 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:15.667963 27139 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:15.751358 25285 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:15.738286 25209 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:15.812896 25285 raft_consensus.cc:2686] T 17584a19690d4010851d65378cfcbafa P 2329e428ebbf484fa3ea9452133df567: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260709 13:08:15.813364 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 62130.1us
I20260709 13:08:15.813537 25359 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:15.813875 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:15.814679 25314 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 3: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:15.815553 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:15.816593 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:15.817456 26914 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:15.818032 25313 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:15.809885 27014 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:15.819473 25313 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:15.777731 27111 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:15.822057 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:15.822801 25245 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:15.745890 27059 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:15.763192 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 329) took 1036 ms. Trace:
I20260709 13:08:15.769276 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 1220) took 1058 ms. Trace:
W20260709 13:08:15.769435 25940 consensus_meta.cc:287] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.696s	user 0.011s	sys 0.001s
I20260709 13:08:15.828337 25244 rpcz_store.cc:276] 0709 13:08:14.709428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:14.711884 (+  2456us) service_pool.cc:224] Handling call
0709 13:08:15.341556 (+629672us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x7b5c0009db8c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297a61c161 00007f297a7bda79 00007f297bc779b6 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:15.767407 (+425851us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":45118720}
I20260709 13:08:15.828193 25240 rpcz_store.cc:276] 0709 13:08:14.726785 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.695202 (+968417us) service_pool.cc:224] Handling call
0709 13:08:15.695469 (+   267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:15.762845 (+ 67376us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:15.763119 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:15.762205 26978 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:15.778671 27207 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.082s	user 0.001s	sys 0.001s
I20260709 13:08:15.347806 27056 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:15.800832 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:15.796277 25239 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:15.745457 26995 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:15.803105 26875 consensus_meta.cc:287] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.628s	user 0.010s	sys 0.002s
I20260709 13:08:15.803027 27165 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:15.804551 27136 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:15.805644 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:15.824564 25395 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:15.824592 25392 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:08:15.761198 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 104322us
I20260709 13:08:15.635710 27121 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:15.790791 26975 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:15.841195 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:15.830360 27217 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:15.842130 27059 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:15.889349 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:15.889508 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:15.891006 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:15.891816 25239 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:15.892474 27108 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:15.893157 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:15.893242 27108 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:15.896102 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:15.897110 25389 raft_consensus.cc:2410] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:08:15.896941 27159 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:08:15.898700 27217 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:15.844823 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 129954us
I20260709 13:08:15.761466 26743 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20260709 13:08:15.761895 25238 rpcz_store.cc:276] 0709 13:08:14.094905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:14.095068 (+   163us) service_pool.cc:224] Handling call
0709 13:08:14.095466 (+   398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:15.115459 (+1019993us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:15.690888 (+575429us) spinlock_profiling.cc:243] Waited 991 ms on lock 0x7b54002d14d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:15.691161 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2280447360}
I20260709 13:08:15.877952 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:15.877108 25394 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:15.811854 27150 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:15.891197 27121 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:15.833619 25940 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:15.823757 27111 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:15.904331 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 333) took 1176 ms. Trace:
I20260709 13:08:15.863806 27145 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:08:15.901420 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 86454.4us
I20260709 13:08:15.906581 25386 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
W20260709 13:08:15.923626 27213 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.267s	user 0.003s	sys 0.000s
I20260709 13:08:15.810868 27144 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:15.932698 25245 raft_consensus.cc:2410] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 13:08:15.938505 25240 rpcz_store.cc:276] 0709 13:08:14.728318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.843073 (+1114755us) service_pool.cc:224] Handling call
0709 13:08:15.865695 (+ 22622us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:15.900882 (+ 35187us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:15.900915 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:15.900971 (+    56us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:15.901001 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:15.901057 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:15.904264 (+  3207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:15.902238 26797 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:15.954891 25318 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:15.955008 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:15.949961 25209 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:15.956060 25245 raft_consensus.cc:2410] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 6 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260709 13:08:15.859577 26875 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:15.960179 27230 log.cc:927] Time spent T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.158s	user 0.001s	sys 0.000s
W20260709 13:08:15.981335 27226 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.138s	user 0.002s	sys 0.000s
I20260709 13:08:15.982013 25391 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260709 13:08:15.982733 27210 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.420s	user 0.001s	sys 0.001s
I20260709 13:08:15.982959 26875 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:15.879901 27056 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:15.961653 27212 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.514s	user 0.000s	sys 0.007s
W20260709 13:08:15.963830 26923 consensus_meta.cc:287] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.791s	user 0.005s	sys 0.005s
I20260709 13:08:15.941603 27145 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:15.999820 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1222) took 1289 ms. Trace:
I20260709 13:08:15.967667 27165 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:15.969487 26797 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:15.970333 27108 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:16.011806 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 61937.2us
I20260709 13:08:15.970952 27150 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:15.971837 27121 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:15.972155 26995 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:15.973007 26846 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:15.974738 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:15.979324 27200 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.436s	user 0.001s	sys 0.001s
I20260709 13:08:15.948568 26975 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:15.993981 25908 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:16.000459 25246 rpcz_store.cc:276] 0709 13:08:14.710337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.583104 (+872767us) service_pool.cc:224] Handling call
0709 13:08:15.609374 (+ 26270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:15.636223 (+ 26849us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:08:15.715794 (+ 79571us) log.cc:844] Serialized 14 byte log entry
0709 13:08:15.999411 (+283617us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:15.999464 (+    53us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:15.999512 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:15.999601 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:15.999749 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68843,"prepare.queue_time_us":60500,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":36,"thread_start_us":171855,"threads_started":1}
I20260709 13:08:16.002429 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:15.952144 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:15.862972 27139 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:16.021497 25285 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:15.960224 27117 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:16.033934 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1225) took 1281 ms. Trace:
W20260709 13:08:16.052981 27063 consensus_meta.cc:287] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.130s	user 0.013s	sys 0.002s
I20260709 13:08:16.006273 27145 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:16.065523 25318 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:16.057838 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.045542 27108 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:16.078280 25391 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:16.071048 25240 rpcz_store.cc:276] 0709 13:08:14.732430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.956036 (+1223606us) service_pool.cc:224] Handling call
0709 13:08:15.996004 (+ 39968us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:16.013183 (+ 17179us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:16.013225 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:16.013295 (+    70us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:16.013344 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:16.013865 (+   521us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:16.014031 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:16.007627 26923 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:16.105894 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 84498.3us
I20260709 13:08:16.111924 25285 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:16.115296 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1231) took 1361 ms. Trace:
I20260709 13:08:16.115767 25245 rpcz_store.cc:276] 0709 13:08:14.744640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.965354 (+1220714us) service_pool.cc:224] Handling call
0709 13:08:15.965690 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:15.965945 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:08:15.966549 (+   604us) log.cc:844] Serialized 14 byte log entry
0709 13:08:16.106140 (+139591us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:16.106187 (+    47us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:16.106226 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:16.106293 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:16.106423 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53440,"prepare.queue_time_us":192,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"thread_start_us":85673,"threads_started":1}
I20260709 13:08:16.073571 26846 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:16.125002 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 76019.8us
I20260709 13:08:16.123384 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.123616 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:16.082964 25939 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:16.130424 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:16.134716 25395 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:16.135234 25391 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:16.111193 26925 raft_consensus.cc:3055] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:08:16.129601 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 135736us
I20260709 13:08:16.112511 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 335) took 1383 ms. Trace:
I20260709 13:08:16.152738 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:16.171140 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.171976 25395 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:16.172122 26978 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:16.173697 25395 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:16.174492 26978 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:08:16.176887 25386 raft_consensus.cc:1275] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:08:16.181622 26772 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:16.182236 26772 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:16.182777 26772 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:16.185741 25942 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
W20260709 13:08:16.188014 27232 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:08:16.189386 27189 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:16.192680 27075 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36848 (request call id 80) took 5113 ms. Trace:
I20260709 13:08:16.144953 27136 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:16.193100 27075 rpcz_store.cc:276] 0709 13:08:11.134529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:11.149070 (+ 14541us) service_pool.cc:224] Handling call
0709 13:08:11.362116 (+213046us) ts_tablet_manager.cc:1151] Deleting tablet 17584a19690d4010851d65378cfcbafa
0709 13:08:11.398459 (+ 36343us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 13:08:13.587705 (+2189246us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b5400044f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cba362 00007f297a6a534c 00007f297bcc860a 00007f297bcf92d0 00007f297bcdf83d 00007f297bcdf7e9 00007f297bcdf779 00007f297bcdf741 00007f297bcde96d 000000000038f354
0709 13:08:14.079037 (+491332us) tablet_metadata.cc:665] Metadata flushed
0709 13:08:15.048901 (+969864us) tablet_metadata.cc:665] Metadata flushed
0709 13:08:15.963659 (+914758us) spinlock_profiling.cc:243] Waited 36.9 ms on lock 0x7b58000e3498. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8e1a2 00007f297cacfbb9 00007f297bc00d88 00007f297bcc9c8e 00007f297bcccc59 00007f297bcf5a7e 00007f297bcf5a49 00007f297bcf5a05 00007f297bce6c66
0709 13:08:16.192543 (+228884us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2028947,"spinlock_wait_cycles":316614784,"tablet-delete.queue_time_us":109422,"thread_start_us":19609,"threads_started":1}
W20260709 13:08:16.194234 27218 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.637s	user 0.001s	sys 0.002s
I20260709 13:08:16.099666 27063 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:16.204457 25391 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:08:16.214478 25237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 325) took 1503 ms. Trace:
I20260709 13:08:16.214743 26972 consensus_queue.cc:1059] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:08:16.145493 26925 raft_consensus.cc:740] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:16.215142 25237 rpcz_store.cc:276] 0709 13:08:14.725086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.696882 (+971796us) service_pool.cc:224] Handling call
0709 13:08:15.777822 (+ 80940us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:15.948890 (+171068us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:15.954989 (+  6099us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:15.968434 (+ 13445us) log.cc:844] Serialized 26 byte log entry
0709 13:08:16.122070 (+153636us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:16.133631 (+ 11561us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:16.142876 (+  9245us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:16.200869 (+ 57993us) raft_consensus.cc:1685] finished
0709 13:08:16.200895 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:16.214389 (+ 13494us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":88039,"raft.queue_time_us":98085,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":10573,"thread_start_us":103737,"threads_started":1,"wal-append.queue_time_us":214712}
I20260709 13:08:16.225057 27063 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:16.195253 27108 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:16.225962 27221 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:16.228294 25936 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: []
W20260709 13:08:16.230252 25940 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 393) took 2113 ms (client timeout 1502 ms). Trace:
I20260709 13:08:16.194267 27136 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:16.201709 27189 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:16.230458 27139 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:16.166608 27121 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:16.214478 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1677) took 2148 ms. Trace:
I20260709 13:08:16.167663 27191 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:16.360805 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:16.217299 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:16.157974 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:16.363508 25314 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:16.387347 25321 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:16.394173 27166 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:16.395166 27166 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:16.252326 27205 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.867s	user 0.003s	sys 0.000s
I20260709 13:08:16.399966 25317 rpcz_store.cc:276] 0709 13:08:14.058618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:14.058866 (+   248us) service_pool.cc:224] Handling call
0709 13:08:14.059286 (+   420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:14.512238 (+452952us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:14.512305 (+    67us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:14.514493 (+  2188us) log.cc:844] Serialized 26 byte log entry
0709 13:08:14.909553 (+395060us) spinlock_profiling.cc:243] Waited 4.11 ms on lock 0x7b1800437e00. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f29768520d1 00007f2976c6da0f
0709 13:08:15.121182 (+211629us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:08:15.121238 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:15.576602 (+455364us) spinlock_profiling.cc:243] Waited 452 ms on lock 0x7b540002c6d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:15.576674 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:15.978598 (+401924us) spinlock_profiling.cc:243] Waited 67.3 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:16.195356 (+216758us) raft_consensus.cc:1685] finished
0709 13:08:16.195386 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:16.203764 (+  8378us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":122,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":3900,"spinlock_wait_cycles":1204873600,"thread_start_us":129945,"threads_started":1,"wal-append.queue_time_us":1028499}
I20260709 13:08:16.403116 27121 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:16.254046 26996 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:16.408254 26996 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:16.252683 27144 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:16.250147 25936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 556) took 1024 ms. Trace:
I20260709 13:08:16.409931 25936 rpcz_store.cc:276] 0709 13:08:15.226183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.880549 (+654366us) service_pool.cc:224] Handling call
0709 13:08:16.227835 (+347286us) spinlock_profiling.cc:243] Waited 2.6 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:16.228080 (+   245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:16.249843 (+ 21763us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:16.250078 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5983616}
W20260709 13:08:16.253798 26997 consensus_meta.cc:287] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.035s	user 0.013s	sys 0.000s
I20260709 13:08:16.411985 26997 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:16.413555 25387 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:16.249601 27221 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:16.418393 27221 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:16.421857 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 310009us
I20260709 13:08:16.422986 25285 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:16.425588 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:16.426445 25943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 561) took 1198 ms. Trace:
I20260709 13:08:16.271487 27149 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:16.426779 25943 rpcz_store.cc:276] 0709 13:08:15.231451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.254272 (+ 22821us) service_pool.cc:224] Handling call
0709 13:08:16.425465 (+1171193us) spinlock_profiling.cc:243] Waited 4.25 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8deee 0000000000375939 00007f297bc011e0 00007f297bcca0c4 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:16.426367 (+   902us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9778432}
I20260709 13:08:16.271421 27136 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:16.246559 27238 log.cc:927] Time spent T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
W20260709 13:08:16.439975 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 168783us
I20260709 13:08:16.440984 25908 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:16.245659 26975 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:16.242673 26978 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:16.154763 25242 rpcz_store.cc:276] 0709 13:08:14.772928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.115619 (+342691us) service_pool.cc:224] Handling call
0709 13:08:15.974518 (+858899us) spinlock_profiling.cc:243] Waited 18.4 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:16.106265 (+131747us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":42374912}
I20260709 13:08:16.240753 26972 consensus_queue.cc:1059] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.216s
W20260709 13:08:16.457899 25392 consensus_meta.cc:287] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.558s	user 0.011s	sys 0.000s
I20260709 13:08:16.388403 27189 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:16.540709 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1232) took 1779 ms. Trace:
I20260709 13:08:16.538692 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 334) took 1810 ms. Trace:
I20260709 13:08:16.514627 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:16.547426 25246 rpcz_store.cc:276] 0709 13:08:14.761484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:16.194433 (+1432949us) service_pool.cc:224] Handling call
0709 13:08:16.511635 (+317202us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:16.532509 (+ 20874us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:16.532572 (+    63us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:16.540418 (+  7846us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:16.540460 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:16.540531 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:16.540639 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:16.547685 25244 rpcz_store.cc:276] 0709 13:08:14.728665 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.855237 (+1126572us) service_pool.cc:224] Handling call
0709 13:08:15.878839 (+ 23602us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:15.924986 (+ 46147us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:15.925028 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:15.936898 (+ 11870us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:15.945689 (+  8791us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:16.538355 (+592666us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b54002b2854. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:16.538421 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:16.538614 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":51458816}
W20260709 13:08:16.516234 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 75376.2us
W20260709 13:08:16.546947 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 124078us
I20260709 13:08:16.550663 25285 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:16.544422 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:16.547643 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:16.548478 25392 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
W20260709 13:08:16.439328 27014 consensus_meta.cc:287] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.620s	user 0.007s	sys 0.003s
I20260709 13:08:16.469532 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1684) took 2056 ms. Trace:
I20260709 13:08:16.559129 25314 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:16.561369 25315 rpcz_store.cc:276] 0709 13:08:14.061624 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:14.106264 (+ 44640us) service_pool.cc:224] Handling call
0709 13:08:14.106593 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:14.332910 (+226317us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:14.332955 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:14.333007 (+    52us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:08:14.333044 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:16.065080 (+1732036us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b54000870d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:16.072296 (+  7216us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:16.117885 (+ 45589us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":519973248}
I20260709 13:08:16.440716 27117 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:16.574195 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.533310 26997 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:16.484474 27132 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:16.500926 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 405) took 1377 ms. Trace:
I20260709 13:08:16.558882 27108 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:16.577363 27245 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
I20260709 13:08:16.569325 27014 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:16.577832 25246 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:08:16.578099 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:16.578687 27236 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
W20260709 13:08:16.595793 26936 consensus_meta.cc:287] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.871s	user 0.002s	sys 0.008s
W20260709 13:08:16.564105 26875 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.000s	sys 0.034s
W20260709 13:08:16.596354 26875 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.000s	sys 0.034s
I20260709 13:08:16.596243 26936 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:16.576792 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.571682 25316 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:16.573561 25386 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:16.598204 27014 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:16.557016 27121 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:16.558465 26923 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:16.585498 25911 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
W20260709 13:08:16.601531 27039 consensus_meta.cc:287] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.777s	user 0.005s	sys 0.006s
I20260709 13:08:16.601822 27039 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:16.602689 25245 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:16.602630 27121 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:16.603616 27039 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:16.589084 25244 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 13:08:16.604151 25244 pending_rounds.cc:85] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 1
W20260709 13:08:16.250309 25940 rpcz_store.cc:269] 0709 13:08:14.116468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:14.125200 (+  8732us) service_pool.cc:224] Handling call
0709 13:08:16.229998 (+2104798us) spinlock_profiling.cc:243] Waited 818 ms on lock 0x7b5400671e54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:16.230164 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1882325248}
I20260709 13:08:16.564512 26978 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:16.590530 26972 consensus_queue.cc:1059] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:08:16.604429 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 331) took 1855 ms. Trace:
I20260709 13:08:16.604816 25241 rpcz_store.cc:276] 0709 13:08:14.727535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.713458 (+985923us) service_pool.cc:224] Handling call
0709 13:08:15.713847 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:15.733506 (+ 19659us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:15.733547 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:15.764280 (+ 30733us) log.cc:844] Serialized 26 byte log entry
0709 13:08:15.901014 (+136734us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:15.901064 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:15.901172 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:16.583082 (+681910us) raft_consensus.cc:1685] finished
0709 13:08:16.583140 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:16.583324 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12085,"raft.queue_time_us":81446,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":11155,"thread_start_us":76292,"threads_started":1,"wal-append.queue_time_us":308661}
I20260709 13:08:16.606324 26978 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:16.606909 26978 raft_consensus.cc:3055] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:16.607236 26978 raft_consensus.cc:740] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:16.608186 26978 consensus_queue.cc:260] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:16.592363 25945 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 563) took 1321 ms. Trace:
I20260709 13:08:16.588425 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:16.598711 27247 consensus_queue.cc:1059] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:08:16.610186 27247 raft_consensus.cc:3055] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:16.609187 25945 rpcz_store.cc:276] 0709 13:08:15.270564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.908669 (+638105us) service_pool.cc:224] Handling call
0709 13:08:15.938067 (+ 29398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:15.953288 (+ 15221us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:15.953322 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:15.998188 (+ 44866us) log.cc:844] Serialized 26 byte log entry
0709 13:08:16.108626 (+110438us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:16.470615 (+361989us) log.cc:844] Serialized 14 byte log entry
0709 13:08:16.478014 (+  7399us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:16.478119 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:16.592136 (+114017us) raft_consensus.cc:1685] finished
0709 13:08:16.592157 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:16.592308 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7048,"prepare.queue_time_us":488617,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":9379,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":18565,"thread_start_us":436860,"threads_started":2,"wal-append.queue_time_us":273743}
I20260709 13:08:16.598969 27117 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:16.610682 27247 raft_consensus.cc:740] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:16.611099 27117 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:16.611572 27117 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:16.599493 25313 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:16.603531 27121 raft_consensus.cc:697] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:16.551769 27063 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:16.599731 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.615628 25390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 248) took 1452 ms. Trace:
I20260709 13:08:16.604439 25244 pending_rounds.cc:107] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 13:08:16.578785 27149 consensus_queue.cc:1059] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 13:08:16.581982 27056 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:16.621979 25244 raft_consensus.cc:2889] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:16.619916 25390 rpcz_store.cc:276] 0709 13:08:15.163169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.163342 (+   173us) service_pool.cc:224] Handling call
0709 13:08:16.032817 (+869475us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8deee 0000000000375939 00007f297bc011e0 00007f297bcca0c4 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:16.032978 (+   161us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:16.118821 (+ 85843us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:08:16.490654 (+371833us) log.cc:844] Serialized 14 byte log entry
0709 13:08:16.615153 (+124499us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:16.615224 (+    71us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:08:16.615290 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:16.615381 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:16.615556 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6580,"prepare.queue_time_us":59039,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":33808,"spinlock_wait_cycles":3827584,"thread_start_us":374563,"threads_started":2}
I20260709 13:08:16.643500 25908 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:16.611912 27247 consensus_queue.cc:260] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:16.707118 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.529080 26925 consensus_queue.cc:260] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:16.717933 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.718745 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.716183 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:16.716347 25287 leader_election.cc:336] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.637s (SENT)
I20260709 13:08:16.671811 25237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1233) took 1881 ms. Trace:
I20260709 13:08:16.722055 25287 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:16.702826 27237 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:16.700927 26996 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:16.719278 26797 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:16.730762 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.731504 25395 raft_consensus.cc:2410] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:08:16.721082 27111 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:16.717589 25242 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:08:16.732666 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.738711 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.745235 27121 consensus_queue.cc:237] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:16.749424 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:16.749855 26772 consensus_meta.cc:287] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.566s	user 0.010s	sys 0.000s
I20260709 13:08:16.750195 25395 raft_consensus.cc:2368] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4.
I20260709 13:08:16.751078 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:16.751397 25939 consensus_meta.cc:287] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.582s	user 0.007s	sys 0.001s
I20260709 13:08:16.751729 25939 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:16.583076 25941 rpcz_store.cc:276] 0709 13:08:15.123588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.158676 (+ 35088us) service_pool.cc:224] Handling call
0709 13:08:15.170309 (+ 11633us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:15.170943 (+   634us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:15.170983 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:15.171587 (+   604us) log.cc:844] Serialized 26 byte log entry
0709 13:08:15.209023 (+ 37436us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:15.209186 (+   163us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:15.209390 (+   204us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:16.193499 (+984109us) spinlock_profiling.cc:243] Waited 51.7 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:16.484256 (+290757us) raft_consensus.cc:1685] finished
0709 13:08:16.490954 (+  6698us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:16.500713 (+  9759us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":123,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"spinlock_wait_cycles":118973312,"thread_start_us":36761,"threads_started":1,"wal-append.queue_time_us":248735}
I20260709 13:08:16.730394 26925 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:16.730324 26996 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:16.738231 26846 consensus_meta.cc:287] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.598s	user 0.011s	sys 0.000s
I20260709 13:08:16.754653 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1230) took 2010 ms. Trace:
I20260709 13:08:16.750216 26772 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:16.752276 25939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 558) took 1480 ms. Trace:
I20260709 13:08:16.713827 27063 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:16.744948 25391 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:16.751729 25395 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:16.755021 26846 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:16.728374 27249 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:16.755115 25238 rpcz_store.cc:276] 0709 13:08:14.744338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.968883 (+1224545us) service_pool.cc:224] Handling call
0709 13:08:15.969251 (+   368us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:16.073233 (+103982us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:16.082643 (+  9410us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:16.222194 (+139551us) log.cc:844] Serialized 26 byte log entry
0709 13:08:16.630037 (+407843us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:16.644748 (+ 14711us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:16.710855 (+ 66107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:16.754403 (+ 43548us) raft_consensus.cc:1685] finished
0709 13:08:16.754433 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:16.754588 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":69459,"raft.queue_time_us":100658,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":46289,"thread_start_us":397585,"threads_started":1,"wal-append.queue_time_us":530567}
I20260709 13:08:16.779513 27063 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:16.781615 26996 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:16.783219 26936 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:16.743521 27111 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:16.786381 27111 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:16.739841 27237 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:16.798460 25390 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:16.799846 27237 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:16.762461 26846 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 13:08:16.752959 27145 consensus_meta.cc:287] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.621s	user 0.005s	sys 0.002s
I20260709 13:08:16.812690 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.814004 25392 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:16.821981 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:16.828243 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.829368 25391 raft_consensus.cc:2410] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:08:16.829834 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.759865 25939 rpcz_store.cc:276] 0709 13:08:15.272414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.867206 (+594792us) service_pool.cc:224] Handling call
0709 13:08:16.752200 (+884994us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:16.831583 27189 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:16.847012 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:16.848114 27246 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.600s	user 0.006s	sys 0.006s
W20260709 13:08:16.849402 27255 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
I20260709 13:08:16.816754 26846 raft_consensus.cc:697] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 4 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:08:16.852921 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 286046us
I20260709 13:08:16.853333 25392 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
W20260709 13:08:16.854583 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 211194us
I20260709 13:08:16.857542 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.817584 27145 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:16.860821 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:16.824277 27237 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.030s
W20260709 13:08:16.865289 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 150812us
I20260709 13:08:16.865363 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:16.871910 25391 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:16.874909 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.879360 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.884413 27145 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:16.859918 27136 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:16.893456 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.894268 25391 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:16.890537 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.895354 25395 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:16.891204 25392 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
W20260709 13:08:16.900255 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:16.871677 27117 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:16.906627 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 343392us
W20260709 13:08:16.907562 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:16.907766 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.908665 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:16.910287 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:16.909382 25316 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:16.909518 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:16.912623 25320 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:16.909615 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:16.916064 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:16.917189 25315 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:16.918285 26772 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:16.921554 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:16.926734 27249 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:16.927067 26772 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:16.927359 27249 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:16.927656 26772 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:16.725031 25237 rpcz_store.cc:276] 0709 13:08:14.762832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:16.252796 (+1489964us) service_pool.cc:224] Handling call
0709 13:08:16.253224 (+   428us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:16.253459 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:08:16.253754 (+   295us) log.cc:844] Serialized 14 byte log entry
0709 13:08:16.573356 (+319602us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:16.586230 (+ 12874us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:16.616013 (+ 29783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:16.618687 (+  2674us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:16.644605 (+ 25918us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":2391,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":25624,"raft.queue_time_us":79,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":9789,"thread_start_us":316245,"threads_started":1,"wal-append.queue_time_us":457251}
I20260709 13:08:16.927284 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:16.942880 25316 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:16.947574 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.914616 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:16.965422 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:16.938362 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:16.979028 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.978708 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:16.921329 27063 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:16.947338 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:16.984356 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:16.906591 26846 consensus_queue.cc:237] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:16.984946 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.983884 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:16.991163 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:16.993028 27136 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:17.003909 25945 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:08:17.007606 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:16.989460 25128 catalog_manager.cc:5028] TS 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): tablet 17584a19690d4010851d65378cfcbafa (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:08:16.987078 27108 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:16.983067 27249 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:16.994501 25944 raft_consensus.cc:2376] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3. Re-sending same reply.
W20260709 13:08:17.059816 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:17.015707 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:17.041270 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:17.010937 26978 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:17.005957 27063 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.045548 27108 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:17.046360 25911 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:17.046804 27264 log.cc:927] Time spent T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20260709 13:08:17.066965 26772 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:17.049968 25128 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 144823us
I20260709 13:08:17.050781 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:17.071985 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:17.072607 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:17.080353 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:17.056032 25284 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:17.057374 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:17.082487 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:17.057788 25392 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:08:17.056579 27014 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:17.066963 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:08:17.073364 25129 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet eda233ace2a24b379fef320a5f610c7a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 46de6919f85341c6a9b574dff112e35a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:17.074306 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:17.012228 27263 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:17.039641 27117 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:17.077324 27259 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:08:17.077904 27108 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:17.066150 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:17.103682 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:17.106794 25241 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:17.089354 25312 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:17.101831 25284 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:17.102785 25391 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 13:08:17.108172 25391 pending_rounds.cc:85] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 5
I20260709 13:08:17.108480 25391 pending_rounds.cc:107] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 13:08:17.108747 25391 raft_consensus.cc:2889] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:17.105710 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:17.063202 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:17.114344 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:17.115192 25392 raft_consensus.cc:2368] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4.
I20260709 13:08:17.088786 27263 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:17.116005 27108 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:08:17.116925 27219 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 1.540s	user 0.021s	sys 0.013s
I20260709 13:08:17.117467 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 254) took 1214 ms. Trace:
I20260709 13:08:17.084777 27189 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:17.090991 27260 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:17.121348 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:17.126185 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1224) took 2394 ms. Trace:
I20260709 13:08:17.126657 25239 rpcz_store.cc:276] 0709 13:08:14.731862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.892507 (+1160645us) service_pool.cc:224] Handling call
0709 13:08:15.892834 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:15.947176 (+ 54342us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:15.947221 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:15.947280 (+    59us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:15.954159 (+  6879us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:17.125906 (+1171747us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b5400364a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:17.125967 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:17.126119 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49253632}
I20260709 13:08:17.116884 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:17.133845 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:17.134251 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:17.136634 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 245) took 2013 ms. Trace:
I20260709 13:08:17.098811 27249 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260709 13:08:17.121466 27263 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:17.137102 25387 rpcz_store.cc:276] 0709 13:08:15.122848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.124519 (+  1671us) service_pool.cc:224] Handling call
0709 13:08:15.124881 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:16.414007 (+1289126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:17.133506 (+719499us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54001ebc54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:17.136561 (+  3055us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2962814336}
I20260709 13:08:17.139112 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:17.090293 26772 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260709 13:08:17.114588 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:17.128351 27260 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:17.148703 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:17.150039 25392 raft_consensus.cc:2368] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4.
I20260709 13:08:17.143549 25941 raft_consensus.cc:2368] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4.
I20260709 13:08:17.154332 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:17.155051 25239 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:17.140517 27117 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:17.138844 27217 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:17.161852 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:17.159932 25321 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:17.162987 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:17.164469 25321 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:17.166044 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:17.152817 27260 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:17.171265 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:17.174898 25312 raft_consensus.cc:2410] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:08:17.122203 25389 rpcz_store.cc:276] 0709 13:08:15.902928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:17.116547 (+1213619us) service_pool.cc:224] Handling call
0709 13:08:17.116951 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:17.117144 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:17.117174 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:17.117215 (+    41us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:17.117249 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:17.117302 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:17.117398 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":33280}
I20260709 13:08:17.172091 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:17.180500 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:17.151677 27189 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.189711 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 400) took 3071 ms. Trace:
I20260709 13:08:17.184871 25387 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:17.172752 27117 raft_consensus.cc:697] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:17.185935 27261 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:17.186132 27132 raft_consensus.cc:3055] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:17.186527 25319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 243) took 1957 ms. Trace:
I20260709 13:08:17.186254 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:17.181468 27136 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:17.187431 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1234) took 2227 ms. Trace:
W20260709 13:08:17.190582 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 114944us
I20260709 13:08:17.201706 25245 rpcz_store.cc:276] 0709 13:08:14.959585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:16.253371 (+1293786us) service_pool.cc:224] Handling call
0709 13:08:16.253667 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:16.603270 (+349603us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:17.187169 (+583899us) spinlock_profiling.cc:243] Waited 349 ms on lock 0x7b5400193054. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:17.187365 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":802241920}
I20260709 13:08:17.190188 25942 rpcz_store.cc:276] 0709 13:08:14.118534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.575032 (+1456498us) service_pool.cc:224] Handling call
0709 13:08:15.575497 (+   465us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:16.186301 (+610804us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:17.189436 (+1003135us) spinlock_profiling.cc:243] Waited 610 ms on lock 0x7b54001b2bd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:17.189642 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1402506496}
I20260709 13:08:17.195173 25319 rpcz_store.cc:276] 0709 13:08:15.175929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.177244 (+  1315us) service_pool.cc:224] Handling call
0709 13:08:15.177724 (+   480us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:15.178231 (+   507us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:15.178282 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:15.178830 (+   548us) log.cc:844] Serialized 26 byte log entry
0709 13:08:15.244569 (+ 65739us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:15.244614 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:15.244686 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:17.116285 (+1871599us) spinlock_profiling.cc:243] Waited 3.18 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:17.133332 (+ 17047us) raft_consensus.cc:1685] finished
0709 13:08:17.133355 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:17.133532 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":108,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":119,"spinlock_wait_cycles":7321344,"thread_start_us":65271,"threads_started":1,"wal-append.queue_time_us":397698}
I20260709 13:08:17.195447 25391 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:17.197785 27189 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:17.203027 25316 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:17.207080 25387 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:17.208446 25241 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:17.213039 25391 raft_consensus.cc:3055] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:17.213447 25391 raft_consensus.cc:740] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:17.214174 25391 consensus_queue.cc:260] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:17.215130 25391 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:17.189764 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:17.173691 27217 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.195003 27132 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:17.226737 25316 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:17.204764 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 1767) took 1316 ms. Trace:
I20260709 13:08:17.225741 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:17.234297 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:17.235059 25387 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:17.235992 25387 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:17.209481 25389 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:17.237824 25391 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 13:08:17.238232 25391 pending_rounds.cc:85] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 2
I20260709 13:08:17.238492 25391 pending_rounds.cc:107] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 13:08:17.238778 25391 raft_consensus.cc:2889] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:17.245736 27117 consensus_queue.cc:237] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.246663 25391 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:08:17.245782 27063 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:17.222947 25242 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:17.219914 27261 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.252151 25391 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:08:17.228510 27136 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.263972 25239 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:17.227952 25388 rpcz_store.cc:276] 0709 13:08:15.885505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:16.878266 (+992761us) service_pool.cc:224] Handling call
0709 13:08:16.908649 (+ 30383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:16.938050 (+ 29401us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:16.938102 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:17.007349 (+ 69247us) log.cc:844] Serialized 26 byte log entry
0709 13:08:17.139957 (+132608us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:08:17.144669 (+  4712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:17.146988 (+  2319us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:17.192367 (+ 45379us) raft_consensus.cc:1685] finished
0709 13:08:17.192387 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:17.201654 (+  9267us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7769,"raft.queue_time_us":28690,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":14404,"thread_start_us":119360,"threads_started":1,"wal-append.queue_time_us":170011}
W20260709 13:08:17.261934 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:17.286609 27261 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:17.287288 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:17.292016 25319 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:17.296559 27132 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.297472 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1214) took 2583 ms. Trace:
I20260709 13:08:17.297627 27263 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.298333 26975 raft_consensus.cc:3055] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:17.297869 25243 rpcz_store.cc:276] 0709 13:08:14.698401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:14.716796 (+ 18395us) service_pool.cc:224] Handling call
0709 13:08:15.121911 (+405115us) spinlock_profiling.cc:243] Waited 7.73 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:15.690092 (+568181us) spinlock_profiling.cc:243] Waited 7.54 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8de9c 00007f2973b8da5b 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:15.690322 (+   230us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:16.031852 (+341530us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:16.031895 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:16.058391 (+ 26496us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:16.058465 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:17.281339 (+1222874us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b54003a9dd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:17.281397 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:17.281532 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":820022784}
I20260709 13:08:17.308627 25940 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:17.312903 27263 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:17.313894 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.323776 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1107) took 1175 ms. Trace:
I20260709 13:08:17.299283 26975 raft_consensus.cc:740] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:17.324347 25386 rpcz_store.cc:276] 0709 13:08:16.148326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:16.178124 (+ 29798us) service_pool.cc:224] Handling call
0709 13:08:16.557251 (+379127us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:16.598263 (+ 41012us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:17.309112 (+710849us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b54001e4950. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:17.316142 (+  7030us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":866558336}
W20260709 13:08:17.327337 27223 log.cc:927] Time spent T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 1.080s	user 0.004s	sys 0.012s
I20260709 13:08:17.328902 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 246) took 2205 ms. Trace:
I20260709 13:08:17.329173 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:17.329290 25393 rpcz_store.cc:276] 0709 13:08:15.123242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.125159 (+  1917us) service_pool.cc:224] Handling call
0709 13:08:15.125647 (+   488us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:15.164653 (+ 39006us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:15.164684 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:15.197409 (+ 32725us) log.cc:844] Serialized 26 byte log entry
0709 13:08:15.333455 (+136046us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:15.333509 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:15.333602 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:17.324899 (+1991297us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:17.328494 (+  3595us) raft_consensus.cc:1685] finished
0709 13:08:17.328515 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:17.328749 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":36674,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":10194,"spinlock_wait_cycles":428980608,"thread_start_us":74519,"threads_started":1,"wal-append.queue_time_us":904401}
I20260709 13:08:17.330768 26975 consensus_queue.cc:260] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.331991 26975 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:17.360965 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:17.362777 27252 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.765s	user 0.000s	sys 0.002s
I20260709 13:08:17.362309 27111 consensus_queue.cc:1059] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 13:08:17.364096 27242 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 1.093s	user 0.005s	sys 0.008s
I20260709 13:08:17.364245 27111 raft_consensus.cc:3055] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:17.365206 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.367022 25392 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 13:08:17.364756 27111 raft_consensus.cc:740] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:17.347847 27263 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:17.347129 27260 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.366529 27132 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:17.373617 27217 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:17.374116 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 254) took 1701 ms. Trace:
W20260709 13:08:17.380465 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:17.364805 25242 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:17.368616 27136 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:17.382668 25313 rpcz_store.cc:276] 0709 13:08:15.673203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.820342 (+147139us) service_pool.cc:224] Handling call
0709 13:08:15.820707 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:16.616300 (+795593us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:17.373835 (+757535us) spinlock_profiling.cc:243] Waited 779 ms on lock 0x7b54000f9454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:17.374042 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1790576000}
I20260709 13:08:17.381268 27263 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.411103 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.397651 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:17.423610 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260709 13:08:17.426267 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:08:17.427208 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:17.436602 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.437593 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.437011 26875 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.004s
I20260709 13:08:17.442250 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:08:17.445237 27220 log.cc:927] Time spent T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 1.085s	user 0.006s	sys 0.009s
I20260709 13:08:17.445694 26936 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260709 13:08:17.447455 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 554) took 2266 ms. Trace:
I20260709 13:08:17.428763 26996 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:17.438313 25287 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:17.449178 25244 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: []
W20260709 13:08:17.452845 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:17.454542 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:17.455367 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.456264 25239 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:08:17.456795 25287 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:17.458020 25287 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:17.462199 25287 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:17.468513 25244 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:08:17.431712 27237 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:17.471484 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.471372 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:17.461753 27247 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:08:17.474148 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:17.487457 27240 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.860s	user 0.009s	sys 0.011s
I20260709 13:08:17.466460 27063 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:08:17.468046 27117 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:17.490913 27263 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:17.461108 26875 consensus_queue.cc:1059] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 13:08:17.487856 25287 leader_election.cc:400] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4.
I20260709 13:08:17.458855 26996 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.466281 27271 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:17.429391 27111 consensus_queue.cc:260] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.467393 26936 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:17.500121 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.500903 25239 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
W20260709 13:08:17.495826 27240 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.898s	user 0.009s	sys 0.011s
I20260709 13:08:17.498729 25287 leader_election.cc:403] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 13:08:17.447880 25938 rpcz_store.cc:276] 0709 13:08:15.180678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:15.181667 (+   989us) service_pool.cc:224] Handling call
0709 13:08:15.182058 (+   391us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:15.182619 (+   561us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:15.182659 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:15.183348 (+   689us) log.cc:844] Serialized 26 byte log entry
0709 13:08:15.289553 (+106205us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:15.293159 (+  3606us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:15.299912 (+  6753us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:17.444625 (+2144713us) spinlock_profiling.cc:243] Waited 61.5 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:17.447046 (+  2421us) raft_consensus.cc:1685] finished
0709 13:08:17.447065 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:17.447279 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1383,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":6456,"spinlock_wait_cycles":141470976,"thread_start_us":70528,"threads_started":1,"wal-append.queue_time_us":1176026}
W20260709 13:08:17.541347 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 1.40627e+06us
I20260709 13:08:17.504220 25284 leader_election.cc:400] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4.
I20260709 13:08:17.505668 27260 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:17.509382 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:17.510316 27274 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:08:17.510942 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:17.512769 27144 raft_consensus.cc:3055] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:17.512701 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:17.501816 27237 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.515439 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.566026 27237 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:17.515713 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:17.566375 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.567097 25940 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:17.560811 27144 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:17.572559 25240 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:08:17.516896 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:17.573889 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 57096.5us
W20260709 13:08:17.526036 27278 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 13:08:17.574550 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:17.575680 25358 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:17.530759 25212 connection.cc:256] RPC call timeout handler was delayed by -1.1274623199999496s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1635s for a response
I20260709 13:08:17.538003 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.547031 25388 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:08:17.546983 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:17.548337 26996 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:17.548740 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:17.549271 26972 consensus_queue.cc:1059] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:17.548782 27111 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:17.551317 25316 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:08:17.557972 25393 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:17.562423 25239 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:17.563773 27285 consensus_queue.cc:1059] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:08:17.577983 25243 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:17.573002 25209 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:17.576939 25212 connection.cc:256] RPC call timeout handler was delayed by -1.0608623199999556s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17090000000000002s for a response
I20260709 13:08:17.580576 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.588622 25245 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 3: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:17.579605 27237 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:17.583318 25389 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:17.587010 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:17.610181 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.586205 26996 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:17.620074 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:17.623591 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:17.624682 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:17.621836 25943 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:17.614730 27165 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:17.622090 25937 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:17.633919 26925 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:08:17.637866 27248 log.cc:927] Time spent T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.920s	user 0.002s	sys 0.002s
I20260709 13:08:17.619987 26875 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:17.638734 26875 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:08:17.639163 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 151463us
I20260709 13:08:17.639269 26875 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:17.640288 25287 leader_election.cc:400] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4.
I20260709 13:08:17.633188 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:17.630460 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:17.642035 25940 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:17.642939 27165 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:17.644279 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:17.644865 25318 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:17.624320 27285 raft_consensus.cc:3055] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:17.645383 27285 raft_consensus.cc:740] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:17.646605 27285 consensus_queue.cc:260] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.647419 27247 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4.
I20260709 13:08:17.647910 27285 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:17.649330 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:17.650483 26925 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:17.651036 26925 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.653360 25285 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:17.653662 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 162061us
I20260709 13:08:17.653620 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:17.655117 26925 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:17.656984 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.619910 27263 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 13:08:17.661268 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:17.661880 25395 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:17.663811 25245 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:17.664978 27282 raft_consensus.cc:3055] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:17.665391 27282 raft_consensus.cc:740] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:17.665498 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:17.666159 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:17.666296 27282 consensus_queue.cc:260] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:17.666153 25940 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:17.667160 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:17.667663 27282 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:17.666797 25942 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:17.669492 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:17.670997 25318 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:17.671284 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:17.671558 25318 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 1782) took 1416 ms (client timeout 1528 ms). Trace:
I20260709 13:08:17.671783 25936 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 13:08:17.671927 25318 rpcz_store.cc:269] 0709 13:08:16.255347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:16.276477 (+ 21130us) service_pool.cc:224] Handling call
0709 13:08:17.644094 (+1367617us) spinlock_profiling.cc:243] Waited 1.65 ms on lock 0x7b58000e3498. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:17.671479 (+ 27385us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3796864}
I20260709 13:08:17.666800 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:17.674597 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:17.675266 25939 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
W20260709 13:08:17.678027 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:17.671015 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:17.672991 27260 consensus_queue.cc:1059] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:08:17.680703 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:17.686152 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:17.686980 25939 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:17.689709 27166 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:17.691826 27260 raft_consensus.cc:3055] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:17.692258 27260 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:17.693595 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.693463 27260 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.694487 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:17.695883 27260 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:17.697918 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:17.697976 25243 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
W20260709 13:08:17.699769 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 46519.8us
W20260709 13:08:17.725798 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 71430.4us
W20260709 13:08:17.727110 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 120101us
W20260709 13:08:17.732333 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 144276us
I20260709 13:08:17.738258 25359 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:17.744877 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:17.747397 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:17.748236 25943 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:17.750254 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:17.751339 25943 raft_consensus.cc:2368] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4.
I20260709 13:08:17.740429 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:17.755338 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 54641.9us
I20260709 13:08:17.756264 25285 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:17.704444 27165 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:17.758598 25285 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:17.698966 26996 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:17.757454 27271 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:17.760023 27145 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:17.760488 27159 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:17.761116 25284 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:17.761621 27249 consensus_queue.cc:1059] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:08:17.761910 25359 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:17.777505 27287 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:17.777879 27287 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:17.777925 25359 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:17.778337 27287 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:17.778870 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:17.762360 27286 consensus_queue.cc:1059] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:08:17.765480 25212 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:17.768024 27259 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:17.753994 27014 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:17.767856 25285 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:17.759704 27108 consensus_queue.cc:1059] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 13:08:17.774784 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 121237us
I20260709 13:08:17.782810 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:17.783367 25287 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:17.724345 26846 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:17.780928 25212 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:17.784186 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:17.769660 27166 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:17.796097 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:17.772516 27145 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:17.783582 27189 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:08:17.836601 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 55737.5us
I20260709 13:08:17.781234 25389 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:17.798935 27136 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:17.791894 26846 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:17.800827 26925 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:17.846166 27289 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:17.797466 27259 raft_consensus.cc:697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:17.847925 27247 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:17.851511 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:17.813095 27145 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:08:17.853287 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:17.850615 26875 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:17.773578 26996 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:17.872812 27299 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:17.873660 27299 consensus_queue.cc:1059] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:08:17.852638 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:17.874655 25387 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 3: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:17.876181 26996 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:17.877288 27282 raft_consensus.cc:3055] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:17.878376 26875 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:17.840075 27189 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:17.879264 27189 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:08:17.882717 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 121672us
W20260709 13:08:17.867646 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 88395.5us
I20260709 13:08:17.838001 26772 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:17.890376 25911 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:17.886780 25284 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:17.887490 25212 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:17.894253 25359 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:17.867410 26846 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:17.877645 27282 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:17.838922 27014 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20260709 13:08:17.900345 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:17.905812 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.892834 25284 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:17.868108 26925 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:17.909950 25243 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:17.910933 27296 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:17.910990 27259 consensus_queue.cc:237] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:17.911630 27296 raft_consensus.cc:697] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:17.897630 27300 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:17.895512 27263 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:17.899303 27159 raft_consensus.cc:697] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:17.921406 25239 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:17.922308 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.923880 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:17.906571 26772 raft_consensus.cc:697] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:08:17.920976 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 108990us
I20260709 13:08:17.924203 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.909165 27166 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:17.913196 26925 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:17.890448 27287 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:17.927088 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:17.869859 27289 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:17.917245 27247 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:17.933473 25312 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:08:17.930804 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:17.910858 25141 catalog_manager.cc:5697] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 2 to 4, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 4 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:08:17.932678 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.934509 25240 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:08:17.934505 25358 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:17.929908 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 146650us
I20260709 13:08:17.913012 27144 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.935827 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:17.936285 27144 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:17.921283 27217 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:17.937098 27159 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.937362 27217 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:17.917801 27300 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:17.941399 27300 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:17.941510 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.942269 25243 raft_consensus.cc:2393] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:17.940968 27217 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:17.941417 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:17.943598 25319 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:17.941625 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:17.945231 25316 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 1.
I20260709 13:08:17.943063 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.952601 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.953424 25239 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:17.954447 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.955137 25239 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 1.
I20260709 13:08:17.956102 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.958580 25243 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:17.957074 25239 raft_consensus.cc:2410] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:08:17.960150 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.951930 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:17.964965 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:17.924700 27263 raft_consensus.cc:697] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 4 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:17.967613 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:17.912709 27296 consensus_queue.cc:237] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.970686 25239 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:17.969185 25246 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:08:17.974165 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:17.975020 25244 raft_consensus.cc:2410] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:08:17.950291 27247 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:17.975778 27247 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:17.974291 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:17.998178 27247 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:17.967296 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:18.001017 25287 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:18.001375 27117 raft_consensus.cc:3055] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:17.973923 27249 consensus_queue.cc:1059] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260709 13:08:18.002743 25141 catalog_manager.cc:5697] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 1 to 3, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:08:17.972402 27014 consensus_queue.cc:1059] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260709 13:08:18.004221 27285 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:18.006259 25393 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:08:18.009836 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:18.022080 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:18.024864 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:18.028682 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:18.027557 25315 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:18.023739 25386 raft_consensus.cc:2393] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:18.014631 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:18.038183 27287 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:18.038316 25909 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:18.039139 27111 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:18.038977 27189 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:18.039897 27111 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:17.971238 27166 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:18.040268 27111 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:18.040625 27111 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:18.026312 27303 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 13:08:18.041874 27166 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:18.042670 27302 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:18.045285 27108 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:18.015869 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:18.045871 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:18.026825 26772 consensus_queue.cc:237] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:18.048457 27263 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:18.009276 27293 log.cc:927] Time spent T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 13:08:18.030861 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:18.017055 27117 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:18.011284 25209 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:18.061549 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 50350.5us
W20260709 13:08:18.076792 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:18.082037 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:18.051647 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 117209us
W20260709 13:08:18.086259 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 55505.5us
I20260709 13:08:18.097558 25287 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:18.020283 26972 consensus_queue.cc:1059] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 13:08:18.101329 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:18.102466 25209 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:18.177668 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:18.100207 27014 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:18.005796 27282 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:18.045786 27108 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:18.084729 27301 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.085s
I20260709 13:08:18.165202 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:18.171231 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 73745.9us
W20260709 13:08:18.171801 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 69337.8us
I20260709 13:08:18.158973 27268 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:08:18.181557 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:18.084515 27165 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:18.179183 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:18.110905 25239 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 13:08:18.228482 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 182858us
W20260709 13:08:18.183738 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:18.229357 25909 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:18.185175 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 297725us
W20260709 13:08:18.186725 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 84314.4us
I20260709 13:08:18.224795 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:18.193466 25359 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:18.237627 25209 leader_election.cc:400] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4.
I20260709 13:08:18.238154 25209 leader_election.cc:403] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:08:18.238595 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 59496us
I20260709 13:08:18.239548 25358 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:18.182533 27302 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:18.223208 27268 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:18.228302 27166 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:18.233490 27165 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:18.248029 26972 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:18.231433 26772 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260709 13:08:18.249702 27302 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:18.233788 27259 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260709 13:08:18.274542 26846 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:18.270360 27268 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:18.306842 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:18.310370 27144 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:18.310928 27144 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:18.284129 27296 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260709 13:08:18.274084 27166 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:08:18.283155 27237 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.016s	sys 0.016s
I20260709 13:08:18.283422 27165 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:18.322839 25317 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 3: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:18.294279 27247 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:18.289757 26972 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:18.333328 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 104052us
I20260709 13:08:18.313092 27144 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:18.303153 27302 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:18.379148 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 148243us
W20260709 13:08:18.380457 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 61918.2us
I20260709 13:08:18.291131 27320 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:18.411818 27320 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:18.412417 27320 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:18.412938 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:18.413635 25316 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
W20260709 13:08:18.314841 27309 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
W20260709 13:08:18.357059 27237 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.016s	sys 0.016s
I20260709 13:08:18.388836 25909 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:18.358076 27166 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:18.415819 27333 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:18.416373 25909 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:18.356385 27305 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:18.417276 27333 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:18.415378 27237 raft_consensus.cc:697] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:18.418035 25909 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:18.413580 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:18.384667 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 118208us
I20260709 13:08:18.419353 25319 raft_consensus.cc:2393] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 3: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:18.417878 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:18.317456 27289 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:18.384639 27287 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:18.317407 27310 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:18.389377 26972 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:18.354373 27217 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:18.357884 26875 consensus_queue.cc:1059] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.072s
W20260709 13:08:18.440071 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 200602us
W20260709 13:08:18.441781 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 211488us
I20260709 13:08:18.295185 27117 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:18.358924 27301 consensus_queue.cc:1048] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20260709 13:08:18.430868 27287 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:18.454375 25212 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:18.458284 25287 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:18.452843 27314 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 13:08:18.434320 27310 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:18.435842 27285 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:18.471917 25239 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:18.466670 27287 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:18.466456 27317 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:18.473245 27108 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:18.476322 27310 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:18.476374 27014 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:18.418676 27237 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:18.486347 25212 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:18.494346 25240 consensus_meta.cc:287] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.543s	user 0.007s	sys 0.004s
I20260709 13:08:18.495479 25240 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
W20260709 13:08:18.501920 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:18.473785 27317 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260709 13:08:18.453632 27312 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
W20260709 13:08:18.595921 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 109632us
W20260709 13:08:18.626204 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 208227us
W20260709 13:08:18.459506 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 46355.7us
I20260709 13:08:18.537784 27301 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:18.537949 26975 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:18.476994 27014 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:18.628949 25209 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:18.628731 25212 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:18.537149 27286 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:18.627038 25358 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.549s (SENT) [suppressed 25 similar messages]
I20260709 13:08:18.537376 27289 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4.
W20260709 13:08:18.565357 27322 log.cc:927] Time spent T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.248s	user 0.001s	sys 0.000s
I20260709 13:08:18.660874 25209 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:18.671000 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:18.672529 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:18.676817 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 197082us
I20260709 13:08:18.628610 27263 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:18.619799 27144 consensus_queue.cc:1059] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.103s
W20260709 13:08:18.664734 27293 log.cc:927] Time spent T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 13:08:18.564678 27268 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:18.682320 27111 consensus_meta.cc:287] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.638s	user 0.007s	sys 0.003s
I20260709 13:08:18.646458 27327 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:18.668258 27317 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:08:18.693897 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 236021us
W20260709 13:08:18.727778 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 100724us
I20260709 13:08:18.729557 25909 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:18.721939 27145 consensus_meta.cc:287] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.829s	user 0.010s	sys 0.000s
I20260709 13:08:18.679145 27301 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20260709 13:08:18.701020 27285 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:18.726441 27340 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
W20260709 13:08:18.744050 27321 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 13:08:18.745216 27268 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:18.720928 27317 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:18.698271 27263 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:18.722229 27111 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:18.722921 26846 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:18.677800 27286 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:18.772318 25909 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:18.444175 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:08:18.749722 27136 consensus_meta.cc:287] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.888s	user 0.010s	sys 0.000s
I20260709 13:08:18.798105 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:18.775288 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 146665us
I20260709 13:08:18.802062 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:18.798871 25943 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:18.780781 25141 catalog_manager.cc:5697] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:08:18.807261 25312 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:18.808880 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:18.809618 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:18.775179 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:18.794047 27317 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:18.840410 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:18.781685 27349 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:18.725387 27327 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:18.806051 27136 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:18.774801 27320 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:18.776953 27335 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:18.777176 27144 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:18.809468 25321 raft_consensus.cc:2376] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4. Re-sending same reply.
I20260709 13:08:18.817741 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:18.817842 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:18.855926 25320 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:18.818655 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:18.801580 25212 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:18.770047 27247 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.032s
W20260709 13:08:18.805843 27260 consensus_meta.cc:287] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 1.109s	user 0.005s	sys 0.007s
I20260709 13:08:18.787195 27217 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:18.848107 25316 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:18.848280 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:18.815392 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 142929us
I20260709 13:08:18.782109 27145 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:18.861829 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:18.831120 26846 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:18.795526 27285 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:18.844378 25141 catalog_manager.cc:5697] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:18.856855 25312 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
W20260709 13:08:18.848531 26925 consensus_meta.cc:287] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.897s	user 0.009s	sys 0.001s
W20260709 13:08:18.856488 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 229455us
I20260709 13:08:18.845820 27341 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:18.842561 27331 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:18.864810 25314 raft_consensus.cc:2393] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:18.870350 25320 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:18.879184 25237 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:18.879588 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 273) took 1197 ms. Trace:
I20260709 13:08:18.881222 25143 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39582 (request call id 305) took 6590 ms. Trace:
I20260709 13:08:18.881564 27145 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:18.879973 25388 rpcz_store.cc:276] 0709 13:08:17.682154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:17.704509 (+ 22355us) service_pool.cc:224] Handling call
0709 13:08:17.780804 (+ 76295us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:18.179494 (+398690us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:18.183399 (+  3905us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:18.282210 (+ 98811us) log.cc:844] Serialized 26 byte log entry
0709 13:08:18.628448 (+346238us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:18.630467 (+  2019us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:18.662653 (+ 32186us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:18.879366 (+216713us) raft_consensus.cc:1685] finished
0709 13:08:18.879391 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:18.879518 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":76218,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"thread_start_us":456214,"threads_started":2,"wal-append.queue_time_us":504488}
I20260709 13:08:18.882447 27327 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:18.883658 27331 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:18.884208 27331 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:18.884409 27285 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:18.883875 25141 catalog_manager.cc:5697] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 2 to 4, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 4 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:08:18.858768 27335 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:18.885515 25141 catalog_manager.cc:5697] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 4 to 6, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 6 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:08:18.886710 27335 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:18.887028 25141 catalog_manager.cc:5697] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:08:18.888226 27320 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:18.877249 27217 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:18.876096 26972 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:18.889377 27341 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:18.870297 27340 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:18.891927 27341 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:18.891795 27340 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:18.892930 27287 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:18.894246 27340 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:18.895570 27287 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:18.897399 27165 consensus_meta.cc:287] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.519s	user 0.002s	sys 0.008s
I20260709 13:08:18.897843 27165 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:18.888470 25141 catalog_manager.cc:5697] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 2 to 4, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 4 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:08:18.862133 27357 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:18.863353 27329 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:18.868752 27301 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:08:18.874917 27300 consensus_meta.cc:287] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.933s	user 0.012s	sys 0.000s
W20260709 13:08:18.858347 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 319499us
I20260709 13:08:18.889410 27217 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:18.906646 27263 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:18.906741 27357 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:18.928925 27285 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:18.929481 27285 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:18.929629 27217 consensus_queue.cc:1059] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:08:18.930517 27357 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:18.930676 27217 raft_consensus.cc:3055] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:18.931071 27217 raft_consensus.cc:740] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:18.931569 27285 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:18.931804 27217 consensus_queue.cc:260] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:18.932413 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 424) took 1021 ms. Trace:
I20260709 13:08:18.902065 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:18.911923 25358 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:18.871106 27014 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:18.967479 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:18.921192 25942 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:18.925458 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:18.926658 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:18.926832 27361 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:18.935847 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:18.933320 25312 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:08:18.983688 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:18.977090 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 75267.6us
I20260709 13:08:18.969312 25141 catalog_manager.cc:5697] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:18.877417 27342 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:18.968152 25395 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:18.999074 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:18.999699 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:18.974386 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:19.001253 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:19.002482 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:18.936383 27345 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.232s	user 0.001s	sys 0.000s
I20260709 13:08:18.908031 27287 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:18.877686 26925 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:18.941321 27299 raft_consensus.cc:3055] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:08:19.013293 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 101437us
I20260709 13:08:18.908610 27301 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:19.007030 25394 raft_consensus.cc:2393] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:18.919718 27354 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:18.936547 27165 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:18.995873 27361 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:18.933068 27217 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:18.937284 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:18.972103 27111 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:18.881585 25143 rpcz_store.cc:276] 0709 13:08:12.247615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:12.287721 (+ 40106us) service_pool.cc:224] Handling call
0709 13:08:18.818618 (+6530897us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:08:14.514507 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:08:14.587990 (+ 73483us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:08:14.590601 (+  2611us) write_op.cc:435] Acquired schema lock
0709 13:08:14.592254 (+  1653us) tablet.cc:662] Decoding operations
0709 13:08:14.642516 (+ 50262us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:08:14.660703 (+ 18187us) write_op.cc:641] Partition lock acquired for write op
0709 13:08:14.660760 (+    57us) tablet.cc:685] Acquiring locks for 5 operations
0709 13:08:14.661557 (+   797us) tablet.cc:701] Row locks acquired
0709 13:08:14.661585 (+    28us) write_op.cc:260] PREPARE: finished
0709 13:08:14.661771 (+   186us) write_op.cc:270] Start()
0709 13:08:14.661916 (+   145us) write_op.cc:276] Timestamp: P: 1783602494661721 usec, L: 0
0709 13:08:14.661945 (+    29us) op_driver.cc:348] REPLICATION: starting
0709 13:08:14.662873 (+   928us) log.cc:844] Serialized 2084 byte log entry
0709 13:08:15.608590 (+945717us) op_driver.cc:464] REPLICATION: finished
0709 13:08:15.899227 (+290637us) spinlock_profiling.cc:243] Waited 3.22 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f2973ba80ac 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297a6e8ef0
0709 13:08:16.229925 (+330698us) write_op.cc:301] APPLY: starting
0709 13:08:16.230080 (+   155us) tablet.cc:1367] starting BulkCheckPresence
0709 13:08:16.230353 (+   273us) tablet.cc:1370] finished BulkCheckPresence
0709 13:08:16.230402 (+    49us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:08:16.484607 (+254205us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:08:16.597152 (+112545us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0709 13:08:16.597250 (+    98us) write_op.cc:312] APPLY: finished
0709 13:08:16.597654 (+   404us) log.cc:844] Serialized 47 byte log entry
0709 13:08:17.393655 (+796001us) spinlock_profiling.cc:243] Waited 19 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f2973ba80ac 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b
0709 13:08:17.513222 (+119567us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:17.513379 (+   157us) write_op.cc:454] Released schema lock
0709 13:08:17.513438 (+    59us) write_op.cc:341] FINISH: Updating metrics
0709 13:08:17.513915 (+   477us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:17.513945 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":972380,"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":593735,"apply.run_cpu_time_us":24821,"apply.run_wall_time_us":1283986,"num_ops":5,"prepare.queue_time_us":804489,"prepare.run_cpu_time_us":18384,"prepare.run_wall_time_us":407439,"raft.queue_time_us":613708,"raft.run_cpu_time_us":54958,"raft.run_wall_time_us":449710,"replication_time_us":946369,"spinlock_wait_cycles":50992256,"thread_start_us":2173491,"threads_started":5,"wal-append.queue_time_us":1281842,"wal-append.run_cpu_time_us":58512,"wal-append.run_wall_time_us":2350873}]]}
I20260709 13:08:18.940254 27326 raft_consensus.cc:3055] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:19.015758 25358 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:19.016589 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:19.017445 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:19.006134 25209 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:08:19.054171 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 48106.1us
I20260709 13:08:19.052964 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:19.055333 25321 raft_consensus.cc:2393] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:19.018851 27301 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:19.007517 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:19.041581 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:19.042068 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:19.042309 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:19.043845 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:19.044593 25244 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:19.016675 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:18.972162 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:19.005260 27310 consensus_meta.cc:287] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.528s	user 0.009s	sys 0.001s
I20260709 13:08:19.015403 27287 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:19.057463 26925 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:19.058920 27166 consensus_meta.cc:287] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.643s	user 0.005s	sys 0.006s
I20260709 13:08:18.932896 25938 rpcz_store.cc:276] 0709 13:08:17.910982 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:17.911360 (+   378us) service_pool.cc:224] Handling call
0709 13:08:17.911711 (+   351us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:17.912086 (+   375us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:17.912123 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:17.912667 (+   544us) log.cc:844] Serialized 26 byte log entry
0709 13:08:18.094726 (+182059us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:18.098422 (+  3696us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:18.129830 (+ 31408us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:18.932146 (+802316us) raft_consensus.cc:1685] finished
0709 13:08:18.932167 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:18.932344 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1131,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":3126,"thread_start_us":181454,"threads_started":1,"wal-append.queue_time_us":1014462}
I20260709 13:08:19.073733 27286 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:19.075861 27310 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:19.087165 25938 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:19.087845 25388 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:19.059695 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:19.089226 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:19.063565 27165 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:19.089869 27165 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:19.090279 27165 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:19.021212 27335 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:19.060159 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:19.061048 25240 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:19.093708 25241 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:08:19.094249 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:18.943516 27300 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:18.914114 27329 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:19.013347 27299 raft_consensus.cc:740] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:08:19.066888 27308 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.752s	user 0.004s	sys 0.009s
I20260709 13:08:19.064242 25391 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
W20260709 13:08:19.067937 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 52271.1us
I20260709 13:08:19.069644 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:19.069839 25287 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:19.040668 27333 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:19.118659 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 317865us
I20260709 13:08:19.119105 25245 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 2: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:19.062994 27353 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:19.065017 25387 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:19.041111 27361 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:19.053261 27326 raft_consensus.cc:740] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:19.077394 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:19.085649 26925 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:19.053290 27285 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:19.136324 27285 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:19.099084 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:19.137952 25392 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:19.100375 25243 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:19.076969 25941 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:19.139643 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:19.140242 25241 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
W20260709 13:08:19.109400 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:19.117098 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:19.117241 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 47487.9us
I20260709 13:08:19.119572 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:19.120671 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:19.123075 27323 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:19.123212 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:19.123752 25942 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:19.124109 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:19.123965 27336 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:08:19.085649 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:19.124742 27353 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:08:19.124934 27329 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:19.125551 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:19.126433 27326 consensus_queue.cc:260] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:19.127506 25246 raft_consensus.cc:2393] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 2: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:19.123147 27368 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:19.128425 25313 raft_consensus.cc:1240] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: []
W20260709 13:08:19.128711 27302 consensus_meta.cc:287] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.698s	user 0.000s	sys 0.007s
I20260709 13:08:19.135006 27300 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:19.136230 26925 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:19.039369 27354 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:19.076804 27237 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:19.094899 27335 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:19.089489 27310 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:19.151151 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:19.152473 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:19.156030 27237 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:19.159399 27237 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:19.159507 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:19.159956 25358 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:19.145296 27333 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:19.161433 27355 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:19.162062 27333 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:19.162127 27355 consensus_queue.cc:1059] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:08:19.163244 27320 raft_consensus.cc:3055] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:19.098637 27166 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:19.163576 27320 raft_consensus.cc:740] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:19.165148 27299 consensus_queue.cc:260] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:19.166970 27189 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:19.167122 27354 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:19.167388 27300 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:19.154223 27302 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:19.169500 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:19.157778 27326 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:19.172515 25358 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:19.183339 25358 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:19.171357 25287 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:19.152616 27368 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:19.149592 27352 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:19.178138 25940 raft_consensus.cc:2393] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 2: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:19.158685 27335 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:19.184432 25241 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:19.184937 27310 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:19.121783 27317 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:19.188367 25318 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:19.192871 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:19.193686 25937 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:19.185628 27368 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:19.151363 27331 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:19.185885 27299 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:19.201139 27166 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:19.205283 27287 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:19.151973 27323 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:19.187484 27341 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:19.208688 27323 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:19.211129 27302 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:19.211748 27320 consensus_queue.cc:260] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:19.212818 27285 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:19.212913 27320 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:19.214102 26925 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:19.195946 27286 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:19.217927 27286 raft_consensus.cc:697] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:19.218796 27286 consensus_queue.cc:237] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:19.208738 27317 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:19.220834 27261 raft_consensus.cc:3055] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:19.220782 27301 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:19.221726 27261 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:19.222600 27261 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:19.223248 27301 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:19.223578 27261 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:08:19.206897 27324 log.cc:927] Time spent T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.579s	user 0.000s	sys 0.002s
I20260709 13:08:19.228750 27361 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:19.230046 27237 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:19.247226 25386 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 1792) took 1471 ms (client timeout 1643 ms). Trace:
W20260709 13:08:19.247838 25386 rpcz_store.cc:269] 0709 13:08:17.775995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:17.778572 (+  2577us) service_pool.cc:224] Handling call
0709 13:08:19.246930 (+1468358us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b5400283cd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:19.247115 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":438036352}
I20260709 13:08:19.255908 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:19.260077 25386 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:19.241848 27237 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:19.258001 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1270) took 1211 ms. Trace:
I20260709 13:08:19.259073 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:19.247769 27331 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:19.245029 27165 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:19.281620 27368 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:19.274238 25238 rpcz_store.cc:276] 0709 13:08:18.046214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:18.047587 (+  1373us) service_pool.cc:224] Handling call
0709 13:08:18.048099 (+   512us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:18.048578 (+   479us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:18.048626 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:18.049291 (+   665us) log.cc:844] Serialized 26 byte log entry
0709 13:08:18.294982 (+245691us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:18.303666 (+  8684us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:18.319393 (+ 15727us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:19.206237 (+886844us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:19.257674 (+ 51437us) raft_consensus.cc:1685] finished
0709 13:08:19.257703 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:19.257931 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":770,"raft.queue_time_us":1501,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":50806,"spinlock_wait_cycles":38025472,"thread_start_us":232549,"threads_started":1,"wal-append.queue_time_us":417346}
I20260709 13:08:19.335132 27305 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:19.350489 27305 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:19.347718 27335 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:19.332521 27259 consensus_queue.cc:1059] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260709 13:08:19.359858 27259 raft_consensus.cc:3055] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:19.360234 27259 raft_consensus.cc:740] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:08:19.409510 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:19.388706 27063 raft_consensus.cc:3055] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:19.390057 27355 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:19.424360 27368 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:19.423123 27063 raft_consensus.cc:740] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:19.455405 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:19.361326 27259 consensus_queue.cc:260] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:19.483808 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:19.479403 27303 raft_consensus.cc:3055] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:19.485741 27303 raft_consensus.cc:740] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:19.487341 26875 raft_consensus.cc:1064] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:08:19.497977 27271 raft_consensus.cc:3055] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:19.498446 27271 raft_consensus.cc:740] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:19.499325 27271 consensus_queue.cc:260] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:19.501657 26975 raft_consensus.cc:3055] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:19.502177 26975 raft_consensus.cc:740] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:19.503238 26975 consensus_queue.cc:260] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:19.509827 25937 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:19.723380 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:19.536074 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:19.725435 25319 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:19.535461 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:19.727154 25312 raft_consensus.cc:2393] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:19.728175 25388 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:19.731272 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:19.731393 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:19.732532 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:19.722361 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:19.732079 25315 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:19.449299 27335 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:19.747354 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:19.749143 25141 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.005s	sys 0.038s
I20260709 13:08:19.510011 25941 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: []
W20260709 13:08:19.751693 25141 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.005s	sys 0.038s
W20260709 13:08:19.754601 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:19.460682 27342 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:19.768395 25909 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:19.777910 27261 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:19.777969 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:19.778504 27261 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:19.778995 27261 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:19.779500 27266 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:19.574862 25939 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:19.784840 25239 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:19.573894 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:19.511127 25936 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:08:19.735051 27271 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:19.785183 25245 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 13:08:19.801002 25358 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.012s	sys 0.033s
I20260709 13:08:19.743826 27331 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:19.741166 26972 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:19.758795 26975 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:08:19.791359 27320 consensus_meta.cc:287] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.562s	user 0.005s	sys 0.006s
W20260709 13:08:19.822384 25358 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.013s	sys 0.033s
I20260709 13:08:19.769527 27333 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:19.844602 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:19.846342 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:19.846606 25237 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:19.849406 27165 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:19.849804 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:19.813308 25386 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.014s	sys 0.035s
W20260709 13:08:19.862941 25393 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.678s	user 0.012s	sys 0.064s
W20260709 13:08:19.885192 25393 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.013s	sys 0.064s
W20260709 13:08:19.873692 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 690332us
I20260709 13:08:19.886204 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:19.873881 25212 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.753s	user 0.042s	sys 0.058s
W20260709 13:08:19.887622 25212 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.042s	sys 0.058s
I20260709 13:08:19.887428 25358 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:19.887921 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 767127us
W20260709 13:08:19.843572 27341 consensus_meta.cc:287] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.626s	user 0.005s	sys 0.005s
I20260709 13:08:19.690485 27063 consensus_queue.cc:260] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:19.856694 27361 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:19.850186 26772 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.050s
W20260709 13:08:19.885083 25386 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.014s	sys 0.035s
I20260709 13:08:19.910120 27320 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:08:19.976936 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:19.908663 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:19.980365 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:19.777488 27342 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:19.920991 27165 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:19.910516 27286 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:20.010838 27320 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:20.015743 25245 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:19.917371 27341 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:19.963156 27063 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:19.984724 27311 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:20.148434 25320 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:19.916864 27335 consensus_queue.cc:1059] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.205s
I20260709 13:08:20.095093 27326 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:19.972348 27159 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.157s
I20260709 13:08:20.114347 27320 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:20.112007 27342 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:20.124270 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:20.150753 27361 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:19.887408 26875 consensus_queue.cc:237] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:20.115401 27165 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:20.228756 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:20.273710 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:20.320645 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:20.321799 27378 log.cc:927] Time spent T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.310s	user 0.000s	sys 0.001s
I20260709 13:08:20.322129 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 1804) took 1520 ms. Trace:
W20260709 13:08:20.322779 27380 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.218s	user 0.000s	sys 0.000s
I20260709 13:08:20.322829 25313 rpcz_store.cc:276] 0709 13:08:18.802005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:18.805878 (+  3873us) service_pool.cc:224] Handling call
0709 13:08:18.852915 (+ 47037us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:19.154162 (+301247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:20.321743 (+1167581us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b540005df54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:20.322053 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":627696000}
W20260709 13:08:20.324676 27305 consensus_meta.cc:287] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.974s	user 0.000s	sys 0.011s
I20260709 13:08:20.325088 27305 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:20.327344 27305 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:20.327904 27305 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:20.328375 27305 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:20.328624 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1530) took 1258 ms. Trace:
I20260709 13:08:20.329183 25241 rpcz_store.cc:276] 0709 13:08:19.070408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:19.153879 (+ 83471us) service_pool.cc:224] Handling call
0709 13:08:19.154145 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:19.187363 (+ 33218us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:20.328310 (+1140947us) spinlock_profiling.cc:243] Waited 30.1 ms on lock 0x7b54001f63d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:20.328545 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":69149440}
I20260709 13:08:20.331635 25241 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:08:20.335706 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1269) took 2325 ms. Trace:
I20260709 13:08:20.336223 25244 rpcz_store.cc:276] 0709 13:08:18.010085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:18.029689 (+ 19604us) service_pool.cc:224] Handling call
0709 13:08:18.056253 (+ 26564us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:19.065823 (+1009570us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:20.335405 (+1269582us) spinlock_profiling.cc:243] Waited 984 ms on lock 0x7b540028bed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:20.335627 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2262950784}
I20260709 13:08:20.337858 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1164) took 1280 ms. Trace:
I20260709 13:08:20.338287 27305 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:20.338984 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:20.339392 27305 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:20.339653 25317 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
W20260709 13:08:20.340637 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:20.339876 27305 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:20.342185 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1172) took 1283 ms. Trace:
I20260709 13:08:20.342948 27305 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:20.338384 25387 rpcz_store.cc:276] 0709 13:08:19.057041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:19.149119 (+ 92078us) service_pool.cc:224] Handling call
0709 13:08:19.149510 (+   391us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:19.149726 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:19.149752 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:19.153810 (+  4058us) log.cc:844] Serialized 26 byte log entry
0709 13:08:19.323959 (+170149us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:19.324023 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:19.324130 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:20.337684 (+1013554us) raft_consensus.cc:1685] finished
0709 13:08:20.337703 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:20.337800 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":168111,"threads_started":1,"wal-append.queue_time_us":769401}
I20260709 13:08:20.342562 25392 rpcz_store.cc:276] 0709 13:08:19.059097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:19.389300 (+330203us) service_pool.cc:224] Handling call
0709 13:08:20.341555 (+952255us) spinlock_profiling.cc:243] Waited 6.89 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:20.341732 (+   177us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:20.341897 (+   165us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:20.341921 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:20.341959 (+    38us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:20.341991 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:20.342034 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:20.342130 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15844096}
I20260709 13:08:20.337858 25943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 425) took 1516 ms. Trace:
I20260709 13:08:20.345150 25943 rpcz_store.cc:276] 0709 13:08:18.821063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:18.837657 (+ 16594us) service_pool.cc:224] Handling call
0709 13:08:18.841098 (+  3441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:18.972403 (+131305us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:18.972446 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:18.972504 (+    58us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:18.972556 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:20.337656 (+1365100us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b5400702bd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:20.337695 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:20.337800 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":301447040}
I20260709 13:08:20.348619 25284 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:20.367120 25284 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:20.200933 26772 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:20.370581 25359 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:20.177664 27326 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:20.184708 27311 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:20.366959 27307 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:20.350277 27014 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:20.233352 27361 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:20.378525 26996 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:20.429884 25908 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.695s	user 0.005s	sys 0.004s
W20260709 13:08:20.430992 25908 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.005s	sys 0.005s
W20260709 13:08:20.431571 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 697201us
W20260709 13:08:20.432475 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:20.370299 27348 consensus_queue.cc:1059] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260709 13:08:20.450676 25245 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 1. Current term is 2. Ops: []
W20260709 13:08:20.460217 25285 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.012s	sys 0.056s
W20260709 13:08:20.461791 25285 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.012s	sys 0.056s
W20260709 13:08:20.462080 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 612808us
I20260709 13:08:20.462352 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:20.436780 27392 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:20.468475 25316 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:20.469048 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:20.470208 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:20.472476 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:20.456805 27368 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.011s	sys 0.056s
I20260709 13:08:20.495528 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:20.496203 25319 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:20.473901 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:20.497138 25316 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:08:20.433640 25938 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.005s	sys 0.010s
I20260709 13:08:20.385324 26772 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:20.469631 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:20.498440 25909 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.719s	user 0.031s	sys 0.110s
W20260709 13:08:20.499152 25909 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.031s	sys 0.110s
W20260709 13:08:20.499456 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 720206us
I20260709 13:08:20.435627 27165 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:20.503676 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:20.465338 27348 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:20.504684 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:20.398940 27307 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:20.495872 27368 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.012s	sys 0.056s
I20260709 13:08:20.432358 27014 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:20.438159 26996 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:20.516767 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:20.519043 25393 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.002s	sys 0.015s
W20260709 13:08:20.519527 25393 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.002s	sys 0.015s
I20260709 13:08:20.510782 27165 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:20.523540 25212 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.009s	sys 0.009s
W20260709 13:08:20.523998 25212 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.009s	sys 0.009s
W20260709 13:08:20.498658 25938 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.005s	sys 0.011s
W20260709 13:08:20.527935 25942 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.733s	user 0.056s	sys 0.137s
W20260709 13:08:20.528335 25942 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.744s	user 0.057s	sys 0.137s
W20260709 13:08:20.529846 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:20.524317 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 635832us
I20260709 13:08:20.522078 27165 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:20.515231 27364 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:08:20.520072 27271 consensus_meta.cc:287] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.682s	user 0.010s	sys 0.000s
I20260709 13:08:20.510265 27348 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:08:20.541576 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 171063us
W20260709 13:08:20.525163 27334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.659s	user 0.005s	sys 0.011s
I20260709 13:08:20.498410 27392 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:20.530758 27381 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.217s	user 0.000s	sys 0.001s
W20260709 13:08:20.534050 25240 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.765s	user 0.001s	sys 0.017s
I20260709 13:08:20.544130 25359 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
W20260709 13:08:20.559628 25240 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.791s	user 0.001s	sys 0.017s
I20260709 13:08:20.537460 25212 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:20.551498 27117 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:20.552413 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:20.543882 27348 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:20.542295 27364 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:20.545902 27334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.005s	sys 0.011s
W20260709 13:08:20.543052 25238 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.011s	sys 0.007s
I20260709 13:08:20.588315 25316 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:20.588557 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:20.545477 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:20.597863 25245 raft_consensus.cc:2393] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:08:20.585357 27390 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.201s	user 0.000s	sys 0.001s
I20260709 13:08:20.584167 27305 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:20.585518 25936 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 13:08:20.616106 27389 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260709 13:08:20.617072 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:20.617689 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1166) took 1560 ms. Trace:
I20260709 13:08:20.617777 25936 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:20.618886 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:20.618211 25386 rpcz_store.cc:276] 0709 13:08:19.057572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:19.260823 (+203251us) service_pool.cc:224] Handling call
0709 13:08:19.272744 (+ 11921us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:19.275453 (+  2709us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:19.275489 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:19.278889 (+  3400us) log.cc:844] Serialized 26 byte log entry
0709 13:08:19.985386 (+706497us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:20.016521 (+ 31135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:20.090198 (+ 73677us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:20.617443 (+527245us) raft_consensus.cc:1685] finished
0709 13:08:20.617469 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:20.617613 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":724,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":1831,"thread_start_us":645865,"threads_started":1,"wal-append.queue_time_us":1170327}
I20260709 13:08:20.620152 25321 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:20.621559 27305 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:20.623956 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:20.624616 25936 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:20.624382 27320 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:20.609838 27311 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:20.625586 27311 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:20.626062 27307 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:20.626125 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:20.626941 25245 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:20.627513 27400 consensus_queue.cc:1059] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:08:20.586678 27326 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:20.628486 27326 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:20.628536 27320 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:20.586989 27364 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:20.629076 27326 raft_consensus.cc:697] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:20.630277 27326 consensus_queue.cc:237] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:20.615517 25940 raft_consensus.cc:2393] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:08:20.588466 25238 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.011s	sys 0.008s
I20260709 13:08:20.627789 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:20.637527 25245 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:20.639158 25245 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:08:20.641650 27403 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:20.642206 27403 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:20.642628 27403 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:20.639143 26996 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:20.655866 26975 consensus_meta.cc:287] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.797s	user 0.011s	sys 0.000s
I20260709 13:08:20.650269 27361 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:20.644699 27394 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:20.734664 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 233764us
I20260709 13:08:20.737334 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 568) took 1229 ms. Trace:
I20260709 13:08:20.737902 25942 rpcz_store.cc:276] 0709 13:08:19.508000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:19.512812 (+  4812us) service_pool.cc:224] Handling call
0709 13:08:19.728181 (+215369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:19.736260 (+  8079us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:08:19.784097 (+ 47837us) log.cc:844] Serialized 14 byte log entry
0709 13:08:20.736914 (+952817us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:20.736969 (+    55us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:20.737019 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:20.737105 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:20.737271 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":32485,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"thread_start_us":952388,"threads_started":2}
W20260709 13:08:20.755712 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:20.761180 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 297838us
W20260709 13:08:20.763146 25287 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.879s	user 0.000s	sys 0.018s
W20260709 13:08:20.763593 25287 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.879s	user 0.000s	sys 0.018s
I20260709 13:08:20.786171 25285 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:20.737636 26975 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:20.739862 27361 raft_consensus.cc:3055] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:08:20.800258 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 256251us
I20260709 13:08:20.801275 25359 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:20.765002 27108 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:20.802865 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 265487us
I20260709 13:08:20.805210 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:20.819073 25316 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:20.819005 25212 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:20.820329 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:20.805907 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:20.806599 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:20.806820 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 246015us
I20260709 13:08:20.822938 25316 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:20.823616 25359 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:20.791098 27307 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:20.790344 26975 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:20.828477 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:20.829047 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:20.830470 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:20.836287 25212 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:20.837388 25212 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:20.844120 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:20.844820 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:20.804849 27271 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:20.852396 25359 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:20.850147 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:20.745669 27394 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:20.850898 25358 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:20.802289 27361 raft_consensus.cc:740] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:08:20.887961 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 340530us
I20260709 13:08:20.824739 27402 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:20.805752 27159 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
W20260709 13:08:20.891168 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 96202.6us
I20260709 13:08:20.828650 27307 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:20.825147 26772 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:20.893292 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:20.865806 27383 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.009s	sys 0.020s
W20260709 13:08:20.905510 27383 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.009s	sys 0.020s
W20260709 13:08:20.905625 27395 log.cc:927] Time spent T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
I20260709 13:08:20.826318 26996 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:20.895561 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:20.825282 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:20.830132 27404 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:20.850751 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 574) took 1329 ms. Trace:
I20260709 13:08:20.849632 27326 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:20.894862 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:20.855149 27165 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:20.910961 25319 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:20.840251 27405 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:20.888242 27014 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:20.915534 26996 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:20.917658 27286 raft_consensus.cc:3055] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:20.919845 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:20.889248 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1275) took 1165 ms. Trace:
I20260709 13:08:20.846745 27320 consensus_queue.cc:1059] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20260709 13:08:20.910008 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1276) took 1186 ms. Trace:
I20260709 13:08:20.896536 27392 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:20.903250 27307 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:20.955785 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:20.826258 27108 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:20.920823 27404 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:20.938376 26772 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260709 13:08:20.865859 27271 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:21.108846 25319 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:21.003949 25287 leader_election.cc:336] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.005s (SENT)
I20260709 13:08:20.912068 27416 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:08:21.049094 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:20.959596 25240 rpcz_store.cc:276] 0709 13:08:19.723841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:19.723941 (+   100us) service_pool.cc:224] Handling call
0709 13:08:19.724241 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:19.724439 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:08:20.560156 (+835717us) log.cc:844] Serialized 14 byte log entry
0709 13:08:20.909550 (+349394us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:20.909618 (+    68us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:20.909692 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:20.909793 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:20.909937 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":827286,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":81382,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":95492,"thread_start_us":1140205,"threads_started":3}
I20260709 13:08:21.117115 25287 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:21.119470 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 282126us
I20260709 13:08:21.138691 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:21.139796 25319 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:21.140569 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:21.120277 25908 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:21.141531 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:21.151388 25240 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:21.155705 25319 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:21.156536 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:21.157366 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:21.018296 27392 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:21.099951 26936 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:21.158605 27392 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:21.099946 27117 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:20.941694 27419 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:21.157428 25240 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:21.158622 26936 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:21.079595 27415 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:21.158792 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1712) took 2801 ms. Trace:
I20260709 13:08:21.159335 27346 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:21.157828 25319 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:21.186770 25318 rpcz_store.cc:276] 0709 13:08:18.357242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:18.379727 (+ 22485us) service_pool.cc:224] Handling call
0709 13:08:18.409503 (+ 29776us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:19.189895 (+780392us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:21.158584 (+1968689us) spinlock_profiling.cc:243] Waited 779 ms on lock 0x7b540002c6d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:21.158739 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1790655360}
W20260709 13:08:20.967020 27261 consensus_meta.cc:287] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 1.181s	user 0.014s	sys 0.001s
W20260709 13:08:21.141880 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 234123us
I20260709 13:08:20.957429 27165 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:20.984723 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:21.177000 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 173093us
I20260709 13:08:20.957037 27286 raft_consensus.cc:740] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:21.116569 27404 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:21.187997 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:20.959892 27014 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:21.135017 27063 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:21.197458 25287 leader_election.cc:336] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.010s (SENT)
I20260709 13:08:21.218206 25287 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:21.220640 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:20.954833 27326 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:21.206068 25319 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:21.206233 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:20.935835 25938 rpcz_store.cc:276] 0709 13:08:19.510145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:19.511359 (+  1214us) service_pool.cc:224] Handling call
0709 13:08:19.511732 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:19.512001 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:08:19.512386 (+   385us) log.cc:844] Serialized 14 byte log entry
0709 13:08:20.536190 (+1023804us) log.cc:844] Serialized 14 byte log entry
0709 13:08:20.803410 (+267220us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:20.806589 (+  3179us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:20.827805 (+ 21216us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:20.827906 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:20.838997 (+ 11091us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":910,"prepare.queue_time_us":20071,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":501908,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":7359,"thread_start_us":1205885,"threads_started":3,"wal-append.queue_time_us":705265}
I20260709 13:08:21.187005 27410 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:21.222659 27423 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 13:08:20.959365 25238 rpcz_store.cc:276] 0709 13:08:19.723516 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:19.729369 (+  5853us) service_pool.cc:224] Handling call
0709 13:08:19.830436 (+101067us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:19.874574 (+ 44138us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:08:20.633445 (+758871us) log.cc:844] Serialized 14 byte log entry
0709 13:08:20.856424 (+222979us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:20.888819 (+ 32395us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:20.888914 (+    95us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:20.889010 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:20.889186 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":725007,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":100129,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":9820,"thread_start_us":872695,"threads_started":2,"wal-append.queue_time_us":285192}
I20260709 13:08:21.194554 27415 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:21.223714 27432 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:21.226864 27368 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:08:21.227746 27286 consensus_queue.cc:260] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:21.228464 27403 consensus_meta.cc:287] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.585s	user 0.008s	sys 0.002s
I20260709 13:08:21.279078 27403 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:21.279764 27286 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:21.230171 27331 consensus_meta.cc:287] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.379s	user 0.012s	sys 0.000s
I20260709 13:08:21.230839 27108 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:21.230895 25939 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:08:21.232051 26996 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:21.186182 27117 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:21.233495 27271 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:21.233225 27014 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:21.233618 27412 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:21.194940 27299 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:21.233997 27342 consensus_meta.cc:287] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.004s	user 0.010s	sys 0.002s
I20260709 13:08:20.959417 27361 consensus_queue.cc:260] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:21.277493 25319 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:08:21.217831 26975 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:21.233973 27063 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:21.283998 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 163789us
I20260709 13:08:21.256898 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:21.196076 27261 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:21.291363 27436 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:21.195951 27346 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:21.292048 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 398803us
I20260709 13:08:21.297638 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:21.288303 27320 raft_consensus.cc:3055] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 13:08:21.305445 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:21.306166 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 425639us
I20260709 13:08:21.290071 27331 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:21.308611 25908 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:21.309269 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 458436us
I20260709 13:08:21.311259 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:21.267839 27440 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:08:21.313335 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:21.330228 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:21.297335 27398 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:21.298214 27403 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:21.321695 25209 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:21.323472 25911 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:21.326309 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:21.312779 27417 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20260709 13:08:21.305056 27409 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:21.309098 26875 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:21.334892 25944 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 427) took 2502 ms (client timeout 2186 ms). Trace:
I20260709 13:08:21.304039 27430 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:21.275743 27410 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:21.313169 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:21.336308 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:21.384368 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 75833.7us
W20260709 13:08:21.411403 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 89788.6us
W20260709 13:08:21.411399 25944 rpcz_store.cc:269] 0709 13:08:18.832398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:18.832640 (+   242us) service_pool.cc:224] Handling call
0709 13:08:21.297442 (+2464802us) spinlock_profiling.cc:243] Waited 4.35 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:21.334789 (+ 37347us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10007424}
I20260709 13:08:21.412685 25908 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:21.413043 25209 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:08:21.385377 25129 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet eda233ace2a24b379fef320a5f610c7a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 46de6919f85341c6a9b574dff112e35a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:21.303488 27299 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:21.367868 25245 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
W20260709 13:08:21.447667 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 108892us
W20260709 13:08:21.447911 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 121717us
I20260709 13:08:21.448490 27444 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:21.289393 27432 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:21.322566 27320 raft_consensus.cc:740] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:21.562719 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:21.563872 26996 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:21.507819 25287 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:21.303650 27412 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:21.565201 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:21.546857 27165 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:21.565881 27437 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:21.566401 27437 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:21.567044 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:21.315107 27368 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.178s
I20260709 13:08:21.329929 27440 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:21.389674 27411 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:21.496308 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:21.533501 25244 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:21.563414 27320 consensus_queue.cc:260] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:21.307821 27342 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:21.534543 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1281) took 1731 ms. Trace:
W20260709 13:08:21.459131 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 135570us
I20260709 13:08:21.329293 27425 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:21.525609 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:21.568322 27364 consensus_meta.cc:287] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.939s	user 0.011s	sys 0.000s
I20260709 13:08:21.595979 25240 raft_consensus.cc:2410] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20260709 13:08:21.384189 27424 log.cc:927] Time spent T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
W20260709 13:08:21.382771 27413 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 13:08:21.385543 26975 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:21.383296 27441 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:21.423949 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:08:21.640024 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 216138us
I20260709 13:08:21.641625 27414 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:21.642241 27414 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:21.642772 27414 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:21.646157 27454 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:21.593695 25237 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
W20260709 13:08:21.585283 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 77587.6us
I20260709 13:08:21.397145 27415 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:21.522836 25238 raft_consensus.cc:2393] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:21.433754 27331 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election won for term 2
W20260709 13:08:21.589946 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 93763.9us
I20260709 13:08:21.679385 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:21.680362 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:21.531595 27346 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:21.509630 27409 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:08:21.476377 27348 consensus_meta.cc:287] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.873s	user 0.010s	sys 0.000s
I20260709 13:08:21.689975 27409 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:21.690418 27409 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:21.690243 27348 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:21.691866 27305 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:21.690523 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:21.593694 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 263422us
I20260709 13:08:21.385360 27438 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:21.530807 27282 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:21.595171 25239 rpcz_store.cc:276] 0709 13:08:19.803492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:19.803882 (+   390us) service_pool.cc:224] Handling call
0709 13:08:21.533582 (+1729700us) spinlock_profiling.cc:243] Waited 5.22 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:21.533830 (+   248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:21.534092 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:21.534136 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:21.534195 (+    59us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:21.534257 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:21.534331 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:21.534464 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12007296}
I20260709 13:08:21.503494 27436 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:21.415258 27416 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:21.532418 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:21.567083 27412 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:21.549649 27063 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:21.460244 27448 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:21.549429 27326 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:21.548605 27261 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:21.547770 27014 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:21.460536 27443 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:21.751328 27443 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:21.751714 27443 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:21.583675 26996 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:21.630455 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:21.631644 25394 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 13:08:21.542775 27410 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:21.383536 27439 consensus_queue.cc:1059] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.212s
I20260709 13:08:21.383579 27447 consensus_queue.cc:1059] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.212s
I20260709 13:08:21.649787 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:21.805095 25244 raft_consensus.cc:2393] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:21.590978 27440 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:21.738937 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:21.671085 25209 connection.cc:256] RPC call timeout handler was delayed by -1.0962026089999881s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1645s for a response
I20260709 13:08:21.810444 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:21.811129 25244 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
W20260709 13:08:21.811167 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:21.811861 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:21.812573 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:21.451375 26875 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:21.593032 27435 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:21.678115 27318 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:21.591061 25392 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:21.359865 27398 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:21.586966 27408 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:21.680980 25315 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:21.446904 27430 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:21.558275 27404 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:21.677603 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:21.823228 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:21.824204 25392 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 3: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:08:21.824127 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 146605us
I20260709 13:08:21.687543 27434 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:21.646741 27454 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:21.637595 27364 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:21.687811 27455 consensus_queue.cc:1059] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 13:08:21.688494 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1717) took 1232 ms. Trace:
I20260709 13:08:21.595566 27411 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:21.692911 25318 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:21.737812 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:21.740744 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 47216.1us
I20260709 13:08:21.673890 27451 raft_consensus.cc:3055] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:21.681514 27403 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:21.745699 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 436) took 1213 ms. Trace:
I20260709 13:08:21.748354 27416 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:21.748939 27388 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 1.375s	user 0.000s	sys 0.003s
I20260709 13:08:21.745883 27436 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:21.750063 27448 raft_consensus.cc:697] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:21.750524 27326 consensus_queue.cc:1059] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:21.683669 27320 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:21.649636 26975 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:21.677380 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:21.753943 25387 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:21.683974 27331 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 2 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:21.595022 27429 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:21.741406 27299 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:21.813252 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:21.696257 27348 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:21.815572 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 316829us
I20260709 13:08:21.966086 25394 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:21.818416 25360 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.001s	sys 0.011s
I20260709 13:08:21.819254 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:21.967254 25360 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.779s	user 0.001s	sys 0.011s
W20260709 13:08:21.967619 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 148455us
I20260709 13:08:21.968526 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:21.630119 27425 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:21.960515 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:21.820776 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:21.821282 25243 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.032s	sys 0.045s
I20260709 13:08:21.804306 27437 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:21.671126 27441 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:21.826148 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 146623us
I20260709 13:08:21.830761 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:21.815863 27442 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:21.837880 25239 raft_consensus.cc:2393] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:21.838313 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:21.993527 25315 raft_consensus.cc:2393] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:08:21.973843 25243 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.032s	sys 0.045s
I20260709 13:08:21.993935 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 1826) took 1186 ms. Trace:
I20260709 13:08:21.994285 25315 rpcz_store.cc:276] 0709 13:08:20.809419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:21.828822 (+1019403us) service_pool.cc:224] Handling call
0709 13:08:21.993880 (+165058us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:21.803932 26996 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:21.824908 27382 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:21.911114 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:21.909693 25319 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:21.827003 27410 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:21.834017 27318 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.007535 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:21.995507 25392 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:21.998734 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:21.837090 27447 raft_consensus.cc:3055] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:21.838188 27451 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:22.019631 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.016603 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:22.019891 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:22.021211 25315 raft_consensus.cc:2393] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:22.019665 25239 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:08:21.973090 27305 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:21.992813 27289 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:22.020457 25944 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:22.024169 25318 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:22.027936 25239 raft_consensus.cc:1275] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:22.026945 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 1831) took 1180 ms. Trace:
I20260709 13:08:22.026986 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:22.028487 25315 rpcz_store.cc:276] 0709 13:08:20.846485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.001370 (+1154885us) service_pool.cc:224] Handling call
0709 13:08:22.026884 (+ 25514us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:21.997193 27382 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:21.916893 27403 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:21.910226 25941 rpcz_store.cc:276] 0709 13:08:20.532200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:20.533303 (+  1103us) service_pool.cc:224] Handling call
0709 13:08:21.532183 (+998880us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:21.745624 (+213441us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26401536}
I20260709 13:08:22.038476 25392 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:21.971645 27063 consensus_queue.cc:1059] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
W20260709 13:08:22.042368 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 209259us
W20260709 13:08:22.022040 27286 consensus_meta.cc:287] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.742s	user 0.010s	sys 0.002s
I20260709 13:08:21.838557 27434 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.002202 27442 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.025736 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:22.045892 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 80689us
I20260709 13:08:22.011014 27342 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.046842 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.012138 27447 raft_consensus.cc:740] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:22.052398 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:22.072716 25315 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:22.034735 27289 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:22.050019 25389 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260709 13:08:22.053167 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 134165us
I20260709 13:08:21.962803 27448 consensus_queue.cc:237] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
W20260709 13:08:22.065692 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:22.037951 27455 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:08:22.068115 25358 leader_election.cc:336] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.692s (SENT)
I20260709 13:08:22.069839 25392 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:22.035781 27432 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:22.044315 27382 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:21.835891 27435 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:22.076264 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 103917us
W20260709 13:08:22.042919 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 73096.9us
W20260709 13:08:22.084854 27452 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.268s	user 0.001s	sys 0.000s
I20260709 13:08:22.037770 27412 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260709 13:08:22.085387 25358 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:22.089314 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 72813us
W20260709 13:08:22.098784 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:22.099342 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:22.098711 25908 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:22.099701 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:22.037601 27346 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260709 13:08:22.079248 27289 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:22.036479 27429 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 13:08:22.076267 27454 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.082129 27329 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:22.101423 25908 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:22.101500 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:22.090186 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:22.104597 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:22.088752 27455 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:22.108980 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:21.833688 25313 rpcz_store.cc:276] 0709 13:08:20.456165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:20.457358 (+  1193us) service_pool.cc:224] Handling call
0709 13:08:20.507950 (+ 50592us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:20.554456 (+ 46506us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:20.561225 (+  6769us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:20.637803 (+ 76578us) log.cc:844] Serialized 26 byte log entry
0709 13:08:20.886977 (+249174us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:20.889339 (+  2362us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:21.626378 (+737039us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b54000889d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:21.629988 (+  3610us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:21.688167 (+ 58179us) raft_consensus.cc:1685] finished
0709 13:08:21.688201 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:21.688412 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":48262,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":10496,"spinlock_wait_cycles":50695424,"thread_start_us":212114,"threads_started":1,"wal-append.queue_time_us":508257}
I20260709 13:08:22.089715 27342 consensus_queue.cc:1059] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20260709 13:08:22.125247 25212 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:22.128208 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:22.100337 27404 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:22.130113 25392 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:22.130950 25908 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:22.108665 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:22.132332 27441 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:22.136706 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:22.138230 27014 raft_consensus.cc:3055] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:22.130949 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:22.103729 27261 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:22.145162 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 63908us
I20260709 13:08:22.110060 27329 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:22.154244 25394 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:22.156337 27329 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:22.156845 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.157470 25394 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:22.110321 27364 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.158298 27471 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:22.159343 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:22.158962 27471 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.159592 25936 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:22.158349 25911 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:22.160722 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:22.161938 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:22.161861 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:22.162197 27471 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:22.123828 27455 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:22.110427 27430 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:22.110919 27398 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.089538 27331 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.166038 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:22.108734 27327 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:22.187548 25941 raft_consensus.cc:2368] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4.
I20260709 13:08:22.192386 25394 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:22.143580 27459 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:22.194082 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.194757 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.195639 25242 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:22.162765 27471 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:22.145146 26875 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.166536 25386 raft_consensus.cc:2393] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:22.199113 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:22.201220 25944 raft_consensus.cc:2410] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 13:08:22.155323 27346 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:22.196462 27341 raft_consensus.cc:3055] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:22.155613 27014 raft_consensus.cc:740] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:22.104912 27464 consensus_queue.cc:1059] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.099s
I20260709 13:08:22.188071 27441 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:22.193903 27473 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:08:22.199473 27384 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.855s	user 0.013s	sys 0.008s
I20260709 13:08:22.206187 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:22.153976 27447 consensus_queue.cc:260] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.219509 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1277) took 2481 ms. Trace:
I20260709 13:08:22.253846 25242 rpcz_store.cc:276] 0709 13:08:19.738252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:19.743545 (+  5293us) service_pool.cc:224] Handling call
0709 13:08:22.195113 (+2451568us) spinlock_profiling.cc:243] Waited 8.39 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:22.195381 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:22.203852 (+  8471us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:22.219427 (+ 15575us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19292416}
I20260709 13:08:22.197885 27416 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:22.207341 27307 consensus_meta.cc:287] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.125s	user 0.011s	sys 0.001s
I20260709 13:08:22.255927 27307 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.257541 25939 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:08:22.258181 27307 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.220052 27432 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:22.258107 27014 consensus_queue.cc:260] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.259248 27432 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.260044 27014 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:08:22.262552 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 131025us
I20260709 13:08:22.098521 27451 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.263568 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:22.264220 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:22.264928 25312 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:22.265792 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:22.266498 25312 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:22.264423 25358 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:22.271713 25358 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:22.275113 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:22.276005 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:22.202528 27411 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:22.277632 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 152515us
W20260709 13:08:22.266183 27384 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.932s	user 0.020s	sys 0.013s
I20260709 13:08:22.279276 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.284405 25944 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:22.205978 27341 raft_consensus.cc:740] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:22.281857 27442 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.281961 27353 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:22.282095 27402 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:22.282861 27305 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:22.297245 27318 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.205897 27346 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:22.285720 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.278354 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:22.270696 27416 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:22.276015 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:22.287456 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:22.315692 25312 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:22.283779 27327 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:22.289036 25942 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 5 LEARNER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:08:22.267045 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.267444 27473 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:22.213250 27471 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.296685 27442 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:22.277153 26875 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:22.334050 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.274207 26996 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:22.298540 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 140282us
I20260709 13:08:22.297617 27259 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:22.209071 26975 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:22.292614 25936 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 5: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:22.319474 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.288034 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.295874 27451 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:22.319191 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 142786us
I20260709 13:08:22.317965 27404 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:22.295969 27403 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.317374 27346 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:22.297438 27469 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:22.341886 25241 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:22.342921 27471 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:22.355261 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:22.343751 27450 log.cc:927] Time spent T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.667s	user 0.002s	sys 0.000s
I20260709 13:08:22.355818 25241 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:22.329139 27473 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:22.313870 27318 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:22.321743 27327 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:22.335601 25394 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:08:22.348608 25312 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:22.321002 27416 raft_consensus.cc:697] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:22.346653 27341 consensus_queue.cc:260] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.356868 25939 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:22.353212 25212 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:22.297288 27305 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.338531 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:22.295950 27303 consensus_queue.cc:260] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.361972 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.369632 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.388583 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1293) took 1580 ms. Trace:
I20260709 13:08:22.388823 25942 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:22.389006 25241 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:22.349253 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:22.363663 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:22.388950 25243 rpcz_store.cc:276] 0709 13:08:20.808068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:21.145931 (+337863us) service_pool.cc:224] Handling call
0709 13:08:21.191138 (+ 45207us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:21.302588 (+111450us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:08:21.994073 (+691485us) log.cc:844] Serialized 14 byte log entry
0709 13:08:22.087796 (+ 93723us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:22.287974 (+200178us) spinlock_profiling.cc:243] Waited 2.23 ms on lock 0x7b2c000014c8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f29755aba11 00007f29755bb372 00007f2976ceda01 00007f2976cb243a 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2
0709 13:08:22.314239 (+ 26265us) log.cc:844] Serialized 26 byte log entry
0709 13:08:22.314450 (+   211us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:22.314489 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:22.314555 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:22.378732 (+ 64177us) raft_consensus.cc:1685] finished
0709 13:08:22.388376 (+  9644us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:22.388524 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":27024,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":44335,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":3611,"spinlock_wait_cycles":5120896,"thread_start_us":832681,"threads_started":3,"wal-append.queue_time_us":135945}
I20260709 13:08:22.377902 25212 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:22.320145 27353 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:22.394966 25320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 280) took 2805 ms (client timeout 2091 ms). Trace:
W20260709 13:08:22.365486 27465 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 13:08:22.380303 27434 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.362147 27404 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:22.384270 27432 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:22.385071 25392 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:22.380461 27474 consensus_queue.cc:1059] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.053s
I20260709 13:08:22.410102 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:22.410440 27474 raft_consensus.cc:3055] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:22.410645 25392 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:22.410817 27474 raft_consensus.cc:740] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:22.385844 25312 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:22.391400 25315 raft_consensus.cc:2410] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:08:22.394426 25212 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:22.412305 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 1824) took 1605 ms. Trace:
W20260709 13:08:22.365061 27282 consensus_meta.cc:287] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.616s	user 0.008s	sys 0.002s
W20260709 13:08:22.364024 27444 consensus_meta.cc:287] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.766s	user 0.017s	sys 0.007s
W20260709 13:08:22.376360 27409 consensus_meta.cc:287] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.685s	user 0.009s	sys 0.000s
W20260709 13:08:22.370175 27414 consensus_meta.cc:287] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.727s	user 0.010s	sys 0.002s
W20260709 13:08:22.365310 27476 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 13:08:22.420467 27386 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:22.420635 25313 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:22.421344 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 1828) took 1611 ms. Trace:
I20260709 13:08:22.408567 27404 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:22.367650 27443 consensus_meta.cc:287] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.616s	user 0.004s	sys 0.006s
I20260709 13:08:22.375516 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 441) took 1183 ms. Trace:
I20260709 13:08:22.337194 27442 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:22.380458 27392 consensus_meta.cc:287] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.196s	user 0.012s	sys 0.011s
I20260709 13:08:22.358716 27403 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:22.372720 27440 consensus_meta.cc:287] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.547s	user 0.007s	sys 0.000s
I20260709 13:08:22.380328 27469 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:22.447445 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:22.448137 25240 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:22.329406 27402 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.399674 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:22.395553 25320 rpcz_store.cc:269] 0709 13:08:19.590932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:19.595471 (+  4539us) service_pool.cc:224] Handling call
0709 13:08:22.394746 (+2799275us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x7b5400442d54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:22.394881 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":954533248}
I20260709 13:08:22.402024 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:22.378621 27327 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:22.364113 27441 raft_consensus.cc:2764] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 13:08:22.413757 25212 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:22.413955 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.412624 25312 rpcz_store.cc:276] 0709 13:08:20.807293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.385301 (+1578008us) service_pool.cc:224] Handling call
0709 13:08:22.385608 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:22.412038 (+ 26430us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:22.412228 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:22.414733 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 1827) took 1605 ms. Trace:
I20260709 13:08:22.397898 27400 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:22.407693 27341 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:22.358091 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.415352 25386 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 2: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:22.363343 27318 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.389513 27329 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.429508 25242 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:22.415736 27432 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:08:22.419023 27444 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:22.420315 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.421731 25313 rpcz_store.cc:276] 0709 13:08:20.810004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.413545 (+1603541us) service_pool.cc:224] Handling call
0709 13:08:22.420442 (+  6897us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:22.421050 (+   608us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:22.421291 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:22.473896 27480 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:22.474012 27346 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.474443 27329 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:22.417337 27416 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.477391 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.429466 25937 rpcz_store.cc:276] 0709 13:08:21.192298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:21.194404 (+  2106us) service_pool.cc:224] Handling call
0709 13:08:22.374732 (+1180328us) spinlock_profiling.cc:243] Waited 8.42 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:22.374944 (+   212us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:22.375150 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:22.375179 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:22.375225 (+    46us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:22.375276 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:22.375336 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:22.375453 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19357568}
I20260709 13:08:22.434823 27261 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:22.443244 27353 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.443610 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.444003 27442 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.433043 27392 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:22.445812 27117 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.421002 27386 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.399266 27303 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:22.450718 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1731) took 1550 ms. Trace:
I20260709 13:08:22.442025 27403 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.460872 25392 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:22.466006 25312 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:22.483556 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:22.483783 25321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1722) took 2015 ms (client timeout 1605 ms). Trace:
I20260709 13:08:22.484097 25392 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:08:22.484404 25321 rpcz_store.cc:269] 0709 13:08:20.468725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:20.468833 (+   108us) service_pool.cc:224] Handling call
0709 13:08:22.483593 (+2014760us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b5400040bd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:22.483704 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":346041088}
I20260709 13:08:22.432089 27443 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.491448 25312 raft_consensus.cc:1275] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:22.466949 27448 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:22.470568 27305 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:22.472045 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:22.470600 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.451383 27469 raft_consensus.cc:697] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:22.475564 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.505656 25942 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:22.455878 25315 rpcz_store.cc:276] 0709 13:08:20.809728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.038547 (+1228819us) service_pool.cc:224] Handling call
0709 13:08:22.414647 (+376100us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:22.479171 25320 rpcz_store.cc:276] 0709 13:08:20.900338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.449956 (+1549618us) service_pool.cc:224] Handling call
0709 13:08:22.450255 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:22.450435 (+   180us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:22.450459 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:22.450496 (+    37us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:22.450528 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:22.450580 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:22.450678 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:22.447088 27455 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.446930 27440 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.480576 25243 raft_consensus.cc:2410] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:08:22.481297 25391 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:22.423301 27414 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.483951 25944 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:22.422801 27409 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.476783 25313 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 13:08:22.489037 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:22.456611 27400 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:22.490092 25321 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:22.417100 27474 consensus_queue.cc:260] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.495921 25312 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 4. Current term is 5. Ops: []
W20260709 13:08:22.498402 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 84752.1us
I20260709 13:08:22.501780 25317 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
W20260709 13:08:22.504318 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:22.505813 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:22.423214 27289 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.474501 27480 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.515182 25392 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
I20260709 13:08:22.524487 25240 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:22.505193 27318 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.526499 25359 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:22.529238 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.528358 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.529899 25240 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:22.530339 25392 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:22.517508 27448 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:22.530789 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.532553 25243 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:22.530400 25359 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:22.520823 27400 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:22.543851 27297 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:22.546358 25246 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:22.521499 27159 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:22.546736 25359 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:22.525549 27434 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:22.527148 27335 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:22.552179 25359 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:22.531042 25388 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: []
W20260709 13:08:22.550846 27385 log.cc:927] Time spent T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 2.155s	user 0.000s	sys 0.003s
I20260709 13:08:22.551224 26875 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:22.557703 25237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1301) took 1411 ms. Trace:
I20260709 13:08:22.558432 27414 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:22.558509 25245 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 1298) took 1716 ms (client timeout 1676 ms). Trace:
I20260709 13:08:22.560289 27480 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:22.560680 27454 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:08:22.554569 27338 consensus_queue.cc:1048] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
W20260709 13:08:22.556944 27320 consensus_meta.cc:287] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.612s	user 0.012s	sys 0.009s
I20260709 13:08:22.553457 25911 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:22.565407 27335 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.566021 27440 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:22.565502 27469 consensus_queue.cc:237] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.566200 27297 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:22.553071 26996 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 13:08:22.551239 27159 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.561782 27434 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.574841 27402 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.568344 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.551971 27318 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:22.596503 27464 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:22.571147 25245 rpcz_store.cc:269] 0709 13:08:20.841774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:20.854520 (+ 12746us) service_pool.cc:224] Handling call
0709 13:08:22.558337 (+1703817us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x7b54002b2354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:22.558440 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":990553856}
I20260709 13:08:22.551654 27474 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:22.572460 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.567376 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:22.599076 25240 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:22.599210 25243 raft_consensus.cc:2393] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:22.590219 27289 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:22.590473 27482 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
I20260709 13:08:22.591894 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1162) took 3535 ms. Trace:
I20260709 13:08:22.579080 27346 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:22.578052 27261 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:22.600540 27409 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:22.579913 27297 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:22.605679 27483 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 13:08:22.605831 27297 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:22.594807 27318 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:22.579420 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.610306 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.610888 25245 raft_consensus.cc:2393] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:22.603142 25395 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:22.602638 25393 rpcz_store.cc:276] 0709 13:08:19.056473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:19.146385 (+ 89912us) service_pool.cc:224] Handling call
0709 13:08:19.155531 (+  9146us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:19.155880 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:19.155906 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:19.174778 (+ 18872us) log.cc:844] Serialized 26 byte log entry
0709 13:08:19.885626 (+710848us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:20.520037 (+634411us) log.cc:844] Serialized 14 byte log entry
0709 13:08:20.520288 (+   251us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:20.520387 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:22.491498 (+1971111us) spinlock_profiling.cc:243] Waited 46 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:22.591596 (+100098us) raft_consensus.cc:1685] finished
0709 13:08:22.591696 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:22.591838 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":727472,"prepare.run_cpu_time_us":151,"prepare.run_wall_time_us":2599,"raft.queue_time_us":108378,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":15723,"spinlock_wait_cycles":148188288,"thread_start_us":1301268,"threads_started":2,"wal-append.queue_time_us":1152921}
I20260709 13:08:22.597057 27464 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.621012 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:22.569994 25237 rpcz_store.cc:276] 0709 13:08:21.145850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:21.225703 (+ 79853us) service_pool.cc:224] Handling call
0709 13:08:21.226030 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:21.670048 (+444018us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:22.557438 (+887390us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b540013ae54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:22.557636 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":786414848}
I20260709 13:08:22.623914 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.607057 27261 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:22.627084 25393 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:22.627116 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.627614 25387 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:22.628186 25243 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:22.618654 27318 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:22.632788 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.634487 25240 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:22.634599 25245 raft_consensus.cc:2410] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:08:22.639937 27434 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:22.630574 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.641045 25386 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:22.629312 27261 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:22.641629 27327 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.586977 27448 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.645493 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:22.648510 25243 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:22.650866 27335 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:22.637874 27341 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:22.654915 25359 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:22.657161 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.636137 27443 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.659782 27289 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:22.660648 26936 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.660332 27289 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.661257 27335 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:22.661142 27341 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.661737 27335 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:22.662123 27335 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:22.662997 27289 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.663354 27341 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:22.653355 27382 consensus_meta.cc:287] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.549s	user 0.004s	sys 0.007s
I20260709 13:08:22.642139 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:22.660386 27404 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.665931 25359 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:22.677685 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.664394 27382 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.681264 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.681757 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:22.682531 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.682898 27394 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:22.686165 25237 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:22.675947 27443 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:22.658568 27455 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:22.681272 27386 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:22.671634 27335 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.692879 25240 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:22.691459 25386 raft_consensus.cc:2393] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:22.688992 27327 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.698738 25245 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:22.692037 27443 raft_consensus.cc:697] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:22.688709 27400 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.697638 27386 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:22.694489 27297 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.695315 27403 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.703604 27386 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.707715 25392 raft_consensus.cc:2393] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:22.709882 27299 raft_consensus.cc:3055] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:22.710292 27299 raft_consensus.cc:740] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:22.710800 27368 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:22.711339 27368 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.715076 27368 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:22.715574 27361 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:22.717824 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:22.719113 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.726907 26875 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:08:22.741173 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:22.740862 27261 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:22.741722 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:22.742956 27261 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:22.746171 27436 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:22.746868 25284 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:22.747191 27289 consensus_queue.cc:1059] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:08:22.747687 27436 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:22.748152 27334 raft_consensus.cc:3055] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:22.748556 27334 raft_consensus.cc:740] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:22.749454 27334 consensus_queue.cc:260] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.750757 27334 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:22.751348 27425 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:22.708657 27327 consensus_queue.cc:1059] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260709 13:08:22.760299 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.765602 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.765653 25241 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:22.766345 25943 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:22.767412 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:22.768759 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.769382 25209 leader_election.cc:400] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4.
I20260709 13:08:22.769760 25209 leader_election.cc:403] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 13:08:22.729424 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.733289 27473 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.770073 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:22.770807 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:22.771958 27473 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:22.724280 27443 consensus_queue.cc:237] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:22.736403 25246 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 1278) took 2986 ms (client timeout 3200 ms). Trace:
W20260709 13:08:22.773047 25246 rpcz_store.cc:269] 0709 13:08:19.750106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:19.750226 (+   120us) service_pool.cc:224] Handling call
0709 13:08:22.285538 (+2535312us) spinlock_profiling.cc:243] Waited 5.16 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:22.736106 (+450568us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b54003a98d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:22.736322 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":545816704}
I20260709 13:08:22.773471 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:22.774183 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:22.774685 25246 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:22.735560 27382 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.775331 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.738605 27318 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.776027 25391 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:22.736444 27404 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.772066 25938 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:22.777426 27318 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.740396 27400 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.779861 27447 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:22.780242 25386 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:22.776835 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.781325 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.781857 25246 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:22.782691 25209 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:22.782298 25943 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:22.783170 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:22.784178 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:22.784901 25937 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:22.785486 25246 raft_consensus.cc:2393] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:22.785564 25359 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:22.788435 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:22.788817 25942 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:22.789785 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.790457 25241 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 2.
I20260709 13:08:22.792204 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.793143 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.793320 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1167) took 3735 ms. Trace:
I20260709 13:08:22.793758 25937 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:22.793694 25389 rpcz_store.cc:276] 0709 13:08:19.057822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:19.279364 (+221542us) service_pool.cc:224] Handling call
0709 13:08:21.678592 (+2399228us) spinlock_profiling.cc:243] Waited 6.33 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8deee 0000000000375939 00007f297bc011e0 00007f297bcca0c4 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:21.678756 (+   164us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:22.085752 (+406996us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:22.793005 (+707253us) spinlock_profiling.cc:243] Waited 345 ms on lock 0x7b5400283cd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:22.793260 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":808547072}
I20260709 13:08:22.794495 27400 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:22.794907 27400 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:22.788911 27382 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:22.795226 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:22.797165 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.797984 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.799890 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.800644 25391 raft_consensus.cc:2410] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:08:22.801376 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.806658 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.807670 25246 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:22.809075 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:22.809779 25314 raft_consensus.cc:2368] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4.
I20260709 13:08:22.811543 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.812171 25937 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:22.812799 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:22.813506 25391 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:22.814190 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.814746 25391 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:22.815618 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.816710 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:22.817462 25314 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:22.817548 25908 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:22.795244 27400 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:22.818892 25389 raft_consensus.cc:2393] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:22.819749 27468 raft_consensus.cc:3055] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:22.819819 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.820238 27468 raft_consensus.cc:740] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:22.820633 25389 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 6: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:22.826761 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:22.827521 25908 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:22.827796 25314 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:22.821028 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1181) took 1146 ms. Trace:
I20260709 13:08:22.829819 25389 rpcz_store.cc:276] 0709 13:08:21.674697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.819586 (+1144889us) service_pool.cc:224] Handling call
0709 13:08:22.820976 (+  1390us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:22.830971 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.832460 25389 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:08:22.832890 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 1799) took 1241 ms. Trace:
I20260709 13:08:22.821046 27468 consensus_queue.cc:260] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.833493 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.833837 27473 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:22.834146 25386 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:22.828656 25908 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:22.834753 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 1814) took 1135 ms. Trace:
I20260709 13:08:22.835117 25386 rpcz_store.cc:276] 0709 13:08:21.699057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.825537 (+1126480us) service_pool.cc:224] Handling call
0709 13:08:22.834683 (+  9146us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:22.835673 27455 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:22.835769 27435 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.836264 27455 raft_consensus.cc:697] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 4 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:22.836333 25386 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:22.837764 27346 consensus_queue.cc:1059] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:08:22.833328 25388 rpcz_store.cc:276] 0709 13:08:21.591392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:21.593838 (+  2446us) service_pool.cc:224] Handling call
0709 13:08:21.677691 (+ 83853us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:22.568450 (+890759us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:22.832609 (+264159us) spinlock_profiling.cc:243] Waited 839 ms on lock 0x7b54001e3f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:22.832790 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1930755840}
I20260709 13:08:22.839403 25388 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:22.839979 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1178) took 1166 ms. Trace:
I20260709 13:08:22.840021 27436 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:22.840569 27455 consensus_queue.cc:237] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.841480 27425 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:22.842084 27394 raft_consensus.cc:3055] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:22.842447 27394 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:22.843259 27394 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.844256 27394 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:22.846240 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:22.847030 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:22.847288 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:22.847203 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.834370 25391 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:22.849103 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1175) took 1176 ms. Trace:
I20260709 13:08:22.849515 25391 rpcz_store.cc:276] 0709 13:08:21.673117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.833954 (+1160837us) service_pool.cc:224] Handling call
0709 13:08:22.834197 (+   243us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:22.848888 (+ 14691us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:22.849063 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:22.789850 27318 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:22.850906 25319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1730) took 1950 ms. Trace:
I20260709 13:08:22.851406 25319 rpcz_store.cc:276] 0709 13:08:20.900038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:21.336042 (+436004us) service_pool.cc:224] Handling call
0709 13:08:22.850685 (+1514643us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b54000429d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:22.850837 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1143363968}
I20260709 13:08:22.853734 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.854337 25391 raft_consensus.cc:2393] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:22.854064 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.853029 25287 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:22.834262 27473 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:22.840669 27436 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:22.854839 25245 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:22.837041 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1177) took 1163 ms. Trace:
I20260709 13:08:22.867250 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.869964 25287 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:22.871481 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:22.872360 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.872731 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:22.873003 25391 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:22.873521 25284 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:22.874459 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 1813) took 1190 ms. Trace:
I20260709 13:08:22.845299 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.840440 25388 rpcz_store.cc:276] 0709 13:08:21.673984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.838931 (+1164947us) service_pool.cc:224] Handling call
0709 13:08:22.839227 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:22.839764 (+   537us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:22.839929 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:22.874845 25391 rpcz_store.cc:276] 0709 13:08:21.684120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.873707 (+1189587us) service_pool.cc:224] Handling call
0709 13:08:22.873969 (+   262us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:22.874141 (+   172us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:22.874167 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:22.874210 (+    43us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:08:22.874256 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:22.874308 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:22.874407 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:22.879043 25284 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:22.864840 27473 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:22.880576 25284 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:22.863907 27318 raft_consensus.cc:697] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:22.878786 27108 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:22.887534 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:22.879081 27343 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:22.879758 27474 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:22.882903 27271 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:22.878718 27410 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:22.893851 25246 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:08:22.895305 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:22.892195 27436 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:22.884970 25284 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:22.885597 27341 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:22.903730 26936 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:22.886741 25209 leader_election.cc:400] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4.
I20260709 13:08:22.904811 25284 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:22.885998 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 1805) took 1266 ms. Trace:
I20260709 13:08:22.867341 25386 rpcz_store.cc:276] 0709 13:08:21.673743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.835773 (+1162030us) service_pool.cc:224] Handling call
0709 13:08:22.836044 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:22.836808 (+   764us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:22.836992 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:22.910655 26936 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:22.920358 25209 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:22.916590 27341 raft_consensus.cc:697] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:22.899507 27474 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:22.900911 27271 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:22.911161 27410 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:22.924582 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:22.925949 25238 raft_consensus.cc:2410] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:08:22.922290 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:22.917806 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.935578 25246 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:22.923782 25388 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 13:08:22.922022 26936 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:22.916363 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 76611.1us
I20260709 13:08:22.922484 27108 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:22.923518 27402 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:22.937074 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:22.928761 27474 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:22.942695 25285 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:22.943225 25246 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:22.933503 27271 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:22.942435 27108 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:22.947364 25285 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:22.948782 27108 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:22.953409 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 1819) took 1141 ms. Trace:
I20260709 13:08:22.942700 27469 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:22.943987 27402 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:22.939930 27305 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:22.958335 27410 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.962459 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1543) took 1287 ms. Trace:
I20260709 13:08:22.947190 27436 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:22.948356 25386 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:22.998375 27436 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:22.998773 27436 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:22.998901 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 1821) took 1087 ms. Trace:
I20260709 13:08:22.999329 25386 rpcz_store.cc:276] 0709 13:08:21.911536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.926843 (+1015307us) service_pool.cc:224] Handling call
0709 13:08:22.939912 (+ 13069us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:22.998570 (+ 58658us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:22.998841 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:23.000392 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:22.994971 27469 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:23.001691 27469 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:23.003058 27159 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:22.965714 27341 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:22.944394 27382 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4.
I20260709 13:08:22.929491 27318 consensus_queue.cc:237] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:23.012061 27108 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:23.018931 27108 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:23.019992 27474 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:23.021848 27464 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:23.024550 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:22.969022 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:22.971369 25395 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1154) took 4002 ms (client timeout 1731 ms). Trace:
I20260709 13:08:22.972659 25390 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:08:22.973344 27435 consensus_queue.cc:1059] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:22.973944 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 1822) took 2167 ms. Trace:
W20260709 13:08:22.974433 25387 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1176) took 1301 ms (client timeout 1613 ms). Trace:
I20260709 13:08:22.975246 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:23.025102 25393 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:22.976732 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:23.030256 25317 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1716) took 2577 ms (client timeout 1645 ms). Trace:
W20260709 13:08:23.051409 25317 rpcz_store.cc:269] 0709 13:08:20.453428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:20.455759 (+  2331us) service_pool.cc:224] Handling call
0709 13:08:23.030075 (+2574316us) spinlock_profiling.cc:243] Waited 1.99 s on lock 0x7b54004399d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:23.030184 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4572608384}
I20260709 13:08:23.052045 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:22.977421 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.052603 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:23.052958 25319 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:23.053350 25317 raft_consensus.cc:2376] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8. Re-sending same reply.
I20260709 13:08:23.050925 25942 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:22.980010 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:22.982513 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:23.084174 25390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1171) took 4025 ms. Trace:
I20260709 13:08:23.084611 25390 rpcz_store.cc:276] 0709 13:08:19.058830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:19.358826 (+299996us) service_pool.cc:224] Handling call
0709 13:08:21.678018 (+2319192us) spinlock_profiling.cc:243] Waited 6.5 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:22.972255 (+1294237us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:23.050366 (+ 78111us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:23.083951 (+ 33585us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400146750. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:23.084104 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2990906496}
I20260709 13:08:22.985267 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:22.985427 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:22.986939 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.995851 27364 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:23.092451 25316 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 3: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:22.988252 27386 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:22.986994 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:23.107479 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:22.988840 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.989459 25244 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 2 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:22.993319 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:22.973400 27415 raft_consensus.cc:3055] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:22.994501 25358 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:22.995965 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:22.995568 25242 rpcz_store.cc:276] 0709 13:08:21.675413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.255148 (+579735us) service_pool.cc:224] Handling call
0709 13:08:22.255485 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:22.467623 (+212138us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:22.962194 (+494571us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b5400422554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:22.962396 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":399714048}
I20260709 13:08:22.994120 27348 raft_consensus.cc:3055] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:22.995240 27402 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:22.981528 27365 consensus_queue.cc:1059] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260709 13:08:22.996407 25285 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:22.997084 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.997179 27355 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:08:22.997102 27305 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:23.003641 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 103092us
I20260709 13:08:23.004299 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:23.005834 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:23.010921 27493 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 13:08:23.012253 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:23.013819 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:08:23.025959 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:22.977763 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:23.031595 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 1817) took 1283 ms. Trace:
I20260709 13:08:23.054474 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:23.034736 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:23.050645 25387 rpcz_store.cc:269] 0709 13:08:21.673489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.006843 (+333354us) service_pool.cc:224] Handling call
0709 13:08:22.974280 (+967437us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x7b540013d8d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:22.974382 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1263372160}
W20260709 13:08:23.039206 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:23.078466 25943 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:08:23.050201 25395 rpcz_store.cc:269] 0709 13:08:18.968488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:18.999470 (+ 30982us) service_pool.cc:224] Handling call
0709 13:08:22.971126 (+3971656us) spinlock_profiling.cc:243] Waited 3.48 s on lock 0x7b5400146754. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:22.971258 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8000726272}
I20260709 13:08:23.000984 25386 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 7
I20260709 13:08:23.050515 25318 rpcz_store.cc:276] 0709 13:08:20.806787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:21.835995 (+1029208us) service_pool.cc:224] Handling call
0709 13:08:22.973412 (+1137417us) spinlock_profiling.cc:243] Waited 21 ms on lock 0x7b5400087ad4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:22.973571 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":48321920}
I20260709 13:08:22.916380 25389 rpcz_store.cc:276] 0709 13:08:21.616688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.831907 (+1215219us) service_pool.cc:224] Handling call
0709 13:08:22.832204 (+   297us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:22.853526 (+ 21322us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:22.878629 (+ 25103us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:23.086068 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:22.987670 25909 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:23.092150 25945 raft_consensus.cc:2393] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:23.047465 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:23.101783 27364 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:23.102864 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:23.044713 27468 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:23.111158 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:23.033438 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:22.995260 27443 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.098s
I20260709 13:08:23.042266 27297 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:22.994179 27404 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.111s
I20260709 13:08:23.260829 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 1815) took 1560 ms. Trace:
I20260709 13:08:23.234067 25141 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39596 (request call id 283) took 9251 ms. Trace:
I20260709 13:08:23.296974 25240 raft_consensus.cc:2393] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:23.204808 25317 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260709 13:08:23.313556 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 59801.7us
I20260709 13:08:23.314922 25942 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:23.296802 25391 rpcz_store.cc:276] 0709 13:08:21.700517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.878233 (+1177716us) service_pool.cc:224] Handling call
0709 13:08:22.878501 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:22.893631 (+ 15130us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:08:22.912323 (+ 18692us) log.cc:844] Serialized 14 byte log entry
0709 13:08:23.014898 (+102575us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b1800133480. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f297685223a 00007f2976cbca2a
0709 13:08:23.260334 (+245436us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:23.260418 (+    84us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:23.260489 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:23.260578 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:23.260752 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14555,"prepare.queue_time_us":18440,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":876,"raft.queue_time_us":42567,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":6109,"spinlock_wait_cycles":9410816,"thread_start_us":202883,"threads_started":1}
I20260709 13:08:23.315672 25312 raft_consensus.cc:2368] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5.
W20260709 13:08:23.317363 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 329746us
I20260709 13:08:23.317750 25239 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:23.318367 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:23.109108 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:23.320291 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:23.321359 25314 raft_consensus.cc:2376] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5. Re-sending same reply.
I20260709 13:08:23.153571 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.323109 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:23.115002 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1538) took 1588 ms. Trace:
I20260709 13:08:23.324178 25395 raft_consensus.cc:2393] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:23.330556 25390 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:23.346954 25944 raft_consensus.cc:2393] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:23.347991 25390 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:23.348677 25390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 290) took 1379 ms. Trace:
I20260709 13:08:23.316532 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:23.349778 25391 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:23.208380 27382 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:23.351056 27382 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:23.352443 25936 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:23.282223 27474 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:23.315250 27415 raft_consensus.cc:740] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:23.353638 27299 consensus_queue.cc:260] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:23.354255 27415 consensus_queue.cc:260] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:23.355639 27415 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:23.208329 27447 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:23.269579 27473 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:23.357396 25287 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:23.357705 27447 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:23.359088 25287 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:23.361174 27447 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:23.361330 27453 consensus_queue.cc:1059] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:08:23.362347 27453 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:23.342214 27364 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:23.363353 27247 raft_consensus.cc:3055] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:23.363754 27247 raft_consensus.cc:740] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:23.363873 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:23.364466 25240 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:23.340936 27394 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:23.365653 25240 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:23.365806 27394 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:23.366410 27394 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:23.195854 27436 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:22.994683 25388 rpcz_store.cc:276] 0709 13:08:21.816238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.880476 (+1064238us) service_pool.cc:224] Handling call
0709 13:08:22.917968 (+ 37492us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:22.941820 (+ 23852us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:22.953346 (+ 11526us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:23.316871 25392 rpcz_store.cc:276] 0709 13:08:21.754682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.651656 (+896974us) service_pool.cc:224] Handling call
0709 13:08:23.031433 (+379777us) spinlock_profiling.cc:243] Waited 48.2 ms on lock 0x7b54003e4754. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:23.031542 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":110922112}
I20260709 13:08:23.323935 25244 rpcz_store.cc:276] 0709 13:08:21.526382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.012041 (+485659us) service_pool.cc:224] Handling call
0709 13:08:22.026999 (+ 14958us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:23.111023 (+1084024us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:23.114664 (+  3641us) spinlock_profiling.cc:243] Waited 962 ms on lock 0x7b54003ab1d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:23.114919 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2212991744}
I20260709 13:08:23.314669 25209 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:23.370975 25359 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:23.337281 25243 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:23.379299 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:23.379925 25239 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:23.380389 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:23.031193 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32994 (request call id 1536) took 1506 ms. Trace:
I20260709 13:08:23.130658 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:23.028599 27335 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:23.341356 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 345006us
I20260709 13:08:23.343232 25393 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:23.140594 25245 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:23.278119 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:23.330027 27438 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:23.349027 25390 rpcz_store.cc:276] 0709 13:08:21.969321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.347512 (+1378191us) service_pool.cc:224] Handling call
0709 13:08:23.347842 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:23.348462 (+   620us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:23.348628 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:23.369002 25287 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:23.335125 25315 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:23.371322 25395 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:23.397568 25395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 291) took 1428 ms. Trace:
I20260709 13:08:23.359134 27473 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:23.274374 27425 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:08:23.378597 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 64019.9us
I20260709 13:08:23.312711 25141 rpcz_store.cc:276] 0709 13:08:13.958223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:13.958665 (+   442us) service_pool.cc:224] Handling call
0709 13:08:23.209086 (+9250421us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:08:20.329267 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:08:20.330157 (+   890us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:08:20.330213 (+    56us) write_op.cc:435] Acquired schema lock
0709 13:08:20.330235 (+    22us) tablet.cc:662] Decoding operations
0709 13:08:20.331257 (+  1022us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:08:20.331339 (+    82us) write_op.cc:641] Partition lock acquired for write op
0709 13:08:20.331372 (+    33us) tablet.cc:685] Acquiring locks for 9 operations
0709 13:08:20.332357 (+   985us) tablet.cc:701] Row locks acquired
0709 13:08:20.332407 (+    50us) write_op.cc:260] PREPARE: finished
0709 13:08:20.332592 (+   185us) write_op.cc:270] Start()
0709 13:08:20.332741 (+   149us) write_op.cc:276] Timestamp: P: 1783602500332514 usec, L: 0
0709 13:08:20.332766 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 13:08:20.333502 (+   736us) log.cc:844] Serialized 3621 byte log entry
0709 13:08:22.132443 (+1798941us) spinlock_profiling.cc:243] Waited 36 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f2973ba80ac 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b
0709 13:08:22.526410 (+393967us) op_driver.cc:464] REPLICATION: finished
0709 13:08:22.625939 (+ 99529us) write_op.cc:301] APPLY: starting
0709 13:08:22.631131 (+  5192us) tablet.cc:1367] starting BulkCheckPresence
0709 13:08:22.651615 (+ 20484us) tablet.cc:1370] finished BulkCheckPresence
0709 13:08:22.653916 (+  2301us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:08:22.709410 (+ 55494us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:08:22.709647 (+   237us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0709 13:08:22.709694 (+    47us) write_op.cc:312] APPLY: finished
0709 13:08:22.709977 (+   283us) log.cc:844] Serialized 71 byte log entry
0709 13:08:22.713269 (+  3292us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:22.713422 (+   153us) write_op.cc:454] Released schema lock
0709 13:08:22.713487 (+    65us) write_op.cc:341] FINISH: Updating metrics
0709 13:08:22.733554 (+ 20067us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:22.733584 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1003488,"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":81929,"apply.run_cpu_time_us":4589,"apply.run_wall_time_us":110070,"num_ops":9,"prepare.queue_time_us":1217113,"prepare.run_cpu_time_us":37883,"prepare.run_wall_time_us":1966347,"raft.queue_time_us":61927,"raft.run_cpu_time_us":11295,"raft.run_wall_time_us":88310,"replication_time_us":2178712,"spinlock_wait_cycles":82814848,"thread_start_us":2656808,"threads_started":4,"wal-append.queue_time_us":1961808}]]}
I20260709 13:08:23.320958 25909 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:23.399104 25209 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:23.396749 25245 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:23.402741 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:23.396749 25390 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:23.380980 25391 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:23.364544 27247 consensus_queue.cc:260] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:23.401757 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 80889.8us
I20260709 13:08:23.362972 27453 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:23.383912 25139 catalog_manager.cc:5697] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:23.384500 27342 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:23.345930 27386 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:23.385720 27382 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:23.028136 27271 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:23.033208 27400 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:23.257983 27289 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:23.401719 27346 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:23.390592 25285 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:23.450793 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 60281.3us
I20260709 13:08:23.391737 25911 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:23.393201 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 1822) took 1430 ms. Trace:
W20260709 13:08:23.451831 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 60167us
I20260709 13:08:23.451726 25387 rpcz_store.cc:276] 0709 13:08:21.963247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.346770 (+1383523us) service_pool.cc:224] Handling call
0709 13:08:23.392576 (+ 45806us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:23.392794 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:23.392822 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:23.392865 (+    43us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:23.392908 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:23.392978 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:23.393107 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:08:23.452845 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:23.366874 25240 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:23.314411 27348 raft_consensus.cc:740] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:23.397915 25395 rpcz_store.cc:276] 0709 13:08:21.969611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.349794 (+1380183us) service_pool.cc:224] Handling call
0709 13:08:23.350150 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:23.397294 (+ 47144us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:23.397498 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:23.412667 27408 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:08:23.334259 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 339810us
I20260709 13:08:23.456874 25285 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:23.457688 25358 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:23.457957 25285 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:23.458750 25358 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:23.394505 25140 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53264 (request call id 203) took 14006 ms (client timeout 15000 ms). Trace:
I20260709 13:08:23.459467 27474 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:23.459414 25140 rpcz_store.cc:269] 0709 13:08:09.289511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:09.289715 (+   204us) service_pool.cc:224] Handling call
0709 13:08:09.389910 (+100195us) spinlock_profiling.cc:243] Waited 87.9 ms on lock 0x7b5400470ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1c090 00007f2976cbaf3a 00007f297ca711cc 00007f297cb6f37e 00007f297b0302a4 00007f297b030232 00007f297b030161 00007f297b0300dc 00007f297b02f392 00007f2974fed48c 00007f2974fec8c6
0709 13:08:23.281719 (+13891809us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":85132,"reported_tablets":8,"spinlock_wait_cycles":202257280}
I20260709 13:08:23.460034 25240 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:23.338901 25394 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:23.404810 25909 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:23.461812 27159 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:08:23.461997 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 57256.5us
I20260709 13:08:23.462788 27425 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:23.463824 25387 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:23.406481 25393 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:23.410414 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:23.464900 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 295) took 1452 ms. Trace:
I20260709 13:08:23.407287 25239 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:23.464915 25244 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:23.465282 27464 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:23.414582 25139 catalog_manager.cc:5697] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:23.465718 25243 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 13:08:23.465867 27464 raft_consensus.cc:697] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:23.467670 27464 consensus_queue.cc:237] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:23.466502 25139 catalog_manager.cc:5697] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:08:23.417490 25941 raft_consensus.cc:2393] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 5: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:23.459467 27289 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:23.469498 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 589) took 1190 ms. Trace:
I20260709 13:08:23.420605 25389 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:23.472288 25139 catalog_manager.cc:5697] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:23.481739 25390 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:23.402683 27259 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:23.423763 25313 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:23.424830 27334 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:08:23.427431 27443 raft_consensus.cc:3055] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:23.428200 25237 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:23.491050 27271 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:23.428706 27343 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:23.391249 27365 consensus_queue.cc:1059] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260709 13:08:23.492048 27305 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:23.492415 27365 consensus_queue.cc:1243] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 6.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:08:23.492928 27365 raft_consensus.cc:3055] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:23.493270 27365 raft_consensus.cc:740] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:23.491117 27443 raft_consensus.cc:740] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:23.494097 27365 consensus_queue.cc:260] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:23.494639 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 288) took 1526 ms. Trace:
I20260709 13:08:23.495414 27365 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:23.495167 25389 rpcz_store.cc:276] 0709 13:08:21.968680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.376489 (+1407809us) service_pool.cc:224] Handling call
0709 13:08:23.409864 (+ 33375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:23.494312 (+ 84448us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:23.494559 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:23.389521 25238 rpcz_store.cc:276] 0709 13:08:21.525208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:21.680465 (+155257us) service_pool.cc:224] Handling call
0709 13:08:23.030976 (+1350511us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54003aa2d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:23.031134 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2539507456}
I20260709 13:08:23.394217 27447 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:23.406450 27403 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:23.497777 27447 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:23.497991 27403 raft_consensus.cc:697] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:23.498320 27447 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:23.498706 27342 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:23.499176 27342 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:23.499028 27403 consensus_queue.cc:237] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:23.434695 27247 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:23.501396 27404 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:23.501957 25388 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260709 13:08:23.503098 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:23.504585 25388 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 13:08:23.437209 25287 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:23.407002 27335 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
W20260709 13:08:23.505802 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 68694.8us
I20260709 13:08:23.506036 27335 consensus_queue.cc:1243] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 13:08:23.506165 27474 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:23.480593 27473 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:23.506675 27335 raft_consensus.cc:3055] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Stepping down as leader of term 1
W20260709 13:08:23.501791 25394 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 1806) took 1884 ms (client timeout 2086 ms). Trace:
W20260709 13:08:23.507834 25394 rpcz_store.cc:269] 0709 13:08:21.617168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.363428 (+746260us) service_pool.cc:224] Handling call
0709 13:08:23.501605 (+1138177us) spinlock_profiling.cc:243] Waited 757 ms on lock 0x7b5400234e54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:23.501734 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1740685184}
I20260709 13:08:23.291817 27469 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:23.461486 25240 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:08:23.462746 25909 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:23.469591 27501 log.cc:927] Time spent T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 13:08:23.422506 26936 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:23.465304 25393 rpcz_store.cc:276] 0709 13:08:22.012135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.390693 (+1378558us) service_pool.cc:224] Handling call
0709 13:08:23.406316 (+ 15623us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:23.464690 (+ 58374us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:23.464851 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:08:23.476473 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:23.413332 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:23.482525 25139 catalog_manager.cc:5697] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 1 to 3, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 3 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:23.469978 25941 rpcz_store.cc:276] 0709 13:08:22.280358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.286713 (+  6355us) service_pool.cc:224] Handling call
0709 13:08:23.033263 (+746550us) spinlock_profiling.cc:243] Waited 1.19 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:23.469434 (+436171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2728192}
I20260709 13:08:23.485196 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:23.530980 25285 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:23.523291 25139 catalog_manager.cc:5697] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 1 to 3, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 3 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:23.505718 27346 consensus_queue.cc:1059] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:08:23.450213 25391 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:23.508812 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:23.536082 25394 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:23.536134 27108 consensus_queue.cc:1059] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:08:23.509032 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.509321 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.537539 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:23.537909 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.538779 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:23.539001 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:23.539659 25315 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:23.540298 27438 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:23.511852 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.541040 27438 consensus_queue.cc:1059] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:08:23.541417 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.542068 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.542373 27346 raft_consensus.cc:3055] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:23.496775 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:23.542593 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.515697 25241 raft_consensus.cc:2393] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 7: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:23.517970 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.543769 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:23.544065 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:23.544607 27469 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:23.545183 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:23.545532 25320 raft_consensus.cc:2410] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:08:23.520244 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 292) took 1542 ms. Trace:
I20260709 13:08:23.545944 25392 rpcz_store.cc:276] 0709 13:08:21.975322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.378318 (+1402996us) service_pool.cc:224] Handling call
0709 13:08:23.513710 (+135392us) spinlock_profiling.cc:243] Waited 3.99 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:23.513877 (+   167us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:23.514038 (+   161us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:23.514062 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:23.514102 (+    40us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:23.516942 (+  2840us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:23.517037 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:23.517334 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9180032}
I20260709 13:08:23.547122 25908 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:23.545941 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.548364 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.525825 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:23.545388 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:23.549458 25945 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:23.485286 27159 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:08:23.548863 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:23.550872 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:23.551414 25319 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:23.551178 25317 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:23.552436 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:23.443027 27453 consensus_queue.cc:1059] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20260709 13:08:23.506969 27474 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:23.535157 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:23.542876 27346 raft_consensus.cc:740] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:23.544878 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:23.533721 25139 catalog_manager.cc:5697] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 2 to 4, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 4 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:08:23.551980 25943 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:23.528039 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:23.506999 27473 raft_consensus.cc:697] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 5 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:23.566491 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:23.582126 25941 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:08:23.582532 25312 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:23.507028 27335 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:23.582653 26936 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:23.529489 27343 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20260709 13:08:23.583413 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.583482 27335 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:23.584013 27346 consensus_queue.cc:260] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:23.584378 27441 consensus_queue.cc:1059] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:23.529517 27289 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:23.520190 27436 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:23.564970 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.590176 25319 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:23.591305 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.592777 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.569676 25942 raft_consensus.cc:2393] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 5: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:23.569785 25359 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:23.594061 25317 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:23.574766 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:23.595206 25392 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:23.560855 25315 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:23.581980 27473 consensus_queue.cc:237] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:23.542678 25314 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:23.598642 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:23.600090 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.583423 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:23.600685 25319 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:23.584787 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.549077 27408 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:23.501818 27404 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:23.587581 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:23.519554 27271 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20260709 13:08:23.530293 25386 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:23.584892 27335 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:23.570901 25139 catalog_manager.cc:5697] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 2 to 4, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 4 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:08:23.595781 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:23.603062 25359 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:23.567763 27474 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:23.626483 27474 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:23.607064 27404 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:23.630471 25316 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:08:23.608019 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:23.632126 25392 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:23.632647 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:23.633435 25387 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:23.634248 25359 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:23.635862 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:23.640962 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:23.644174 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:23.671164 25391 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:23.609565 27441 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:23.654996 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:23.656837 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:23.657369 27355 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:08:23.658046 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:23.659165 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:23.686124 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:23.660764 25393 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260709 13:08:23.686833 25393 pending_rounds.cc:85] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 5
I20260709 13:08:23.687430 25944 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 5: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:23.635638 27343 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260709 13:08:23.620401 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:23.662742 27400 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:23.588255 27453 consensus_queue.cc:1059] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20260709 13:08:23.611690 27327 raft_consensus.cc:2764] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260709 13:08:23.665830 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:23.610443 27436 raft_consensus.cc:3055] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:23.687069 25393 pending_rounds.cc:107] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 5.6
I20260709 13:08:23.702402 25393 raft_consensus.cc:2917] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Aborting config change with OpId 5.6: Aborted: Op aborted by new leader
I20260709 13:08:23.702872 25393 raft_consensus.cc:2889] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:23.669826 25139 catalog_manager.cc:5697] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:23.706348 27343 consensus_queue.cc:1243] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer a2a77ecf2c0941618b5bc79d4c04db61 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:08:23.667703 27318 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:23.683943 25391 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:23.689951 25390 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:23.638993 27365 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:23.697448 25287 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:23.723901 25212 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.017s	sys 0.103s
W20260709 13:08:23.724467 25212 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.017s	sys 0.103s
W20260709 13:08:23.724823 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 578396us
I20260709 13:08:23.708948 25139 catalog_manager.cc:5697] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 1 to 3, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 3 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:23.730827 25139 catalog_manager.cc:5697] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 1 to 3, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 3 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:08:23.756036 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 58668.8us
I20260709 13:08:23.757475 25287 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:23.759964 25212 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.006s (SENT) [suppressed 28 similar messages]
I20260709 13:08:23.763671 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:23.782202 25358 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:23.700722 27441 raft_consensus.cc:697] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:23.760635 27259 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:08:23.767617 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 304934us
I20260709 13:08:23.703188 27436 raft_consensus.cc:740] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:08:23.771564 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 368719us
I20260709 13:08:23.729990 27365 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:23.791110 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:23.791383 25209 leader_election.cc:400] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5.
I20260709 13:08:23.791841 25209 leader_election.cc:403] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:08:23.805693 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:23.769862 27247 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:08:23.854913 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:23.855130 27247 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:23.855567 27247 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:23.855247 27436 consensus_queue.cc:260] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:23.856894 27436 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:23.859887 27400 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 5
I20260709 13:08:23.821826 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.860618 25317 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:23.861914 25317 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:08:23.863745 25317 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:23.822713 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.864749 25319 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:23.823505 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:23.825546 25358 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:23.785492 27435 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:23.788849 27318 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:23.868804 25319 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:08:23.874527 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:23.810472 27259 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:23.811466 27365 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:23.874260 25209 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:08:23.880841 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 370825us
W20260709 13:08:23.871366 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 65753.8us
I20260709 13:08:23.895905 25945 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:23.877831 27289 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:23.881554 27259 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:23.867588 27297 consensus_queue.cc:1048] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260709 13:08:23.881994 27307 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:23.900714 27341 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.003s	sys 0.076s
I20260709 13:08:23.886739 27435 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:23.885479 27469 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:23.908358 27289 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:23.917339 27341 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.003s	sys 0.076s
W20260709 13:08:23.959754 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 53429us
W20260709 13:08:23.959532 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 426825us
I20260709 13:08:23.912472 27425 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:23.965757 25243 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:23.965850 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:23.921947 27289 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:23.967445 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:23.968169 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:23.969691 25241 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:23.970692 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:23.986025 25244 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:23.971668 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:23.887594 27468 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
I20260709 13:08:23.899700 27305 consensus_queue.cc:1059] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20260709 13:08:23.980697 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:24.039474 25243 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:23.919541 27469 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:23.987892 25239 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:23.989022 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:08:23.919097 27307 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:23.994820 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 369947us
I20260709 13:08:23.911257 27331 consensus_queue.cc:1059] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20260709 13:08:23.919350 27159 consensus_queue.cc:1059] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260709 13:08:24.051226 25908 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:23.982071 27473 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:23.984484 27509 raft_consensus.cc:3055] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:24.085620 25911 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:24.103188 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:24.039896 27425 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:23.978180 27447 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.057148 27469 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:23.979212 27506 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.131s
I20260709 13:08:23.951738 27441 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:24.125525 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 74406.6us
I20260709 13:08:24.074432 27509 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:08:24.138803 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 233170us
I20260709 13:08:24.113550 25938 raft_consensus.cc:2410] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260709 13:08:24.140647 25212 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:24.110850 27510 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:24.153021 25395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1202) took 1304 ms. Trace:
I20260709 13:08:23.975399 27348 consensus_queue.cc:260] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:24.125799 27415 consensus_meta.cc:287] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.770s	user 0.013s	sys 0.000s
W20260709 13:08:24.134481 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 308986us
W20260709 13:08:24.135254 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 344181us
I20260709 13:08:24.162866 25909 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:24.167484 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:24.136853 27473 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:24.128265 27331 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:24.138971 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:24.174398 25909 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:24.158080 27468 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:24.124673 27403 consensus_queue.cc:1059] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
W20260709 13:08:24.202549 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:24.158747 27520 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:24.154871 25395 rpcz_store.cc:276] 0709 13:08:22.848453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.456166 (+607713us) service_pool.cc:224] Handling call
0709 13:08:23.456456 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:23.456657 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:08:23.888702 (+432045us) log.cc:844] Serialized 14 byte log entry
0709 13:08:24.135978 (+247276us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:24.136049 (+    71us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:24.136112 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:24.136198 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:24.152919 (+ 16721us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":653661,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":786,"spinlock_wait_cycles":17906432,"thread_start_us":646095,"threads_started":2}
I20260709 13:08:24.137039 27515 consensus_queue.cc:1059] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260709 13:08:24.231256 27523 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 13:08:24.231595 27505 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
I20260709 13:08:24.173383 27516 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:24.181751 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1743) took 1201 ms. Trace:
I20260709 13:08:24.218628 27348 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:24.178483 27519 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:24.171764 27297 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.059s
W20260709 13:08:24.243007 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 68725.1us
I20260709 13:08:24.180685 27331 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:24.310109 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:24.257766 27517 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
W20260709 13:08:24.259907 27504 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260709 13:08:24.267051 27447 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:24.227828 27520 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.224583 27400 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5.
W20260709 13:08:24.269018 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 161053us
I20260709 13:08:24.301295 27415 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:24.237105 27513 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:24.307022 25909 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:24.338876 25909 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:24.175210 27435 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
W20260709 13:08:24.252751 27514 log.cc:927] Time spent T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 13:08:24.224834 27468 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:24.321815 25240 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.008s	sys 0.020s
W20260709 13:08:24.341799 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 252729us
I20260709 13:08:24.344408 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:24.346045 25240 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.725s	user 0.009s	sys 0.021s
I20260709 13:08:24.384359 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1558) took 1732 ms. Trace:
I20260709 13:08:24.384799 25242 rpcz_store.cc:276] 0709 13:08:22.652107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.335212 (+683105us) service_pool.cc:224] Handling call
0709 13:08:23.365588 (+ 30376us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:23.368750 (+  3162us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:23.368789 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:23.398145 (+ 29356us) spinlock_profiling.cc:243] Waited 7.8 ms on lock 0x7b2c000014c8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f29755aba11 00007f29755bb372 00007f2976ceda01 00007f2976cb243a 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2
0709 13:08:23.410236 (+ 12091us) log.cc:844] Serialized 26 byte log entry
0709 13:08:23.826178 (+415942us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:23.826216 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:23.826279 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:24.360522 (+534243us) raft_consensus.cc:1685] finished
0709 13:08:24.379146 (+ 18624us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:24.384284 (+  5138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2703,"raft.queue_time_us":2865,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":12111,"spinlock_wait_cycles":17946624,"thread_start_us":392059,"threads_started":1,"wal-append.queue_time_us":545942}
I20260709 13:08:24.382714 25911 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:24.337765 27512 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:24.343343 27447 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:24.236080 27509 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:24.340797 27473 raft_consensus.cc:697] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 5 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:24.302700 27516 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:24.300427 25313 rpcz_store.cc:276] 0709 13:08:22.977663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.981827 (+  4164us) service_pool.cc:224] Handling call
0709 13:08:24.167930 (+1186103us) spinlock_profiling.cc:243] Waited 49.4 ms on lock 0x7b54000627d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:24.178891 (+ 10961us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":113588480}
W20260709 13:08:24.393205 27402 consensus_meta.cc:287] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.047s	user 0.012s	sys 0.001s
I20260709 13:08:24.405781 27402 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.406344 27438 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:24.406847 27438 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:24.407223 27438 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:24.407963 27402 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:24.408748 25246 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:08:24.410291 27520 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:24.410904 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1559) took 1758 ms. Trace:
I20260709 13:08:24.411270 25246 rpcz_store.cc:276] 0709 13:08:22.652374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.342349 (+689975us) service_pool.cc:224] Handling call
0709 13:08:23.342759 (+   410us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:24.409195 (+1066436us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:24.410662 (+  1467us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400317ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:24.410846 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2450931072}
I20260709 13:08:24.405022 27346 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:24.380597 27247 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.391815 27515 consensus_queue.cc:1243] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer a2a77ecf2c0941618b5bc79d4c04db61 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260709 13:08:24.413214 27342 consensus_meta.cc:287] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.914s	user 0.011s	sys 0.000s
W20260709 13:08:24.400379 27408 consensus_meta.cc:287] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.788s	user 0.011s	sys 0.000s
I20260709 13:08:24.345896 27513 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:24.425693 27530 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
W20260709 13:08:24.448915 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 281496us
I20260709 13:08:24.426358 27247 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:24.426892 25320 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: []
W20260709 13:08:24.434372 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:24.440989 27299 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:24.431047 27342 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.441949 27473 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:24.443405 25394 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:08:24.443325 26936 raft_consensus.cc:3055] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:24.428117 27415 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:24.444516 27289 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.396247 27512 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:24.457105 27365 consensus_meta.cc:287] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.565s	user 0.007s	sys 0.002s
I20260709 13:08:24.458359 27468 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:24.450428 27408 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:08:24.431241 27527 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 13:08:24.447564 27447 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:24.462641 27474 consensus_meta.cc:287] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.836s	user 0.011s	sys 0.000s
I20260709 13:08:24.428695 27443 consensus_queue.cc:260] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:24.447300 27515 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:24.494400 25358 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:24.501852 25315 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: []
W20260709 13:08:24.504432 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 116321us
I20260709 13:08:24.495236 26936 raft_consensus.cc:740] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:24.507952 25244 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 4. Current term is 5. Ops: []
W20260709 13:08:24.510098 27404 consensus_meta.cc:287] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.881s	user 0.012s	sys 0.000s
I20260709 13:08:24.509712 27447 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.510234 27342 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:24.497890 27271 raft_consensus.cc:3055] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:24.501250 27331 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:24.511561 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 289) took 1001 ms. Trace:
W20260709 13:08:24.502070 27335 consensus_meta.cc:287] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.863s	user 0.011s	sys 0.000s
I20260709 13:08:24.513623 25358 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:24.515003 27364 consensus_meta.cc:287] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.151s	user 0.013s	sys 0.000s
I20260709 13:08:24.503780 25237 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:24.527402 25358 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:24.500481 27539 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:24.528883 27525 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:08:24.529325 27259 consensus_meta.cc:287] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.610s	user 0.007s	sys 0.003s
I20260709 13:08:24.525614 27271 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:24.530519 27447 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:24.522595 27404 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:24.511217 27343 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:24.520116 27551 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:24.534548 27513 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:24.535701 27394 consensus_meta.cc:287] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 1.169s	user 0.004s	sys 0.008s
I20260709 13:08:24.500362 27365 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:24.537258 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:24.538460 27289 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:24.543862 27382 raft_consensus.cc:3055] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:24.545642 27335 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:24.525862 25318 rpcz_store.cc:276] 0709 13:08:23.510663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.510854 (+   191us) service_pool.cc:224] Handling call
0709 13:08:23.511212 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:23.878162 (+366950us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:23.878196 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:23.910871 (+ 32675us) log.cc:844] Serialized 26 byte log entry
0709 13:08:24.157027 (+246156us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:24.510734 (+353707us) log.cc:844] Serialized 14 byte log entry
0709 13:08:24.511094 (+   360us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:24.511316 (+   222us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:24.511371 (+    55us) raft_consensus.cc:1685] finished
0709 13:08:24.511382 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:24.511507 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39434,"prepare.queue_time_us":262866,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":44,"raft.queue_time_us":50991,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":1862,"spinlock_wait_cycles":5945216,"thread_start_us":854684,"threads_started":3,"wal-append.queue_time_us":374918}
I20260709 13:08:24.544819 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:24.521345 27443 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:24.552069 27259 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.558462 27516 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:24.558137 27271 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:24.563117 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:24.513221 27474 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.527539 27540 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:24.560204 27512 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:24.577047 27394 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:24.538640 27307 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:24.579705 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1315) took 1738 ms. Trace:
I20260709 13:08:24.536816 27438 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:24.505388 27415 consensus_queue.cc:1059] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 13:08:24.582381 27529 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:24.584851 27510 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:24.584939 25358 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:24.585495 27555 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:24.576531 27343 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:24.541874 27364 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:24.585948 27551 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:08:24.591429 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:24.587742 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 74458.3us
I20260709 13:08:24.578311 27404 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:24.578325 27365 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:24.593046 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:24.648090 25318 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:24.582053 27382 raft_consensus.cc:740] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:24.595131 25245 rpcz_store.cc:276] 0709 13:08:22.841085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.402762 (+561677us) service_pool.cc:224] Handling call
0709 13:08:23.403036 (+   274us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:23.403214 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:08:23.886370 (+483156us) log.cc:844] Serialized 14 byte log entry
0709 13:08:24.558430 (+672060us) spinlock_profiling.cc:243] Waited 9.26 ms on lock 0x7b18002e4620. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b3fc0a 00007f297d27f219 00007f2973bc7a40 00007f2973bc24e4 00007f2973bc2455 00007f2976caa060 00007f2976cbc716 00007f2976ccd287 00007f2976ccd208
0709 13:08:24.576608 (+ 18178us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:24.576748 (+   140us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:24.576836 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:24.579360 (+  2524us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:24.579632 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7633,"prepare.queue_time_us":727618,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":45,"raft.queue_time_us":444870,"spinlock_wait_cycles":21298048,"thread_start_us":760026,"threads_started":3,"wal-append.queue_time_us":281169}
I20260709 13:08:24.609059 25285 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:24.611011 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:24.595088 25316 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:24.613639 25942 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:24.613103 25358 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:24.616199 25394 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 4. Current term is 5. Ops: []
W20260709 13:08:24.616895 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 460992us
I20260709 13:08:24.615070 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:24.614816 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:24.594736 27271 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:24.646978 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1570) took 1088 ms. Trace:
W20260709 13:08:24.714781 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 45894.2us
I20260709 13:08:24.716696 25240 rpcz_store.cc:276] 0709 13:08:23.558308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.558494 (+   186us) service_pool.cc:224] Handling call
0709 13:08:23.586141 (+ 27647us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:23.586384 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:08:24.380388 (+794004us) log.cc:844] Serialized 14 byte log entry
0709 13:08:24.646318 (+265930us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:24.646692 (+   374us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:24.646742 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:24.646810 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:24.646927 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":96740,"prepare.queue_time_us":580949,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":134487,"thread_start_us":893193,"threads_started":2,"wal-append.queue_time_us":235947}
I20260709 13:08:24.741863 27559 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:24.729202 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 116161us
I20260709 13:08:24.592662 27533 consensus_queue.cc:1059] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.141s
I20260709 13:08:24.606889 27402 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:24.717039 25314 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:24.714953 27548 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:08:24.744520 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:24.617214 27555 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:24.746851 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:24.747037 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:24.747270 25388 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:24.747756 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:24.587577 27549 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.131s
I20260709 13:08:24.617286 27447 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:24.749707 27404 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:08:24.769515 27536 log.cc:927] Time spent T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
I20260709 13:08:24.729323 25312 raft_consensus.cc:2393] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:08:24.756719 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 147705us
I20260709 13:08:24.757067 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:24.729346 27365 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:24.759577 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:24.766767 27512 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:24.764688 25212 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:24.765558 25395 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:24.594147 27335 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:24.767597 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:24.593830 27543 consensus_queue.cc:1059] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.157s
I20260709 13:08:24.610337 27469 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.784586 25246 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:24.814599 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:24.748001 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:24.798485 25285 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:24.799767 25242 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:08:24.801978 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:24.610353 27307 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:24.729118 27327 consensus_queue.cc:1059] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20260709 13:08:24.593691 27540 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:24.812660 25315 raft_consensus.cc:2368] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4.
I20260709 13:08:24.774897 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:24.734884 27510 consensus_queue.cc:1059] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260709 13:08:24.824296 25318 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:24.811923 27551 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:24.837013 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:24.837733 25242 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:24.839008 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:24.839802 25945 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:24.841213 25395 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
W20260709 13:08:24.845180 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 80566.2us
I20260709 13:08:24.729714 25392 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
mode: GRACEFUL
new_leader_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:52318
I20260709 13:08:24.846740 25392 raft_consensus.cc:606] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Received request to transfer leadership to TS 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:24.847191 25392 raft_consensus.cc:614] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:08:24.809716 27331 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:24.788416 25938 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:24.831291 25358 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:24.832858 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:24.790235 25238 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:24.826465 25942 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260709 13:08:24.866364 25942 pending_rounds.cc:85] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 5
I20260709 13:08:24.866724 25942 pending_rounds.cc:107] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 5.6
I20260709 13:08:24.867093 25942 raft_consensus.cc:2917] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Aborting config change with OpId 5.6: Aborted: Op aborted by new leader
I20260709 13:08:24.867587 25942 raft_consensus.cc:2889] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 6 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:24.869258 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:24.863888 25319 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:24.813194 27512 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:24.859632 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 61243.1us
I20260709 13:08:24.880223 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:24.849457 27447 raft_consensus.cc:3055] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:24.784855 27402 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.836599 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:24.874717 25246 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:24.844637 27435 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:24.845360 25939 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:24.877902 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:24.877884 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:24.897390 25944 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:24.897571 25945 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:24.844465 27318 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:24.881021 25285 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:24.859542 27533 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:24.857703 27539 raft_consensus.cc:697] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:24.812810 27365 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:24.891078 27550 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260709 13:08:24.849293 27404 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 13:08:24.894006 27469 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:24.868145 27348 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:24.877869 25942 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:08:24.884191 27447 raft_consensus.cc:740] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:24.837059 27307 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:24.914523 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:24.937018 25945 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:24.883706 27512 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:24.881564 27299 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:24.783728 27548 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.974572 27402 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:24.975147 27548 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:24.774173 27559 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.977315 27559 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:24.827713 27335 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.978569 27343 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:24.979919 27335 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:24.906417 25940 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:24.981325 25390 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:24.866487 27331 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.919087 27318 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:24.987123 27331 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:24.894510 27259 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:24.941116 27557 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
W20260709 13:08:24.941727 27554 log.cc:927] Time spent T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
I20260709 13:08:24.946771 27509 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:24.989617 27364 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:24.991083 27438 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:24.950037 27305 raft_consensus.cc:3055] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:24.992365 27447 consensus_queue.cc:260] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.935351 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:24.993306 27305 raft_consensus.cc:740] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:24.993763 25942 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:24.993914 27447 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:24.994176 27305 consensus_queue.cc:260] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.994640 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:24.995291 25942 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:24.995457 27305 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:24.996891 25389 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:24.951169 27382 consensus_queue.cc:260] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.955089 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:24.957341 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:24.924973 26936 consensus_queue.cc:260] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:24.958591 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:24.960573 27556 raft_consensus.cc:3055] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:24.961431 27506 raft_consensus.cc:3055] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:24.918375 27435 raft_consensus.cc:697] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:24.963209 27307 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:24.965552 27365 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:24.966235 27519 raft_consensus.cc:3055] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:24.966681 27555 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:08:24.968216 27540 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:24.917969 27474 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:25.018867 26936 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:24.956266 27522 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.644s	user 0.003s	sys 0.002s
I20260709 13:08:25.007190 27382 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:25.008895 25945 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:24.906822 27516 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20260709 13:08:24.927268 27394 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:24.926756 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 606) took 1356 ms. Trace:
I20260709 13:08:25.120306 25941 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
W20260709 13:08:25.120955 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 242298us
I20260709 13:08:25.012638 25938 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:25.119128 25937 rpcz_store.cc:276] 0709 13:08:23.567648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.568979 (+  1331us) service_pool.cc:224] Handling call
0709 13:08:23.642621 (+ 73642us) spinlock_profiling.cc:243] Waited 3.74 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:23.642834 (+   213us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:23.951336 (+308502us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:23.958978 (+  7642us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:24.055913 (+ 96935us) log.cc:844] Serialized 26 byte log entry
0709 13:08:24.311274 (+255361us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:24.311331 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:24.333285 (+ 21954us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:24.876710 (+543425us) raft_consensus.cc:1685] finished
0709 13:08:24.880380 (+  3670us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:24.923297 (+ 42917us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":354162,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":45175,"spinlock_wait_cycles":8599424,"thread_start_us":435831,"threads_started":2,"wal-append.queue_time_us":410616}
I20260709 13:08:24.862509 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:25.117766 25939 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:25.022866 25936 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:25.131868 27331 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:25.132748 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:25.054015 27512 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.132982 27331 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:25.094470 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:25.136117 27512 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:25.136592 27331 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:25.095675 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1328) took 1128 ms. Trace:
W20260709 13:08:25.098191 25237 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 1327) took 1130 ms (client timeout 1505 ms). Trace:
I20260709 13:08:24.880878 25241 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:25.107559 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1325) took 1140 ms. Trace:
W20260709 13:08:25.137509 25237 rpcz_store.cc:269] 0709 13:08:23.967201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.967871 (+   670us) service_pool.cc:224] Handling call
0709 13:08:25.097655 (+1129784us) spinlock_profiling.cc:243] Waited 392 ms on lock 0x7b54003eb054. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:25.097935 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":902383360}
I20260709 13:08:25.138386 25244 rpcz_store.cc:276] 0709 13:08:23.966660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:24.111831 (+145171us) service_pool.cc:224] Handling call
0709 13:08:24.124802 (+ 12971us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:24.518593 (+393791us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:25.106928 (+588335us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b54000a00d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:25.107347 (+   419us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":880612224}
I20260709 13:08:25.094179 25243 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
W20260709 13:08:25.137274 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:25.023380 25944 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:25.141628 25941 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 5 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:25.041159 25945 raft_consensus.cc:3055] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:25.142565 25945 raft_consensus.cc:740] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:25.017299 27435 consensus_queue.cc:237] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:25.146006 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 264910us
I20260709 13:08:25.012758 27506 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:25.150333 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1761) took 1331 ms. Trace:
I20260709 13:08:25.151557 27474 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:25.151042 27506 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.011584 27556 raft_consensus.cc:740] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:25.153452 25314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 1855) took 1609 ms. Trace:
I20260709 13:08:25.153935 25314 rpcz_store.cc:276] 0709 13:08:23.544367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.599806 (+ 55439us) service_pool.cc:224] Handling call
0709 13:08:25.153047 (+1553241us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400352354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:25.153257 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2566996864}
I20260709 13:08:25.154357 27556 consensus_queue.cc:260] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.155908 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1757) took 1338 ms. Trace:
I20260709 13:08:25.156371 25316 rpcz_store.cc:276] 0709 13:08:23.817894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.819659 (+  1765us) service_pool.cc:224] Handling call
0709 13:08:25.155478 (+1335819us) spinlock_profiling.cc:243] Waited 720 ms on lock 0x7b54002f35d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:25.155729 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1655000960}
I20260709 13:08:25.015702 27519 raft_consensus.cc:740] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:24.885763 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:24.975800 27539 consensus_queue.cc:237] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:25.151959 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 483477us
I20260709 13:08:25.152104 27474 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:25.132525 27335 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:25.137324 25239 rpcz_store.cc:276] 0709 13:08:23.967471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:24.227439 (+259968us) service_pool.cc:224] Handling call
0709 13:08:24.231472 (+  4033us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:24.391313 (+159841us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:24.391348 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:24.391397 (+    49us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:24.391429 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:25.094897 (+703468us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b5400364a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:25.095029 (+   132us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:25.095407 (+   378us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":367097088}
I20260709 13:08:25.035053 25942 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:25.150816 25312 rpcz_store.cc:276] 0709 13:08:23.819121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.823253 (+  4132us) service_pool.cc:224] Handling call
0709 13:08:25.149845 (+1326592us) spinlock_profiling.cc:243] Waited 853 ms on lock 0x7b54000640d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:25.150063 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1960807808}
W20260709 13:08:25.007670 25389 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 1826) took 1529 ms (client timeout 1512 ms). Trace:
W20260709 13:08:25.174084 25389 rpcz_store.cc:269] 0709 13:08:23.478857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.496409 (+ 17552us) service_pool.cc:224] Handling call
0709 13:08:25.007483 (+1511074us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b54003246d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:25.007594 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3343151360}
I20260709 13:08:25.174154 27519 consensus_queue.cc:260] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:25.175585 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:25.176367 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:25.047195 27515 consensus_meta.cc:287] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.531s	user 0.011s	sys 0.000s
I20260709 13:08:25.144742 25945 consensus_queue.cc:260] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.181543 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.166926 27474 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:25.184867 27443 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:25.184980 27512 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:25.185801 27443 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:25.185997 27512 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:25.184695 25395 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:25.184361 27335 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:25.190068 25389 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:25.184741 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:25.192408 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:25.193280 25239 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:25.193470 25390 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
mode: GRACEFUL
new_leader_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:52318
I20260709 13:08:25.194527 25390 raft_consensus.cc:606] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Received request to transfer leadership to TS 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:25.194988 25390 raft_consensus.cc:614] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:08:25.188442 27512 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:25.198246 27307 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:25.199158 27307 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:25.200331 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:25.200865 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:25.201387 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:25.201696 27540 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:25.202677 27540 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.204341 25285 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:25.212994 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:25.201768 27555 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:25.182333 27506 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:25.168629 27556 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:25.212472 27307 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:25.214358 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:25.237074 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:25.208648 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:25.215164 27540 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:25.215636 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:25.223309 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 408146us
I20260709 13:08:25.223569 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.224483 25945 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:25.224565 25392 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:08:25.225783 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:25.193383 27335 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:25.228715 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.237078 27271 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:25.237771 25388 raft_consensus.cc:2376] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6. Re-sending same reply.
I20260709 13:08:25.247713 25394 raft_consensus.cc:2376] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8. Re-sending same reply.
I20260709 13:08:25.256904 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:25.259241 27522 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.203s	user 0.001s	sys 0.000s
I20260709 13:08:25.259629 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:25.259940 25386 raft_consensus.cc:2376] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4. Re-sending same reply.
I20260709 13:08:25.260512 25392 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:25.261142 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1214) took 1604 ms. Trace:
I20260709 13:08:25.237278 27318 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:25.261600 25387 rpcz_store.cc:276] 0709 13:08:23.657026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.698861 (+ 41835us) service_pool.cc:224] Handling call
0709 13:08:23.755754 (+ 56893us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:23.756234 (+   480us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:23.756274 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:23.759146 (+  2872us) log.cc:844] Serialized 26 byte log entry
0709 13:08:24.152304 (+393158us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:24.177880 (+ 25576us) log.cc:844] Serialized 14 byte log entry
0709 13:08:24.178138 (+   258us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:24.178241 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:24.444523 (+266282us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x7b18004af260. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b932a6 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b932a6 00007f2973b93240
0709 13:08:24.459120 (+ 14597us) spinlock_profiling.cc:243] Waited 3.94 ms on lock 0x7b18004af260. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc5557 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:08:24.955578 (+496458us) spinlock_profiling.cc:243] Waited 9.56 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:25.260658 (+305080us) raft_consensus.cc:1685] finished
0709 13:08:25.260707 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:25.260894 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23983,"prepare.queue_time_us":21607,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":2752,"raft.queue_time_us":37937,"raft.run_cpu_time_us":329,"raft.run_wall_time_us":20744,"spinlock_wait_cycles":35755776,"thread_start_us":365894,"threads_started":1,"wal-append.queue_time_us":474270}
I20260709 13:08:25.269354 25245 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:25.269532 25394 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:25.263881 27443 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:25.259445 27271 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:25.271833 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 67764us
I20260709 13:08:25.270328 25389 raft_consensus.cc:2393] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 7: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:25.276284 27271 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:25.277906 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:25.279567 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:25.282176 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1756) took 1464 ms. Trace:
I20260709 13:08:25.284197 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:25.292301 27305 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:08:25.292891 27307 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:25.293349 27365 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:25.293841 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:25.294481 27512 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:25.294843 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:25.297269 27515 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:25.298210 25395 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:08:25.298480 27441 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:25.342670 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.343024 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:25.343827 25240 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:25.343426 27441 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.344255 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.345216 27365 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:25.346457 27441 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:25.347009 27441 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:25.347633 27365 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:25.348141 25241 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:25.348677 25238 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:25.349726 25241 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:08:25.350698 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:25.353304 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:25.354176 25395 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:25.263962 27555 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.301384 25359 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:25.359411 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:25.357692 25285 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:25.310652 27516 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:25.308333 25312 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:08:25.310636 25243 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:25.310791 27540 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:25.310897 25313 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 13:08:25.311872 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:25.312060 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:25.312294 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.313349 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:25.313807 25314 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: [3.2-4.3]
I20260709 13:08:25.314141 25287 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:25.320128 27519 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:25.352752 27533 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:25.288739 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:25.325278 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.326212 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:25.363991 25317 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:25.311823 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:25.332580 25387 raft_consensus.cc:2376] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2. Re-sending same reply.
I20260709 13:08:25.338599 25245 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:25.332700 25320 rpcz_store.cc:276] 0709 13:08:23.817536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.817705 (+   169us) service_pool.cc:224] Handling call
0709 13:08:23.818106 (+   401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:24.466333 (+648227us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:25.281575 (+815242us) spinlock_profiling.cc:243] Waited 609 ms on lock 0x7b54004399d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:25.281941 (+   366us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1399597952}
I20260709 13:08:25.340162 25945 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:08:25.342016 25940 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:25.342538 25242 raft_consensus.cc:2393] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:25.339730 27307 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:25.302737 25936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 598) took 1778 ms (client timeout 1508 ms). Trace:
I20260709 13:08:25.307488 25246 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:25.374642 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:25.360693 27474 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:25.375789 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.376576 25387 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:25.375429 27512 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.377748 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:25.369729 25394 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:25.378327 27343 consensus_queue.cc:1059] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:08:25.378978 27512 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:25.379654 27512 raft_consensus.cc:3055] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:25.380003 27512 raft_consensus.cc:740] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:25.366925 25316 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:25.382450 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:25.383262 25238 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:25.384150 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:25.384495 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:25.369868 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:25.384938 25243 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:25.385319 25238 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:25.386385 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:25.347656 27441 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.387069 25238 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:25.387091 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:25.381165 27512 consensus_queue.cc:260] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:25.373292 25390 raft_consensus.cc:2368] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 2. Current term is 3.
I20260709 13:08:25.389439 25241 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
W20260709 13:08:25.372635 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 130697us
I20260709 13:08:25.390180 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.390897 25390 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:25.391644 25321 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:25.391848 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.375422 25942 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:25.389199 25388 raft_consensus.cc:2393] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:25.392465 27516 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.373782 27307 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:25.393610 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:25.393894 27441 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:25.394734 27516 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:25.394975 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.377592 27364 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:25.395746 25285 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:25.390936 27555 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:25.366168 25389 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:25.402966 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.404520 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:25.374248 25936 rpcz_store.cc:269] 0709 13:08:23.524314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:23.529736 (+  5422us) service_pool.cc:224] Handling call
0709 13:08:23.654882 (+125146us) spinlock_profiling.cc:243] Waited 2.17 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8deee 0000000000375939 00007f297bc011e0 00007f297bcca0c4 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:25.302269 (+1647387us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400504e54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:25.302476 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2715002112}
I20260709 13:08:25.405826 27364 raft_consensus.cc:697] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:25.405957 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.406822 25391 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:25.406963 25358 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:25.407292 27364 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:25.409505 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:25.409535 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:25.410384 25940 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:25.410756 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:25.396075 25390 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:25.411535 25941 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:25.411975 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.412688 25390 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:25.410660 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:25.407784 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.413679 25944 raft_consensus.cc:2393] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:25.413498 27474 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:25.370729 25386 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:25.414085 25391 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:25.414932 27435 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:25.415318 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:25.415509 27435 raft_consensus.cc:697] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:25.415596 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.415992 25386 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:25.416380 25391 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:25.414431 27441 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:25.417672 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:25.416603 27435 consensus_queue.cc:237] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:25.418412 25942 raft_consensus.cc:2393] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:25.419412 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:25.420276 25936 raft_consensus.cc:2393] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:25.420801 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.421667 25390 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:25.422706 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.424568 25390 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:25.372424 27540 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.429593 27540 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:25.426162 27474 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:25.430438 27474 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:25.430917 27474 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:25.431847 25243 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:25.423796 27512 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:25.425933 27307 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.320781 25284 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:25.420313 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.432785 25393 raft_consensus.cc:2376] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4. Re-sending same reply.
I20260709 13:08:25.435417 25391 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:25.330698 25392 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:25.439733 25285 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:25.440364 25387 raft_consensus.cc:2410] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 13:08:25.441488 25395 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:25.442193 25239 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:25.448827 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:25.449797 25285 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:08:25.452391 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 138410us
I20260709 13:08:25.455600 25240 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: []
W20260709 13:08:25.461786 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 141148us
I20260709 13:08:25.465583 25316 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
W20260709 13:08:25.467103 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:25.468051 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 101378us
I20260709 13:08:25.470956 25390 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:08:25.473086 25284 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:25.475080 27343 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:25.454056 27271 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:25.473443 25395 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:08:25.475550 27343 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:25.479564 27343 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:25.484211 25287 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:25.485587 25393 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:25.485977 27307 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:25.486479 26936 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:25.486922 25209 leader_election.cc:400] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4.
I20260709 13:08:25.487365 25209 leader_election.cc:403] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 13:08:25.493279 27343 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.495986 27343 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:25.497826 27474 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:25.503067 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:25.505640 25911 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:25.484167 25392 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 3. Current term is 4. Ops: []
W20260709 13:08:25.506403 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 66483.5us
W20260709 13:08:25.507336 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 72049us
I20260709 13:08:25.508219 27540 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:25.508752 27540 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:25.509210 27540 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:25.512830 27546 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:25.513497 27271 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 13:08:25.513723 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:25.515060 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:25.515347 27348 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:25.515733 27348 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:25.516060 27348 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:25.517364 27533 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4.
I20260709 13:08:25.518608 25209 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:25.519956 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:25.521355 27382 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:25.486889 26936 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:25.525467 27382 raft_consensus.cc:697] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:25.528985 26936 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:25.529886 27382 consensus_queue.cc:237] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:25.529714 25143 catalog_manager.cc:5697] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 2 to 3, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 3 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:25.533430 25359 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:25.533627 27271 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:25.506282 27474 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:25.535866 27533 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:25.536309 27533 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:25.536659 27533 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:25.539584 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:25.540480 25245 raft_consensus.cc:2393] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:25.542948 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:25.543934 27447 consensus_queue.cc:1059] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:08:25.544832 27318 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:25.545050 27365 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:25.545416 27318 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.545186 27556 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:08:25.547693 27318 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:25.548827 25359 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:25.549714 25359 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:25.551200 27365 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:25.551631 27365 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:25.551998 27365 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:25.552726 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:25.553503 27443 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:25.554179 27443 raft_consensus.cc:697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:08:25.555408 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:25.555378 27443 consensus_queue.cc:237] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:25.545156 27447 raft_consensus.cc:3055] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:25.557301 27335 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 13:08:25.557996 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:25.557335 27447 raft_consensus.cc:740] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:25.500378 25941 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:25.560117 27447 consensus_queue.cc:260] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.544308 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:25.561571 25242 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:08:25.562816 25394 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:25.584251 25386 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:25.584977 27435 raft_consensus.cc:2764] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 13:08:25.585371 27404 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:25.513497 27546 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.575376 26936 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:25.619746 27546 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:25.621821 26936 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:25.623553 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:25.623951 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:25.624464 25246 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:25.625285 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:25.625957 25242 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:08:25.627267 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:25.627532 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:25.628093 25239 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:25.629513 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:25.630836 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:25.631731 25241 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:25.632845 25238 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:25.587427 27365 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:25.634985 27365 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:25.636008 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:25.639535 27540 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.646138 25240 raft_consensus.cc:1240] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:25.611968 27404 raft_consensus.cc:697] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:25.653112 25239 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:25.653726 27331 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:08:25.671784 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:25.671988 25358 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:25.672801 27562 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:25.647467 27318 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:08:25.685372 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 61595.5us
I20260709 13:08:25.673573 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:25.647567 27533 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.696156 27533 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:25.693660 27404 consensus_queue.cc:237] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.698545 27348 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.699973 27540 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:25.704913 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:25.709729 25394 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:25.673408 27562 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.700963 27348 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:25.692935 27318 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:25.717094 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:25.718740 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:25.719764 27318 raft_consensus.cc:697] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:25.719971 25209 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:25.720789 27318 consensus_queue.cc:237] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.712706 27435 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:25.726673 25911 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:25.728442 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:25.724466 27271 raft_consensus.cc:697] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:25.734467 25386 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 13:08:25.739892 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 188815us
I20260709 13:08:25.775161 25394 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
W20260709 13:08:25.769255 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:25.777201 27540 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:25.796594 25143 catalog_manager.cc:5697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 4 to 5, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 5 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:25.799639 25911 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:25.809396 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 346916us
W20260709 13:08:25.812387 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 292231us
W20260709 13:08:25.812934 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 332756us
I20260709 13:08:25.807827 27335 consensus_queue.cc:1059] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.004s
I20260709 13:08:25.806391 27540 raft_consensus.cc:697] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:25.814646 25143 catalog_manager.cc:5697] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:08:25.817274 27546 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:08:25.836113 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:25.836866 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 163342us
I20260709 13:08:25.841037 25285 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:25.846036 25285 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:25.849303 25284 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:25.854266 27590 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:25.844373 27546 raft_consensus.cc:697] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 6 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:25.850807 27348 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:25.851560 25285 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:25.848491 27562 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:25.886865 27540 consensus_queue.cc:237] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.887115 25287 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:25.884080 27435 raft_consensus.cc:697] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:25.887964 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:25.888381 25143 catalog_manager.cc:5697] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 3 to 4, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 4 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:25.890161 27469 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:25.851367 27271 consensus_queue.cc:237] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:25.953649 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 102203us
I20260709 13:08:25.954149 27589 consensus_queue.cc:1059] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.118s
I20260709 13:08:25.954519 25285 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:25.955058 27592 consensus_queue.cc:1059] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.108s
I20260709 13:08:25.955717 27415 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.135s
I20260709 13:08:25.882064 27512 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:25.956755 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 69762.1us
W20260709 13:08:25.941880 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 215699us
I20260709 13:08:25.957438 27595 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.127s
I20260709 13:08:25.958252 27382 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:25.959050 27382 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:08:25.959265 27259 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:08:25.947090 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 62941.6us
I20260709 13:08:25.962466 27343 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:25.961696 25284 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:25.957885 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:25.959163 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:25.956700 27512 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:25.969079 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:25.885574 27364 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:25.892809 27348 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:25.954878 25139 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39572 (request call id 288) took 8083 ms. Trace:
I20260709 13:08:25.986176 26936 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:25.986533 27589 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:25.986781 26936 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:25.987244 27589 raft_consensus.cc:697] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:25.987273 26936 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:25.990796 27562 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:25.991339 27562 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:25.991767 27562 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:25.993618 27546 consensus_queue.cc:237] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:25.970978 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 168673us
I20260709 13:08:25.997761 27435 consensus_queue.cc:237] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.004655 27415 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:26.004920 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:26.005347 27415 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:25.952080 25237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1334) took 1160 ms. Trace:
I20260709 13:08:26.005198 25143 catalog_manager.cc:5697] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 3, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 3 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
W20260709 13:08:26.012529 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 53425.2us
W20260709 13:08:25.986442 27588 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 13:08:25.988402 27589 consensus_queue.cc:237] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:25.952744 27533 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:26.008320 27348 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:26.023840 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:26.030892 25239 raft_consensus.cc:2410] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:08:26.035981 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:26.038666 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 80942.6us
W20260709 13:08:26.064066 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 102471us
I20260709 13:08:26.039707 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:26.065142 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 51687.9us
W20260709 13:08:26.066674 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 121957us
I20260709 13:08:26.026432 27555 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:26.040443 25143 catalog_manager.cc:5697] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 4 to 5, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 5 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:08:26.076375 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:26.040253 27519 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:26.043131 25388 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:52318
I20260709 13:08:26.083068 25284 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:26.067394 27469 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:08:26.084461 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:26.085929 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 131509us
I20260709 13:08:26.043838 27364 raft_consensus.cc:3055] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:26.070576 27331 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:26.098201 25240 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:26.100692 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:26.107661 25245 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:26.101446 25390 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:26.080302 27415 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:26.111037 27415 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:26.111557 27415 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.088783 27343 raft_consensus.cc:697] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:26.085587 27469 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:26.117883 27595 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:26.085659 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:26.116019 25359 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:26.124110 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:26.125279 25911 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:26.090376 27364 raft_consensus.cc:740] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:26.135248 25242 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:26.136232 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:26.137879 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:26.101678 27307 consensus_queue.cc:1059] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 13:08:26.088461 27519 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.120445 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:26.079450 27555 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.156898 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:26.120283 27259 raft_consensus.cc:697] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:26.130522 27469 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:26.012666 25237 rpcz_store.cc:276] 0709 13:08:24.792091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:25.304090 (+511999us) service_pool.cc:224] Handling call
0709 13:08:25.304704 (+   614us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:25.310979 (+  6275us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:08:25.454984 (+144005us) log.cc:844] Serialized 14 byte log entry
0709 13:08:25.951677 (+496693us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:25.951736 (+    59us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:25.951784 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:25.951853 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:25.952009 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":269801,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":176946,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":102,"thread_start_us":580542,"threads_started":3,"wal-append.queue_time_us":348942}
W20260709 13:08:26.157831 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 89983.1us
I20260709 13:08:26.157049 27595 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:26.157431 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:26.162009 27594 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.277s	user 0.000s	sys 0.004s
W20260709 13:08:26.163353 27599 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 13:08:25.975605 25139 rpcz_store.cc:276] 0709 13:08:17.743718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:17.743968 (+   250us) service_pool.cc:224] Handling call
0709 13:08:25.824712 (+8080744us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:08:24.232001 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:08:24.232769 (+   768us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:08:24.232810 (+    41us) write_op.cc:435] Acquired schema lock
0709 13:08:24.232832 (+    22us) tablet.cc:662] Decoding operations
0709 13:08:24.338467 (+105635us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:08:24.345626 (+  7159us) write_op.cc:641] Partition lock acquired for write op
0709 13:08:24.360560 (+ 14934us) tablet.cc:685] Acquiring locks for 11 operations
0709 13:08:24.501539 (+140979us) tablet.cc:701] Row locks acquired
0709 13:08:24.501558 (+    19us) write_op.cc:260] PREPARE: finished
0709 13:08:24.501718 (+   160us) write_op.cc:270] Start()
0709 13:08:24.501848 (+   130us) write_op.cc:276] Timestamp: P: 1783602504501664 usec, L: 0
0709 13:08:24.501867 (+    19us) op_driver.cc:348] REPLICATION: starting
0709 13:08:24.577932 (+ 76065us) log.cc:844] Serialized 4397 byte log entry
0709 13:08:24.963300 (+385368us) spinlock_profiling.cc:243] Waited 3.04 ms on lock 0x7b5400470ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cade28 00007f297a6e8581 00007f297a6e7d58 00007f297a6ec9f1 00007f297a6ec9a9 00007f297a6ec939 00007f297a6ec901 00007f297a6ebb2d 000000000038f354 000000000038f199
0709 13:08:25.055785 (+ 92485us) op_driver.cc:464] REPLICATION: finished
0709 13:08:25.064141 (+  8356us) write_op.cc:301] APPLY: starting
0709 13:08:25.064247 (+   106us) tablet.cc:1367] starting BulkCheckPresence
0709 13:08:25.064567 (+   320us) tablet.cc:1370] finished BulkCheckPresence
0709 13:08:25.064595 (+    28us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:08:25.120146 (+ 55551us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:08:25.120673 (+   527us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0709 13:08:25.120772 (+    99us) write_op.cc:312] APPLY: finished
0709 13:08:25.121818 (+  1046us) log.cc:844] Serialized 83 byte log entry
0709 13:08:25.122229 (+   411us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:25.122818 (+   589us) write_op.cc:454] Released schema lock
0709 13:08:25.122923 (+   105us) write_op.cc:341] FINISH: Updating metrics
0709 13:08:25.123632 (+   709us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:25.123677 (+    45us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":20009,"reported_tablets":25,"spinlock_wait_cycles":88832,"child_traces":[["op",{"apply.queue_time_us":8027,"apply.run_cpu_time_us":6103,"apply.run_wall_time_us":59401,"num_ops":11,"prepare.queue_time_us":363038,"prepare.run_cpu_time_us":24200,"prepare.run_wall_time_us":731500,"raft.queue_time_us":163451,"raft.run_cpu_time_us":1117,"raft.run_wall_time_us":2052,"replication_time_us":553727,"spinlock_wait_cycles":6989568,"thread_start_us":669429,"threads_started":4,"wal-append.queue_time_us":311808}]]}
I20260709 13:08:26.109004 27590 raft_consensus.cc:697] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:08:26.269809 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:26.164228 27343 consensus_queue.cc:237] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:26.323940 27556 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:26.324316 27365 consensus_queue.cc:1059] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:08:26.128947 25212 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:26.325402 25287 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:26.094559 25388 raft_consensus.cc:606] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:26.157334 25943 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:08:26.363087 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:26.327096 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:26.364235 25395 raft_consensus.cc:2410] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:08:26.328116 25244 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:08:26.330354 25945 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 13:08:26.334070 25246 raft_consensus.cc:2393] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:26.337728 25240 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:26.329054 25243 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:26.330364 25387 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:26.335844 25284 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:26.341318 25242 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
W20260709 13:08:26.360837 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:26.341727 27318 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:26.368191 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:26.403123 25942 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:26.328646 27443 raft_consensus.cc:3055] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 13:08:26.372051 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:26.337543 27307 consensus_queue.cc:1243] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer f646dfd532974dceadfe00005d2bbf59 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:08:26.336970 25237 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:08:26.335460 27555 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:26.331085 27441 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:26.368285 25388 raft_consensus.cc:614] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:08:26.327543 27331 raft_consensus.cc:697] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:26.333425 27519 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:26.382107 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:26.347220 27271 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:26.337446 27595 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:26.348125 27506 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:26.435737 27506 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:26.353320 27512 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:26.410913 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 282054us
I20260709 13:08:26.439005 27382 consensus_queue.cc:1059] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:08:26.439957 27382 raft_consensus.cc:3055] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:26.440340 27382 raft_consensus.cc:740] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:26.441406 27271 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:26.443799 27271 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:26.418550 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:26.446314 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:26.447131 25246 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:26.447921 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:26.366582 25945 pending_rounds.cc:85] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 5
I20260709 13:08:26.360410 25936 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:08:26.381297 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:26.450537 27622 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:26.451015 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:26.451151 27622 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.449029 25246 raft_consensus.cc:2410] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:08:26.361512 26936 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.453780 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:26.381733 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:26.455641 25937 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:26.456064 27596 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:26.381779 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:26.456712 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:26.457361 25939 raft_consensus.cc:2393] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:26.457541 25937 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:26.458408 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:26.454552 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:26.459408 25939 raft_consensus.cc:2410] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:08:26.433722 27516 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:26.434937 27606 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:26.460142 27606 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:26.441833 27605 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:26.422000 27555 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:26.422369 25943 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:26.461493 27605 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.461860 27555 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.381256 27598 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:26.442467 27590 consensus_queue.cc:237] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.463893 27598 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.464141 27516 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:26.466046 27331 consensus_queue.cc:237] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.442382 27382 consensus_queue.cc:260] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:26.424443 27441 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:26.449342 25945 pending_rounds.cc:107] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 13:08:26.449331 27614 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:26.470649 27382 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:26.470832 25945 raft_consensus.cc:2889] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:26.471129 27614 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:26.472769 27612 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 13:08:26.334739 27259 consensus_queue.cc:237] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:26.433571 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:26.476722 25238 raft_consensus.cc:2393] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:26.364748 27556 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:26.478819 25944 raft_consensus.cc:2368] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5.
I20260709 13:08:26.478961 27556 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:26.388114 27447 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:26.469548 25244 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 13:08:26.507078 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:26.511857 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:26.402415 27318 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:26.537024 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:26.468859 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:26.409818 27443 raft_consensus.cc:740] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:08:26.565356 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:26.432353 25941 raft_consensus.cc:2393] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 5: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:26.567214 25244 raft_consensus.cc:3055] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:08:26.567354 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 231602us
I20260709 13:08:26.587249 25244 raft_consensus.cc:740] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:26.593075 25284 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:26.556139 27318 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:26.566933 25246 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
W20260709 13:08:26.597878 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 165635us
W20260709 13:08:26.598635 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 273340us
I20260709 13:08:26.598696 25244 consensus_queue.cc:260] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.600026 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:26.561256 25936 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:26.550912 27592 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:26.600723 25244 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:26.583813 26936 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:26.607362 27620 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:26.621443 27623 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:26.613565 27615 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 13:08:26.623759 27348 consensus_meta.cc:287] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.559s	user 0.007s	sys 0.006s
I20260709 13:08:26.607182 27516 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:26.628636 25243 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
W20260709 13:08:26.697927 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 72191.9us
I20260709 13:08:26.624684 27443 consensus_queue.cc:260] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:26.613750 27614 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:26.630647 27474 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:26.631866 27441 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:26.556161 27365 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.083s
W20260709 13:08:26.635636 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 197240us
I20260709 13:08:26.637228 25394 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:08:26.708858 25390 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
mode: GRACEFUL
new_leader_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:52318
I20260709 13:08:26.632332 27506 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:26.629808 27605 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:26.636406 27620 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:26.629961 27622 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:26.586792 27364 consensus_queue.cc:260] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.715435 27533 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:26.718540 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:26.771358 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 171402us
I20260709 13:08:26.772745 25244 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:08:26.720005 27623 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:26.697806 27348 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:26.775291 27605 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:08:26.720017 27441 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:26.715720 27519 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
W20260709 13:08:26.776600 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 47216.5us
I20260709 13:08:26.775445 27533 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.728636 27614 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:26.725929 27620 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:26.737391 27606 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:26.742184 27443 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:26.780577 27562 consensus_meta.cc:287] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.780s	user 0.013s	sys 0.008s
I20260709 13:08:26.740412 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:26.753315 27615 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:26.753571 27555 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:26.767154 27598 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:26.708360 27592 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:26.793622 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:26.808866 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:26.809508 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:26.812696 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:26.816182 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:26.793035 27614 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:26.818145 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:26.780792 27469 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.843775 25244 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:26.844713 25909 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:26.844731 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:26.810077 25237 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:26.818102 27555 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:26.858378 25316 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:08:26.798909 27562 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.862345 27592 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:26.845888 27615 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:26.873476 27630 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:26.878181 25359 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:26.906656 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 94745.2us
I20260709 13:08:26.907660 25212 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:26.878468 25238 raft_consensus.cc:1275] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:26.881841 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:26.808383 27441 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:26.850389 27622 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:26.891848 27364 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:26.894610 27415 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:26.895138 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:26.915812 25246 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 7: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:26.896397 27271 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.820837 27624 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:26.916688 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 102981us
I20260709 13:08:26.919407 25245 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:26.861883 27533 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:26.821244 27614 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:26.909410 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:26.909931 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:26.911075 25358 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:26.810211 27606 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260709 13:08:26.913391 25316 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:08:26.797353 27623 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.918680 27382 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election won for term 4
W20260709 13:08:26.921686 25240 consensus_meta.cc:287] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.537s	user 0.013s	sys 0.002s
I20260709 13:08:26.922526 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:26.922773 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:26.918558 25938 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:26.928964 25212 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:26.917351 25940 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:26.931236 25358 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:26.930159 25319 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:26.917681 27365 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:26.932404 25240 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:26.947062 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:26.957839 25394 raft_consensus.cc:2393] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:08:26.936825 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:26.946714 25285 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:26.942116 25212 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:26.934988 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:26.935645 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:26.904815 27555 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.940990 27307 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:26.981601 27307 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:26.982165 27555 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:26.933590 27635 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 13:08:26.983326 27307 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:26.935052 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:26.959998 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:26.984120 25395 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:26.960584 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:26.960791 25313 raft_consensus.cc:2393] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:26.961320 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:26.984815 25387 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:26.985143 25392 raft_consensus.cc:2393] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:26.961445 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:26.959368 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:26.941705 27305 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:26.986578 25320 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 1.
I20260709 13:08:26.931231 27447 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:26.972875 27335 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:26.962698 27587 raft_consensus.cc:3055] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 13:08:26.963776 27632 log.cc:927] Time spent T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20260709 13:08:26.917446 27638 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260709 13:08:26.964311 25388 raft_consensus.cc:2410] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:08:26.966493 27623 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:26.932952 27602 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 13:08:26.970439 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:26.970978 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 115399us
I20260709 13:08:26.971450 27556 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:26.971755 27546 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:08:26.972057 27562 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:27.107805 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:27.108613 25394 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 1.
I20260709 13:08:27.056262 27590 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:27.109619 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:27.110328 25394 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:26.972723 27343 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:26.975205 27622 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:26.975962 27607 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:08:26.977228 27441 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:26.978250 27592 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:27.111963 25209 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:26.979247 27469 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:26.976003 27348 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:26.980252 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:27.138164 25241 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
W20260709 13:08:26.997259 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:26.938634 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:27.010941 25316 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:27.013855 25315 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:08:27.015820 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 69320us
I20260709 13:08:26.962311 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:26.917568 27605 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260709 13:08:26.941545 27382 raft_consensus.cc:697] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:26.984277 27365 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:27.067569 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:27.147940 25319 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 6: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:08:27.079267 25143 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.003s	sys 0.008s
I20260709 13:08:27.068440 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:27.067746 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:27.075515 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:27.062052 25389 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
W20260709 13:08:27.103911 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 161864us
I20260709 13:08:27.112588 25911 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:27.069259 27533 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 13:08:27.122627 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 179151us
I20260709 13:08:27.124476 25395 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:27.069046 27271 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:27.056646 27587 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:27.143567 27343 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:27.127785 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:27.322419 25394 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 6: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:27.246966 27365 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:27.141510 25393 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:27.141577 25285 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:27.143203 25390 raft_consensus.cc:606] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Received request to transfer leadership to TS 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:27.332677 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:27.333310 25394 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:08:27.128934 25284 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.021s	sys 0.025s
I20260709 13:08:26.925944 27624 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:27.149523 25143 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.003s	sys 0.008s
I20260709 13:08:27.138031 27623 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:27.180251 25387 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
W20260709 13:08:27.126235 25937 consensus_meta.cc:287] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.668s	user 0.009s	sys 0.005s
I20260709 13:08:27.179761 25320 raft_consensus.cc:2410] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 13:08:27.209465 27556 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:27.151508 27606 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:27.236666 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 500003us
I20260709 13:08:27.317723 27630 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:27.339332 25318 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:27.179270 27435 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:27.345906 27435 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:27.346226 27651 log.cc:927] Time spent T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:08:27.346227 25941 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:08:27.348021 27343 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:27.348349 27435 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:27.349321 27447 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:27.349383 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:27.349773 27647 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 13:08:27.350135 25938 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:27.349980 27447 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:27.350354 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:27.350984 25389 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:27.351506 25943 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:08:27.351974 27447 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:27.353341 27623 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:27.355456 27623 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:27.339742 27271 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:27.357652 27271 raft_consensus.cc:697] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:27.358616 27271 consensus_queue.cc:237] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:27.336215 27443 consensus_meta.cc:287] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.541s	user 0.008s	sys 0.003s
I20260709 13:08:27.130587 27590 raft_consensus.cc:697] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:27.209218 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:27.247601 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 66383.4us
W20260709 13:08:27.252933 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 142205us
I20260709 13:08:27.179941 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:27.125483 27622 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:27.289538 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:27.261376 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:27.248692 25359 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:27.298231 27641 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.159s	user 0.000s	sys 0.000s
I20260709 13:08:27.298394 27382 consensus_queue.cc:237] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:27.299248 27633 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 13:08:27.304605 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1591) took 1247 ms. Trace:
I20260709 13:08:27.378252 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:27.378165 25239 rpcz_store.cc:276] 0709 13:08:26.040671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:26.064693 (+ 24022us) service_pool.cc:224] Handling call
0709 13:08:26.075438 (+ 10745us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:26.120748 (+ 45310us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:26.120781 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:26.162893 (+ 42112us) log.cc:844] Serialized 26 byte log entry
0709 13:08:26.588213 (+425320us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:26.588268 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:26.588361 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:27.257047 (+668686us) raft_consensus.cc:1685] finished
0709 13:08:27.287296 (+ 30249us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:27.287457 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":206798,"raft.queue_time_us":39361,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":10420,"thread_start_us":204347,"threads_started":1,"wal-append.queue_time_us":433949}
W20260709 13:08:27.126580 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:26.940501 27318 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:27.235960 27540 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:27.381188 27623 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:27.382714 27540 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:27.381793 25911 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:27.384186 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:27.300890 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:27.334514 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 193006us
I20260709 13:08:27.385524 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:27.392807 25285 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:27.386046 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:27.395179 25241 raft_consensus.cc:2410] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 13:08:27.335904 25284 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.022s	sys 0.025s
I20260709 13:08:27.336658 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:27.317553 27555 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:27.179301 27592 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:27.398823 27622 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:27.398933 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:27.399230 27607 consensus_queue.cc:1059] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 13:08:27.344239 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:27.400135 27607 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:27.400503 27607 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:27.398216 27435 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:27.400947 27607 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:27.401096 27435 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:27.401571 27435 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:27.382473 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:27.385282 27415 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20260709 13:08:27.384868 27540 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:27.386016 27623 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:27.332160 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:27.341212 25937 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
W20260709 13:08:27.421753 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 828707us
I20260709 13:08:27.316880 27469 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260709 13:08:27.366587 25386 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:27.366962 25943 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:08:27.362335 25315 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:08:27.370455 25209 leader_election.cc:400] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5.
W20260709 13:08:27.372723 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 124059us
I20260709 13:08:27.373713 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:27.432375 25245 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:27.373872 27590 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:27.374347 25940 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
W20260709 13:08:27.376363 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 233779us
I20260709 13:08:27.382051 27318 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:27.397275 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:27.385735 27519 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:27.382454 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:27.398198 25316 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 13:08:27.405450 25942 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:27.409214 25243 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.003s	sys 0.032s
W20260709 13:08:27.452823 25243 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.003s	sys 0.032s
I20260709 13:08:27.344051 25212 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
W20260709 13:08:27.454173 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 119397us
I20260709 13:08:27.454932 25212 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:27.456285 25212 leader_election.cc:400] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 2. Current term is 3.
I20260709 13:08:27.456542 25212 leader_election.cc:403] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 13:08:27.456660 25940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 615) took 1092 ms. Trace:
I20260709 13:08:27.410446 25393 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
W20260709 13:08:27.361442 27620 consensus_meta.cc:287] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.550s	user 0.009s	sys 0.001s
I20260709 13:08:27.411885 25944 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:27.428834 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:27.467235 25320 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:27.364108 27615 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:27.432483 25285 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:27.384971 25941 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:27.400156 27592 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:27.438194 25287 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:27.442351 25390 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:27.382072 27642 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:27.450296 27636 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260709 13:08:27.450186 27539 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:27.457587 25212 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:27.460311 27624 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:27.433461 27555 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:27.433331 27343 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:27.457829 27562 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:27.462257 25319 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 13:08:27.462383 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 96065.3us
I20260709 13:08:27.510425 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:27.444387 27415 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260709 13:08:27.463994 25315 raft_consensus.cc:1275] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:27.457305 27533 raft_consensus.cc:2764] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260709 13:08:27.476356 25942 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:27.432979 25392 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:08:27.485088 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:27.490826 27652 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:27.458182 27305 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 2. Current term is 3.
I20260709 13:08:27.496503 27539 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:27.491621 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:27.493865 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:27.447965 27630 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
I20260709 13:08:27.495512 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:27.540493 25239 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:27.495877 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:27.427625 27623 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:27.501814 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 63710.5us
I20260709 13:08:27.509213 27343 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:27.543835 25939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 622) took 1128 ms. Trace:
I20260709 13:08:27.547252 27343 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:27.541450 27539 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:27.547375 25939 rpcz_store.cc:276] 0709 13:08:26.415759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:26.460121 (+ 44362us) service_pool.cc:224] Handling call
0709 13:08:26.460526 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:26.461035 (+   509us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:26.461066 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:26.520378 (+ 59312us) log.cc:844] Serialized 26 byte log entry
0709 13:08:26.739307 (+218929us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:26.766895 (+ 27588us) log.cc:844] Serialized 14 byte log entry
0709 13:08:26.784277 (+ 17382us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:26.790264 (+  5987us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:27.543544 (+753280us) raft_consensus.cc:1685] finished
0709 13:08:27.543577 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:27.543747 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":5497,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":1939,"raft.queue_time_us":10141,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":6105,"thread_start_us":204567,"threads_started":1,"wal-append.queue_time_us":710577}
I20260709 13:08:27.549042 27652 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:27.504001 25394 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
W20260709 13:08:27.504855 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 58223.2us
I20260709 13:08:27.552554 25908 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:27.552981 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:27.436820 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 620) took 1042 ms. Trace:
I20260709 13:08:27.509488 27562 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:08:27.491292 27614 consensus_meta.cc:287] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.524s	user 0.006s	sys 0.003s
I20260709 13:08:27.538206 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:27.457056 25940 rpcz_store.cc:276] 0709 13:08:26.364696 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:26.368451 (+  3755us) service_pool.cc:224] Handling call
0709 13:08:26.881715 (+513264us) spinlock_profiling.cc:243] Waited 1.37 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8deee 0000000000375939 00007f297bc011e0 00007f297bcca0c4 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:27.444790 (+563075us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3140864}
I20260709 13:08:27.541409 25320 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:27.543061 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:27.543195 25241 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
W20260709 13:08:27.502243 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 69861.5us
I20260709 13:08:27.508879 27615 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:08:27.511917 25318 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:08:27.553617 25319 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:27.509995 27592 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:27.509315 27318 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:27.482317 27519 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:27.556592 25315 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:27.485733 27620 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:27.550099 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:27.570891 25358 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:27.508527 27555 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:27.573946 25908 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:27.575999 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:27.562505 27415 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:27.592756 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:27.481535 27587 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:27.556294 27533 raft_consensus.cc:2764] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:08:27.577852 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:27.602116 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:27.602994 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:27.603714 25243 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:27.561901 27562 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:27.596603 25908 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:27.570340 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:27.608572 25908 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:27.594151 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:27.598585 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:27.499781 27642 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:27.584132 25244 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:27.566895 27615 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:27.610231 25319 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:27.591876 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:27.573011 27305 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:27.561532 27623 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 13:08:27.616158 25239 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:27.585573 27469 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:27.556134 25937 rpcz_store.cc:276] 0709 13:08:26.389301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:26.456376 (+ 67075us) service_pool.cc:224] Handling call
0709 13:08:27.431628 (+975252us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:27.614238 25358 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:27.565940 27614 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:27.622929 27642 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:27.618264 27649 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:27.601111 27643 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:27.650274 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:27.583114 27318 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:27.625803 25313 raft_consensus.cc:2393] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 6: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:27.626358 27519 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:27.591876 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:27.598919 27546 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:27.647102 27305 raft_consensus.cc:697] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:27.665905 25321 raft_consensus.cc:2393] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 3: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:27.629812 25944 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:27.621028 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:27.646864 25358 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:27.644302 25940 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:27.644390 25939 raft_consensus.cc:2393] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:27.584422 27635 consensus_queue.cc:1059] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 13:08:27.659741 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:27.662447 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:27.670333 25387 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:27.649866 27595 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:27.681645 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:27.682571 25319 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 7: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:27.670404 25241 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:27.620548 27615 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:27.684419 25358 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:27.655519 27556 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:27.615063 27607 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:27.686157 25243 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:27.663933 27587 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:27.645908 25941 raft_consensus.cc:2376] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5. Re-sending same reply.
I20260709 13:08:27.694072 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:27.696041 27649 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:27.696637 27443 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:27.697837 25239 raft_consensus.cc:2393] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:27.698096 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:27.665272 25237 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:27.869470 27443 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:27.869859 27595 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:27.869974 27443 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:27.701378 25942 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:27.657989 27643 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:27.723206 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:27.682056 27447 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:27.690973 27605 consensus_queue.cc:1059] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260709 13:08:27.649318 27469 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:27.847077 27364 consensus_meta.cc:287] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.928s	user 0.007s	sys 0.011s
I20260709 13:08:27.730970 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:27.885778 25319 raft_consensus.cc:2393] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:27.738929 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:27.691666 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:08:27.878435 27606 consensus_meta.cc:287] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.517s	user 0.003s	sys 0.012s
W20260709 13:08:27.858501 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 400996us
I20260709 13:08:27.848963 27318 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:27.926609 27318 raft_consensus.cc:3055] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:27.877806 27649 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:27.927075 27318 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:27.927192 27469 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:27.928141 27318 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:27.928838 25937 raft_consensus.cc:2393] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:27.914996 27607 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:27.931577 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 421306us
I20260709 13:08:27.932754 27318 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:27.933326 25939 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:27.899436 27592 raft_consensus.cc:697] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:27.934547 27592 consensus_queue.cc:237] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:27.936789 27615 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:27.937258 27606 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:27.937601 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:27.938351 27435 consensus_meta.cc:287] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.536s	user 0.006s	sys 0.008s
I20260709 13:08:27.938777 27435 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:27.940796 27435 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:27.944171 27665 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.082s	user 0.002s	sys 0.000s
I20260709 13:08:27.946377 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:27.947365 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:27.700631 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:27.862535 27645 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.235s	user 0.002s	sys 0.006s
I20260709 13:08:27.882025 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:27.894170 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 209816us
I20260709 13:08:27.954093 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:27.954308 27638 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:08:27.954847 25315 raft_consensus.cc:2393] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:27.866264 25390 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
W20260709 13:08:27.955891 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:27.748349 27305 consensus_queue.cc:237] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:27.882710 27657 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.194s	user 0.001s	sys 0.004s
I20260709 13:08:27.861660 27441 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:27.879765 27595 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:27.909175 27670 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:27.909432 27539 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:28.000734 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 1863) took 1794 ms. Trace:
I20260709 13:08:27.954798 27556 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:28.001853 25285 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:27.913117 27555 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:27.915184 27562 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:27.915977 27343 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:27.916291 27605 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:27.861042 25320 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:27.868336 25241 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
W20260709 13:08:27.946632 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 338071us
I20260709 13:08:27.947592 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:28.011734 25319 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:27.881835 27656 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.342s	user 0.004s	sys 0.004s
I20260709 13:08:27.959517 25317 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:27.904027 25942 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
W20260709 13:08:27.879703 25389 consensus_meta.cc:287] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.528s	user 0.011s	sys 0.008s
I20260709 13:08:27.960119 27606 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:27.960152 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:27.965332 25314 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 13:08:28.022871 25314 pending_rounds.cc:85] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 5
I20260709 13:08:28.023100 25314 pending_rounds.cc:107] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 13:08:27.966317 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:27.965904 25911 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:27.975461 27335 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:27.976490 27643 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:27.879482 27447 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:28.037608 27447 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:27.979633 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:28.041132 27443 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:28.042088 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:28.042510 27624 raft_consensus.cc:3055] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:28.042929 27624 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:28.043370 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:28.044014 25315 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:28.043766 27624 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:27.881584 27620 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:27.928838 27649 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:28.046245 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 80014.8us
I20260709 13:08:28.047120 25212 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:28.049459 25936 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:28.050065 27671 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:28.050804 27671 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:28.024415 27606 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:28.001266 25391 rpcz_store.cc:276] 0709 13:08:26.205917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:26.327287 (+121370us) service_pool.cc:224] Handling call
0709 13:08:26.329833 (+  2546us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:26.330155 (+   322us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:26.330180 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:26.870786 (+540606us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b2c000e1828. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f29755aba11 00007f29755bb372 00007f2976ceda01 00007f2976cb243a 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2
0709 13:08:26.871161 (+   375us) log.cc:844] Serialized 26 byte log entry
0709 13:08:27.149157 (+277996us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:27.149193 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:27.149253 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:27.982594 (+833341us) raft_consensus.cc:1685] finished
0709 13:08:27.982615 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:28.000666 (+ 18051us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":308,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":73,"spinlock_wait_cycles":23947648,"thread_start_us":277614,"threads_started":1,"wal-append.queue_time_us":684146}
I20260709 13:08:27.978359 27607 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260709 13:08:28.002512 27441 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:28.018491 25320 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:08:28.066687 25320 raft_consensus.cc:1240] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 1. Current term is 2. Ops: []
I20260709 13:08:28.009284 25285 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:08:28.024871 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 59027.5us
I20260709 13:08:28.020268 27343 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:28.004053 27587 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:28.029078 27539 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:28.012022 27614 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:28.030234 27662 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.168s	user 0.000s	sys 0.000s
I20260709 13:08:28.030385 27562 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:28.031795 27595 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:28.021925 27605 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:28.024397 25389 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:28.007926 27674 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:28.007308 27615 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:28.034941 27556 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 13:08:28.035923 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:28.082448 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:28.083752 27674 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:28.084201 27674 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:28.008301 27670 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:28.041651 27652 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:28.045161 25939 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:28.054894 25359 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:28.048790 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:28.087790 25212 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:28.045059 25315 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:28.014139 25908 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:28.023321 25314 raft_consensus.cc:2889] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:28.068316 25942 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:08:28.089365 25246 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:08:28.071509 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 62421.4us
I20260709 13:08:28.044962 27624 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:28.071224 25319 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:28.045863 27443 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:28.075270 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:28.109860 27365 consensus_meta.cc:287] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.695s	user 0.012s	sys 0.003s
I20260709 13:08:28.123513 27623 raft_consensus.cc:2764] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 13:08:28.173026 27365 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:28.173789 25319 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260709 13:08:28.111552 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 56720.3us
I20260709 13:08:28.141099 25316 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:08:28.175082 25359 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:28.073899 27620 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:28.175839 27620 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:28.176410 27620 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:28.079514 27587 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:28.115506 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 1869) took 1154 ms. Trace:
I20260709 13:08:28.174527 25388 raft_consensus.cc:2393] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:28.175880 27318 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:28.178488 27587 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:28.143204 25314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 306) took 1235 ms. Trace:
I20260709 13:08:28.145704 25312 raft_consensus.cc:1240] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:28.112164 27643 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:08:28.149468 25317 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 304) took 1242 ms (client timeout 1562 ms). Trace:
I20260709 13:08:28.257649 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:28.121573 27441 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:28.182006 27382 raft_consensus.cc:2764] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:08:28.245247 25387 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:28.198971 27606 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:28.153401 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 65666.5us
W20260709 13:08:28.154335 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 140737us
I20260709 13:08:28.276451 25284 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:28.277765 25284 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:28.137878 25320 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 2. Current term is 3. Ops: []
I20260709 13:08:28.130056 25315 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:28.090286 25285 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:28.091677 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:28.208562 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 185457us
I20260709 13:08:28.251644 27533 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:28.241379 25321 rpcz_store.cc:276] 0709 13:08:26.961506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:27.056591 (+ 95085us) service_pool.cc:224] Handling call
0709 13:08:28.115251 (+1058660us) spinlock_profiling.cc:243] Waited 515 ms on lock 0x7b5400443754. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:28.115434 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1184201984}
I20260709 13:08:28.176326 27365 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:28.250864 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:28.282496 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:28.121610 27602 raft_consensus.cc:3055] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:28.256623 25314 rpcz_store.cc:276] 0709 13:08:26.907935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:26.910637 (+  2702us) service_pool.cc:224] Handling call
0709 13:08:26.922427 (+ 11790us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:28.097314 (+1174887us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:28.142902 (+ 45588us) spinlock_profiling.cc:243] Waited 995 ms on lock 0x7b5400275e54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:28.143131 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2288196608}
I20260709 13:08:28.257622 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:28.309396 25284 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:28.322996 25391 raft_consensus.cc:2393] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:28.090317 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:28.182170 27605 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:28.258024 25317 rpcz_store.cc:269] 0709 13:08:26.907131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:26.910498 (+  3367us) service_pool.cc:224] Handling call
0709 13:08:28.149313 (+1238815us) spinlock_profiling.cc:243] Waited 985 ms on lock 0x7b5400274f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:28.149415 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2266389888}
I20260709 13:08:28.111173 25245 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 13:08:28.122010 27556 raft_consensus.cc:697] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 5 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:28.116370 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 1866) took 1155 ms. Trace:
I20260709 13:08:28.265260 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:28.265338 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:28.253078 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:28.266270 25319 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:08:28.341708 25319 raft_consensus.cc:606] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:28.342054 25319 raft_consensus.cc:614] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:08:28.250237 27443 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:28.267303 25942 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:08:28.268837 25941 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:08:28.254238 27607 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:28.270196 25940 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:08:28.348624 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:28.318585 27533 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:28.354465 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:28.271957 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:28.355149 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:28.091876 25242 raft_consensus.cc:1275] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:08:28.274960 25212 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:28.275437 27587 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:28.301015 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 210778us
W20260709 13:08:28.311939 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 143940us
I20260709 13:08:28.524260 25285 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:28.318137 25390 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
W20260709 13:08:28.103327 27637 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.509s	user 0.000s	sys 0.001s
I20260709 13:08:28.258927 27305 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:28.324137 25245 pending_rounds.cc:85] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 3
I20260709 13:08:28.530011 25245 pending_rounds.cc:107] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 13:08:28.530130 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:28.124300 27447 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:28.327891 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:28.532562 25239 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:28.329231 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:28.333827 25284 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:28.533447 25317 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
W20260709 13:08:28.533550 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 199819us
I20260709 13:08:28.280580 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:28.335872 27676 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:28.311640 27615 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:28.281153 27670 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:28.343669 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 305) took 1436 ms. Trace:
I20260709 13:08:28.345155 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:28.317781 27630 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:28.548715 25939 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:28.555681 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:28.556489 25939 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:28.257891 27643 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:28.557382 27643 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:28.348642 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:28.350663 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:28.562357 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:28.562950 27667 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:28.563186 25315 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:28.563588 27667 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:28.564165 27667 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:28.564455 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:28.318015 27592 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:28.565546 27592 raft_consensus.cc:697] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:28.566880 27447 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:28.566815 27592 consensus_queue.cc:237] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:28.260363 27318 raft_consensus.cc:697] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:28.529740 27587 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:28.536756 27305 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:28.569084 27318 consensus_queue.cc:237] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:28.571754 27305 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:28.441825 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:28.572715 25238 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:28.574235 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:28.575268 25238 raft_consensus.cc:2393] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:28.583587 25359 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:28.588984 25359 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:28.534472 27635 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:28.373798 27533 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:28.572490 27335 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:08:28.543326 27624 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:28.407065 25936 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 7: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:28.321022 27602 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:28.348804 27443 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:28.405555 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:28.544102 27669 raft_consensus.cc:2764] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 13:08:28.394465 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:28.341957 27652 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:28.597693 27635 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:28.379940 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:28.515076 25243 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:28.377601 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:28.633363 25945 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:28.349236 25389 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:28.513489 25942 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:28.356150 25320 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 2: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:28.517532 27679 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:28.355301 27607 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:28.639364 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:28.639199 27679 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:28.639567 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:28.519816 27671 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:28.640509 27607 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:28.640640 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:28.328548 25313 rpcz_store.cc:276] 0709 13:08:26.960773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:27.377998 (+417225us) service_pool.cc:224] Handling call
0709 13:08:28.116136 (+738138us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b540005e954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:28.116303 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":540012928}
I20260709 13:08:28.641420 27679 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:28.642445 25313 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:08:28.530337 25245 raft_consensus.cc:2889] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:08:28.325001 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:28.537492 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:28.533166 25285 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:28.539899 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:28.715957 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 176160us
W20260709 13:08:28.540803 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 265896us
I20260709 13:08:28.540844 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:28.541163 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:28.718520 25239 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:28.718865 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:28.545671 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:28.547395 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:28.549204 25312 rpcz_store.cc:276] 0709 13:08:26.907625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:26.910587 (+  2962us) service_pool.cc:224] Handling call
0709 13:08:26.934343 (+ 23756us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:28.145560 (+1211217us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b540021d7a0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976c29dc0 00007f2976cb41c0 00007f2976cb5231 00007f2976cb1d8f 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c
0709 13:08:28.260424 (+114864us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:28.338934 (+ 78510us) spinlock_profiling.cc:243] Waited 505 ms on lock 0x7b5400276354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:28.343594 (+  4660us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1404400512}
I20260709 13:08:28.554003 27687 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:28.387358 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:28.724530 25943 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:28.594739 25316 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:28.443427 27556 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:28.729414 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:28.733467 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:28.539929 25941 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:08:28.342084 27614 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260709 13:08:28.625936 25314 raft_consensus.cc:2393] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 3: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:08:28.417443 27678 log.cc:927] Time spent T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 13:08:28.601990 27605 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:28.511251 25387 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:28.598006 27447 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:28.576856 27620 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:28.634866 27602 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:28.635641 27443 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:28.712694 25242 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:28.710331 25359 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:28.727000 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 193901us
I20260709 13:08:28.723541 27607 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:28.728817 25312 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:08:28.733242 27687 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:28.733534 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:28.801726 25243 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:28.611467 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:28.736689 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:28.802768 25237 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:28.737612 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1788) took 1243 ms. Trace:
I20260709 13:08:28.658490 27635 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:28.632416 27624 raft_consensus.cc:697] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:28.780611 25239 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:28.780125 25212 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:28.551587 27676 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:28.792459 25911 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:28.803659 25318 rpcz_store.cc:276] 0709 13:08:27.494567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:28.122036 (+627469us) service_pool.cc:224] Handling call
0709 13:08:28.122450 (+   414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:28.122729 (+   279us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:08:28.273978 (+151249us) log.cc:844] Serialized 14 byte log entry
0709 13:08:28.737315 (+463337us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:28.737365 (+    50us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:28.737410 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:28.737465 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:28.737563 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":280054,"prepare.run_cpu_time_us":5624,"prepare.run_wall_time_us":71526,"raft.queue_time_us":84,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":38147,"thread_start_us":604770,"threads_started":2}
W20260709 13:08:28.804811 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 94570.6us
I20260709 13:08:28.805629 27602 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:28.795527 27447 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:28.807351 27623 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:28.795852 27605 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:28.808003 27443 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:28.798321 27607 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:28.814271 25313 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 13:08:28.882743 27700 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:08:28.748575 27441 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20260709 13:08:28.883663 27607 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:28.818938 27687 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:28.878911 27671 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:28.710650 27679 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:28.884454 27671 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:28.865995 27447 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:28.884907 27679 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:28.884518 27687 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 13:08:28.877578 27605 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:28.888548 25317 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:08:28.892056 27615 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:28.884909 27671 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:28.894276 27679 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:28.894531 27676 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:28.846909 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 501828us
I20260709 13:08:28.897621 25209 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:28.898692 25209 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:28.899507 27706 raft_consensus.cc:2764] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:08:28.847637 25943 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:28.853463 25319 raft_consensus.cc:2368] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6.
I20260709 13:08:28.900164 27667 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:28.854342 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 61995.2us
I20260709 13:08:28.901916 27587 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:28.847883 27630 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260709 13:08:28.858839 25315 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 13:08:28.724128 27606 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:28.904515 27606 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:28.908195 27606 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:28.865077 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 139411us
I20260709 13:08:28.741976 27348 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:28.908885 27348 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:28.909373 27348 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:28.909632 25212 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:28.909827 25245 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:28.910847 25212 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:28.911599 27669 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:28.912106 27669 raft_consensus.cc:697] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:28.868446 25359 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:28.912979 27669 consensus_queue.cc:237] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:28.913452 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 45092.2us
I20260709 13:08:28.914069 25359 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:28.797917 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:28.726028 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1243) took 1566 ms. Trace:
I20260709 13:08:28.882625 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:28.780983 27365 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.067s
I20260709 13:08:28.900291 27706 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:08:28.866432 27702 log.cc:927] Time spent T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.146s	user 0.001s	sys 0.000s
I20260709 13:08:28.917026 25209 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:28.873761 27622 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:28.936138 25359 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:28.946280 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:28.876720 27443 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:28.909672 27698 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 13:08:28.946993 25245 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:28.925774 25936 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: []
W20260709 13:08:28.933861 27674 consensus_meta.cc:287] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.849s	user 0.017s	sys 0.000s
I20260709 13:08:28.933323 27630 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:28.975528 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1369) took 1399 ms. Trace:
I20260709 13:08:28.941999 27706 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:28.976042 25240 rpcz_store.cc:276] 0709 13:08:27.576021 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:27.577449 (+  1428us) service_pool.cc:224] Handling call
0709 13:08:27.598870 (+ 21421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:27.628580 (+ 29710us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:27.628610 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:27.655014 (+ 26404us) log.cc:844] Serialized 26 byte log entry
0709 13:08:27.983920 (+328906us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:27.984064 (+   144us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:27.984334 (+   270us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:28.975277 (+990943us) raft_consensus.cc:1685] finished
0709 13:08:28.975306 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:28.975459 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":299716,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":120,"thread_start_us":292861,"threads_started":1,"wal-append.queue_time_us":593483}
I20260709 13:08:28.946115 27670 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:28.929616 27382 raft_consensus.cc:2764] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260709 13:08:28.929404 27595 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:28.936765 27335 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:28.970096 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:28.958933 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 111499us
I20260709 13:08:28.983979 27614 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:28.950389 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:28.953279 27443 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:28.878144 27623 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:28.949653 27622 raft_consensus.cc:697] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 6 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:28.949949 27606 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 13:08:29.001607 25237 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:08:28.985152 27670 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:28.969722 27533 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:29.011549 25245 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:28.993497 27335 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:29.064510 27533 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:29.065292 27533 consensus_queue.cc:1059] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:08:29.066303 27652 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:29.066141 27605 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:29.066881 27652 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:29.069525 27382 raft_consensus.cc:3055] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:08:29.069959 27382 raft_consensus.cc:740] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:29.070792 27382 consensus_queue.cc:260] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:28.985829 27630 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:29.074739 27630 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:29.075239 27630 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:29.075683 27630 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:29.009166 27556 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:28.983165 27674 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:29.077366 27562 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:08:29.078218 25321 raft_consensus.cc:2393] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:29.028973 27447 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:29.078487 25313 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:29.079190 27674 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:29.079413 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 481) took 1004 ms. Trace:
I20260709 13:08:28.991209 27624 consensus_queue.cc:237] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:29.080195 27447 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:29.080016 25937 rpcz_store.cc:276] 0709 13:08:28.075295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:28.077819 (+  2524us) service_pool.cc:224] Handling call
0709 13:08:28.939627 (+861808us) spinlock_profiling.cc:243] Waited 43.7 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:29.079317 (+139690us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":100401024}
I20260709 13:08:29.068970 27652 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:29.077484 27556 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:29.082564 27622 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:29.083595 27556 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:29.084019 27623 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:29.039263 27670 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:29.085760 27630 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:29.101117 25246 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: []
W20260709 13:08:29.105437 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:29.108355 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:29.093174 27335 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:29.003121 27614 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:29.012914 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:28.929221 25388 rpcz_store.cc:276] 0709 13:08:27.149512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:27.300626 (+151114us) service_pool.cc:224] Handling call
0709 13:08:28.628854 (+1328228us) spinlock_profiling.cc:243] Waited 764 ms on lock 0x7b54001e3f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:28.710130 (+ 81276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1757557120}
I20260709 13:08:28.982693 27706 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:29.007303 27318 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:29.014288 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260709 13:08:29.016767 27693 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.137s	user 0.000s	sys 0.000s
I20260709 13:08:29.017863 27667 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:29.029429 27615 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:28.993181 27595 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:29.035689 27348 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:28.987962 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 261898us
I20260709 13:08:29.009692 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:29.090005 27443 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:29.116549 27555 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:29.152468 25358 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:29.156241 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:08:29.164804 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:29.077442 27671 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:29.144330 25389 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:29.136402 27606 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:29.181154 27615 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:29.189904 27643 consensus_meta.cc:287] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.632s	user 0.010s	sys 0.001s
I20260709 13:08:29.157452 27595 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:29.194841 25358 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:29.184659 27443 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:29.196487 27443 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:29.196894 27443 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:29.197271 27443 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:29.197378 27615 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:29.193192 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:29.201785 27630 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:29.195403 27643 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:29.202471 27630 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:29.204378 27630 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:29.201759 25937 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:29.204447 27643 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:29.206072 25358 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:29.183230 27556 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:29.207545 25239 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:29.145910 27607 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:29.182832 27318 raft_consensus.cc:697] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:29.190836 27663 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:29.228400 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:29.229295 25943 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
W20260709 13:08:29.223423 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 276661us
I20260709 13:08:29.211308 27615 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:29.231180 25359 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:29.255107 25390 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:29.193356 27606 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:29.257418 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:29.264918 25394 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 2: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:29.259485 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:29.265662 25393 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:29.192469 27707 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.057s
I20260709 13:08:29.276338 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:29.277346 25392 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:29.283026 25394 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:29.286314 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:29.288421 25395 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260709 13:08:29.290184 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:29.279460 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:29.297413 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:29.222991 27630 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20260709 13:08:29.218792 27556 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:29.225154 27643 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20260709 13:08:29.249392 27676 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
W20260709 13:08:29.316890 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 343516us
I20260709 13:08:29.307911 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:29.403925 25243 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:29.308683 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:29.405560 25245 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:29.298826 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:29.263337 27615 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:29.252041 27652 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 13:08:29.409377 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:29.410082 25242 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:29.316313 27348 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:29.375018 27605 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:29.407574 25909 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 1.948s (SENT) [suppressed 29 similar messages]
I20260709 13:08:29.424778 25393 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:29.426106 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:29.427461 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:29.433790 25392 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:29.436239 25243 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:29.414181 27710 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:29.416975 27649 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.022s	sys 0.015s
I20260709 13:08:29.403793 27713 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:29.459172 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:29.461582 25242 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:29.417382 25940 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:29.473448 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:29.480298 25242 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:29.456234 27667 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.007s
W20260709 13:08:29.460348 27649 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.022s	sys 0.016s
I20260709 13:08:29.466008 27605 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:29.464900 27614 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:29.497874 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:29.511657 25243 raft_consensus.cc:2393] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:08:29.476955 27602 consensus_meta.cc:287] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.611s	user 0.009s	sys 0.001s
I20260709 13:08:29.459699 27587 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:29.464745 25386 raft_consensus.cc:1275] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 13:08:29.493321 25238 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.008s	sys 0.033s
W20260709 13:08:29.516322 25212 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.015s	sys 0.033s
I20260709 13:08:29.485894 27671 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:29.456480 27710 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:29.463418 27562 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 13:08:29.419018 27318 consensus_queue.cc:237] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:29.476141 27713 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:29.833581 25944 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.008s	sys 0.030s
W20260709 13:08:29.873034 25944 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.011s	sys 0.034s
I20260709 13:08:29.873185 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:29.873409 25212 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.882s	user 0.046s	sys 0.050s
W20260709 13:08:29.844060 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 356694us
W20260709 13:08:29.874183 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 883075us
I20260709 13:08:29.842770 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:29.875157 25287 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:29.746073 25360 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.018s	sys 0.041s
W20260709 13:08:29.548238 25284 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.003s	sys 0.028s
W20260709 13:08:29.876261 25360 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.918s	user 0.034s	sys 0.049s
I20260709 13:08:29.781600 25245 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
W20260709 13:08:29.876873 25284 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.874s	user 0.009s	sys 0.039s
I20260709 13:08:29.843694 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:29.878302 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:29.878783 25246 raft_consensus.cc:2410] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260709 13:08:29.814340 25942 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.000s	sys 0.010s
W20260709 13:08:29.879513 25942 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.000s	sys 0.010s
W20260709 13:08:29.876411 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 468842us
I20260709 13:08:29.886297 25242 raft_consensus.cc:2410] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:08:29.890064 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:29.890805 25245 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:29.841648 27715 log.cc:927] Time spent T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.428s	user 0.000s	sys 0.001s
W20260709 13:08:29.738025 27712 log.cc:927] Time spent T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.486s	user 0.000s	sys 0.001s
W20260709 13:08:29.887099 25358 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.011s	sys 0.026s
I20260709 13:08:29.493871 27669 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.392s
I20260709 13:08:29.492961 27687 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.425s
I20260709 13:08:29.894400 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:29.511034 27605 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:29.906826 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:29.517424 27720 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:08:29.886503 25361 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.017s	sys 0.031s
W20260709 13:08:30.192116 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 317003us
W20260709 13:08:30.217332 25358 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.006s	user 0.028s	sys 0.039s
I20260709 13:08:29.994125 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:30.217779 27739 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260709 13:08:30.077724 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:29.511649 27642 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:29.516772 27679 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:08:30.218564 27709 log.cc:927] Time spent T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 1.112s	user 0.015s	sys 0.000s
I20260709 13:08:30.222317 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 316) took 1960 ms. Trace:
I20260709 13:08:29.735493 27607 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:29.843694 25209 connection.cc:256] RPC call timeout handler was delayed by -1.5261150030000863s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21190000000000003s for a response
I20260709 13:08:30.245116 25320 rpcz_store.cc:276] 0709 13:08:28.262018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:28.640412 (+378394us) service_pool.cc:224] Handling call
0709 13:08:30.222121 (+1581709us) spinlock_profiling.cc:243] Waited 854 ms on lock 0x7b5400087ad4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:30.222257 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1963487104}
W20260709 13:08:30.240064 25361 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.939s	user 0.019s	sys 0.040s
I20260709 13:08:29.990290 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:30.243026 25287 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:30.240690 25243 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 7: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:08:30.240212 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 1.03416e+06us
W20260709 13:08:30.251335 25209 leader_election.cc:336] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Timed out: RequestConsensusVote RPC to 127.23.140.68:39211 timed out after 1.701s (SENT)
I20260709 13:08:30.251755 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:30.244457 25237 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:30.252202 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:30.205389 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:30.089089 27733 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 13:08:30.226919 25359 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:30.256526 25359 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:30.198724 27710 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:30.083482 25245 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:30.209491 27382 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:08:29.880795 27592 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:30.201537 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 624) took 2834 ms. Trace:
I20260709 13:08:30.254125 25246 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
W20260709 13:08:30.182978 27670 consensus_meta.cc:287] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.097s	user 0.010s	sys 0.003s
I20260709 13:08:29.886680 27620 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:30.255357 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:08:29.881855 27447 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:30.030571 27716 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.008s	sys 0.022s
I20260709 13:08:29.882550 27606 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:30.295840 27441 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 13:08:30.023933 27717 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.212s	user 0.000s	sys 0.001s
I20260709 13:08:30.296725 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:30.297549 25941 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
W20260709 13:08:30.297919 27718 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.482s	user 0.003s	sys 0.003s
I20260709 13:08:30.299572 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:30.301323 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1390) took 1014 ms. Trace:
I20260709 13:08:30.301735 25244 rpcz_store.cc:276] 0709 13:08:29.286947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:29.287074 (+   127us) service_pool.cc:224] Handling call
0709 13:08:29.304576 (+ 17502us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:29.304882 (+   306us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:29.304919 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:29.315105 (+ 10186us) log.cc:844] Serialized 26 byte log entry
0709 13:08:29.875995 (+560890us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:08:29.876048 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:29.876141 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:30.301101 (+424960us) raft_consensus.cc:1685] finished
0709 13:08:30.301121 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:30.301262 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":461790,"threads_started":1,"wal-append.queue_time_us":705966}
I20260709 13:08:30.303674 25244 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:30.304296 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 1896) took 1503 ms. Trace:
I20260709 13:08:30.304597 25390 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:08:30.305043 25392 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:08:30.305815 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:30.307945 27711 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 1.052s	user 0.001s	sys 0.001s
I20260709 13:08:30.264689 27720 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:30.310511 27556 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:30.316555 25940 consensus_meta.cc:287] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.829s	user 0.006s	sys 0.000s
I20260709 13:08:30.289685 27590 consensus_queue.cc:1059] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20260709 13:08:30.317651 25244 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:30.318113 27590 raft_consensus.cc:3055] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:30.318542 27590 raft_consensus.cc:740] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:30.319110 25244 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:08:30.319437 27590 consensus_queue.cc:260] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:30.320120 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:30.321316 27590 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:30.322072 27635 consensus_meta.cc:287] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.447s	user 0.011s	sys 0.000s
I20260709 13:08:30.322520 27635 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:30.322974 27741 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.152s	user 0.002s	sys 0.000s
I20260709 13:08:30.324066 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 347) took 1412 ms. Trace:
I20260709 13:08:30.324615 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1391) took 1037 ms. Trace:
I20260709 13:08:30.324473 25389 rpcz_store.cc:276] 0709 13:08:28.911667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:28.911792 (+   125us) service_pool.cc:224] Handling call
0709 13:08:28.912108 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:29.187015 (+274907us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:30.323833 (+1136818us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54001211d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:30.324007 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":447763200}
I20260709 13:08:30.260905 27734 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:30.325371 25241 rpcz_store.cc:276] 0709 13:08:29.287304 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:29.287803 (+   499us) service_pool.cc:224] Handling call
0709 13:08:29.309988 (+ 22185us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:29.310397 (+   409us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:29.310437 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:29.376294 (+ 65857us) log.cc:844] Serialized 26 byte log entry
0709 13:08:29.792548 (+416254us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:08:29.792611 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:29.792717 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:30.322076 (+529359us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:30.324264 (+  2188us) raft_consensus.cc:1685] finished
0709 13:08:30.324293 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:30.324541 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":580,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":60219,"spinlock_wait_cycles":27615104,"thread_start_us":359014,"threads_started":1,"wal-append.queue_time_us":599917}
W20260709 13:08:30.328542 27706 consensus_meta.cc:287] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.152s	user 0.006s	sys 0.004s
I20260709 13:08:30.258494 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:30.233332 27605 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:30.259244 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:30.304656 25318 rpcz_store.cc:276] 0709 13:08:28.803034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:28.865779 (+ 62745us) service_pool.cc:224] Handling call
0709 13:08:28.914683 (+ 48904us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:28.915006 (+   323us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:28.915033 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:28.915485 (+   452us) log.cc:844] Serialized 26 byte log entry
0709 13:08:29.257405 (+341920us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:29.285638 (+ 28233us) log.cc:844] Serialized 14 byte log entry
0709 13:08:29.307127 (+ 21489us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:29.307236 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:30.297318 (+990082us) spinlock_profiling.cc:243] Waited 9.15 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:30.304031 (+  6713us) raft_consensus.cc:1685] finished
0709 13:08:30.304117 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:30.304243 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":97657,"prepare.queue_time_us":15637,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":25003,"raft.run_cpu_time_us":11444,"raft.run_wall_time_us":101436,"spinlock_wait_cycles":21049088,"thread_start_us":341519,"threads_started":1,"wal-append.queue_time_us":790692}
I20260709 13:08:30.331073 25242 raft_consensus.cc:2393] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:30.331547 27670 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:30.333773 27670 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:30.260696 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1387) took 1254 ms. Trace:
I20260709 13:08:30.336125 25239 rpcz_store.cc:276] 0709 13:08:29.006235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:29.006345 (+   110us) service_pool.cc:224] Handling call
0709 13:08:29.006590 (+   245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:29.212924 (+206334us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:30.260490 (+1047566us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b54003eb554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:30.260644 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":461634816}
I20260709 13:08:29.810168 27725 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.439s
I20260709 13:08:30.338524 27725 raft_consensus.cc:3055] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:08:30.338863 27725 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:30.339706 27725 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:30.340819 27725 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:30.343473 27623 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:08:30.343600 27663 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:08:30.320855 25388 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:30.262573 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:30.345513 27623 consensus_queue.cc:1059] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:30.346298 27607 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:30.346356 25239 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:08:30.260380 25287 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:29.549458 27587 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:30.346907 27607 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:30.347396 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 87099.4us
I20260709 13:08:30.244786 27562 raft_consensus.cc:3055] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:30.348842 27607 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:30.349300 27562 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:30.350140 27562 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:30.351320 27562 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:08:30.353350 27724 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.539s	user 0.008s	sys 0.001s
I20260709 13:08:30.355208 25285 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:30.355901 25314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 1892) took 1765 ms. Trace:
I20260709 13:08:30.203989 27615 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:30.294415 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:30.294970 27447 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:30.357961 25243 raft_consensus.cc:2393] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:30.358767 27670 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:30.359302 27447 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:30.295419 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:30.295709 27716 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.873s	user 0.010s	sys 0.033s
I20260709 13:08:30.328958 27706 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:30.324661 27635 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:30.353451 27719 log.cc:927] Time spent T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.519s	user 0.002s	sys 0.005s
I20260709 13:08:30.348884 27587 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:30.356289 25314 rpcz_store.cc:276] 0709 13:08:28.590963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:28.796067 (+205104us) service_pool.cc:224] Handling call
0709 13:08:28.915316 (+119249us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:28.915677 (+   361us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:28.915709 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:28.925757 (+ 10048us) log.cc:844] Serialized 26 byte log entry
0709 13:08:29.086279 (+160522us) spinlock_profiling.cc:243] Waited 3.68 ms on lock 0x7b4800271ed8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297bbdbaa1 00007f297bbd2b37 00007f297bbd29ca 00007f297bcca26f 00007f297bcc9f37 00007f297bcde55c 00007f297bcde508 00007f297bcde488 00007f297bcde443 00007f297bcdd6ec 00007f297ac07a96
0709 13:08:29.254294 (+168015us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:29.254332 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:29.254391 (+    59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:30.352617 (+1098226us) spinlock_profiling.cc:243] Waited 9.26 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:30.354263 (+  1646us) raft_consensus.cc:1685] finished
0709 13:08:30.354280 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:30.355737 (+  1457us) spinlock_profiling.cc:243] Waited 68 ms on lock 0x7b54002f35d0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:30.355847 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1265,"raft.run_cpu_time_us":2320,"raft.run_wall_time_us":160632,"spinlock_wait_cycles":186068992,"thread_start_us":276098,"threads_started":1,"wal-append.queue_time_us":591680}
I20260709 13:08:30.025559 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1620) took 1741 ms. Trace:
I20260709 13:08:30.361017 25287 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:30.343449 25237 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:30.362181 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 636) took 1135 ms. Trace:
I20260709 13:08:30.387492 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:30.388870 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:30.245357 27723 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:30.248591 27679 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:30.263226 27669 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:30.394724 25287 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:30.365621 25909 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:30.377663 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:30.364401 27595 consensus_meta.cc:287] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.147s	user 0.004s	sys 0.006s
W20260709 13:08:30.388793 27443 consensus_meta.cc:287] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.191s	user 0.011s	sys 0.000s
I20260709 13:08:30.379554 27592 consensus_queue.cc:1059] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260709 13:08:30.364553 27652 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 13:08:30.369787 27335 consensus_meta.cc:287] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.212s	user 0.007s	sys 0.003s
I20260709 13:08:30.370864 27630 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260709 13:08:30.386662 25942 rpcz_store.cc:276] 0709 13:08:29.227275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:29.230135 (+  2860us) service_pool.cc:224] Handling call
0709 13:08:29.230549 (+   414us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:29.230974 (+   425us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:08:29.879818 (+648844us) log.cc:844] Serialized 14 byte log entry
0709 13:08:30.085386 (+205568us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:30.086014 (+   628us) log.cc:844] Serialized 26 byte log entry
0709 13:08:30.086193 (+   179us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:30.086246 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:30.086332 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:30.361981 (+275649us) raft_consensus.cc:1685] finished
0709 13:08:30.361998 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:30.362130 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":552,"prepare.queue_time_us":856949,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":37,"raft.queue_time_us":24214,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":26448,"thread_start_us":720635,"threads_started":2,"wal-append.queue_time_us":431882}
I20260709 13:08:30.392616 27737 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 13:08:30.391731 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 129243us
I20260709 13:08:30.357911 27713 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:30.293459 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:30.387094 25388 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:30.316946 25940 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:30.359304 27670 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:30.416164 27614 consensus_queue.cc:1059] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:08:30.371227 25937 raft_consensus.cc:2410] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:08:30.423623 25285 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:08:30.424254 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 60619.2us
I20260709 13:08:30.258992 27624 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:30.346407 27623 raft_consensus.cc:3055] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:30.364138 27441 raft_consensus.cc:697] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:30.427774 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:30.447544 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 82006.8us
I20260709 13:08:30.504774 25237 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:30.440279 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 45674.7us
I20260709 13:08:30.510212 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:30.511217 25942 raft_consensus.cc:2368] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5.
I20260709 13:08:30.425464 25240 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:30.388959 27687 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:30.449607 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:30.377274 25936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 490) took 1237 ms. Trace:
W20260709 13:08:30.264885 27738 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 13:08:30.369571 27734 raft_consensus.cc:697] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:08:30.485216 25388 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1249) took 1232 ms (client timeout 1640 ms). Trace:
I20260709 13:08:30.486613 25358 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:30.393786 27679 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:30.492151 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 152100us
W20260709 13:08:30.492383 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 198994us
I20260709 13:08:30.396490 27587 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:30.446605 27620 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:08:30.479796 27615 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:08:30.535969 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 49444.1us
I20260709 13:08:30.480199 27720 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:30.452955 27676 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:30.519415 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:30.524725 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:30.539381 25358 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:30.502681 25940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 638) took 1267 ms. Trace:
I20260709 13:08:30.552760 25940 rpcz_store.cc:276] 0709 13:08:29.227904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:29.247635 (+ 19731us) service_pool.cc:224] Handling call
0709 13:08:30.495401 (+1247766us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:30.552906 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:30.504026 27562 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:30.553465 27679 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:30.554021 27562 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:30.554536 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:30.555568 27679 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:30.555922 27743 log.cc:927] Time spent T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 13:08:30.556208 27562 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:30.532567 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:30.557324 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 1893) took 1937 ms. Trace:
I20260709 13:08:30.557521 25320 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 7: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:30.558571 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:30.559283 25320 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:30.557720 25312 rpcz_store.cc:276] 0709 13:08:28.619719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:28.801731 (+182012us) service_pool.cc:224] Handling call
0709 13:08:28.804798 (+  3067us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:28.856683 (+ 51885us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:08:28.880975 (+ 24292us) log.cc:844] Serialized 14 byte log entry
0709 13:08:29.015651 (+134676us) spinlock_profiling.cc:243] Waited 40.1 ms on lock 0x7b1800063280. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:08:29.494497 (+478846us) spinlock_profiling.cc:243] Waited 35.2 ms on lock 0x7b1800063280. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f297685223a 00007f2976cbca2a
0709 13:08:29.994652 (+500155us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:30.014202 (+ 19550us) log.cc:844] Serialized 26 byte log entry
0709 13:08:30.014381 (+   179us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:30.014428 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:30.224040 (+209612us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b54002f3ad4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:30.224087 (+    47us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:30.557097 (+333010us) raft_consensus.cc:1685] finished
0709 13:08:30.557117 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:30.557269 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":633,"prepare.queue_time_us":28387,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":98704,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":39343,"spinlock_wait_cycles":206967168,"thread_start_us":391829,"threads_started":1,"wal-append.queue_time_us":1411403}
I20260709 13:08:30.560051 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:30.560132 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:30.560834 25312 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:30.562103 27605 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:30.564313 27605 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:30.565579 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:30.534094 27744 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:30.567421 27744 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:30.568200 27728 raft_consensus.cc:3055] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:30.568604 27728 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:30.569742 27744 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:08:30.478935 27443 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:30.533627 25388 rpcz_store.cc:269] 0709 13:08:29.252911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:29.268394 (+ 15483us) service_pool.cc:224] Handling call
0709 13:08:30.319988 (+1051594us) spinlock_profiling.cc:243] Waited 9.44 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8deee 0000000000375939 00007f297bc011e0 00007f297bcca0c4 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:30.485120 (+165132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21702912}
I20260709 13:08:30.534439 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:30.519017 27721 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:30.526120 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:30.529524 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 105964us
I20260709 13:08:30.585011 25315 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:30.450228 27595 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:30.536406 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:30.597198 25943 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:30.535825 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:30.598064 25390 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:30.541422 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:30.423082 27723 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:30.541944 27556 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:30.542346 25393 raft_consensus.cc:1275] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:08:30.539744 27624 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:30.542410 27447 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:08:30.542620 27734 consensus_queue.cc:237] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:30.543731 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:30.544299 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:30.540077 27365 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:08:30.536150 27592 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260709 13:08:30.545480 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:30.546118 27676 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:30.547005 25358 leader_election.cc:400] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6.
I20260709 13:08:30.546744 27720 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:30.547320 27670 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:30.547395 27607 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:30.548038 27722 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:08:30.598598 27533 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election won for term 4
W20260709 13:08:30.550778 25240 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32994 (request call id 1630) took 1269 ms (client timeout 1558 ms). Trace:
W20260709 13:08:30.675455 25240 rpcz_store.cc:269] 0709 13:08:29.280959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:29.282275 (+  1316us) service_pool.cc:224] Handling call
0709 13:08:30.550581 (+1268306us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400457254. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:30.550722 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2612264704}
I20260709 13:08:30.550812 25944 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 637) took 1323 ms. Trace:
I20260709 13:08:30.289867 25938 rpcz_store.cc:276] 0709 13:08:27.371145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:27.375001 (+  3856us) service_pool.cc:224] Handling call
0709 13:08:27.375360 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:28.249746 (+874386us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b18004be980. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b932a6 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9
0709 13:08:28.578764 (+329018us) spinlock_profiling.cc:243] Waited 4.37 ms on lock 0x7b18004be980. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc2455 00007f2976caa060 00007f2976caabfb 00007f2976cb564b 00007f2976cb1d8f
0709 13:08:28.873532 (+294768us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:28.873581 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:28.985464 (+111883us) log.cc:844] Serialized 26 byte log entry
0709 13:08:29.494642 (+509178us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:29.494727 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:29.515783 (+ 21056us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:30.201238 (+685455us) raft_consensus.cc:1685] finished
0709 13:08:30.201271 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:30.201443 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33974,"raft.queue_time_us":56241,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":8075,"spinlock_wait_cycles":36515968,"thread_start_us":727067,"threads_started":2,"wal-append.queue_time_us":1194093}
I20260709 13:08:30.501528 27623 raft_consensus.cc:740] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:08:30.573398 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:30.569584 27728 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:30.534901 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:30.487684 27335 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:30.583592 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:30.513090 25908 leader_election.cc:336] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.562s (SENT)
I20260709 13:08:30.518052 27643 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:30.586076 25945 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 13:08:30.589805 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 55484.1us
W20260709 13:08:30.802894 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 217604us
I20260709 13:08:30.591799 25237 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:30.595080 25388 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:08:30.604898 27721 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:30.806713 25388 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 13:08:30.541818 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:30.864661 25317 raft_consensus.cc:2410] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:08:30.610392 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:30.610482 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:30.653455 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:30.655083 25358 leader_election.cc:403] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20260709 13:08:30.549103 25129 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet eda233ace2a24b379fef320a5f610c7a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 46de6919f85341c6a9b574dff112e35a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:08:30.683790 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:30.676882 25944 rpcz_store.cc:276] 0709 13:08:29.227585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:29.231216 (+  3631us) service_pool.cc:224] Handling call
0709 13:08:29.254445 (+ 23229us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:29.256672 (+  2227us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:29.256709 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:29.272122 (+ 15413us) log.cc:844] Serialized 26 byte log entry
0709 13:08:29.873600 (+601478us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:30.091683 (+218083us) log.cc:844] Serialized 14 byte log entry
0709 13:08:30.092702 (+  1019us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:30.092800 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:30.550603 (+457803us) raft_consensus.cc:1685] finished
0709 13:08:30.550620 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:30.550738 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17783,"prepare.queue_time_us":214566,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":16724,"raft.run_cpu_time_us":327,"raft.run_wall_time_us":45359,"thread_start_us":652304,"threads_started":2,"wal-append.queue_time_us":695131}
I20260709 13:08:30.648308 27624 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:30.573518 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:30.708247 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:30.867664 25129 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 318763us
I20260709 13:08:30.870445 25392 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:30.749405 25287 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:30.704463 27441 consensus_queue.cc:237] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:30.763370 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:30.652617 25313 raft_consensus.cc:2376] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4. Re-sending same reply.
I20260709 13:08:30.662556 27706 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:30.710062 25318 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:30.796758 25387 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:30.915422 25315 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:30.748672 25908 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:30.522269 27622 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:30.959653 27622 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:30.961995 27635 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:08:30.962608 27622 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:30.803366 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:30.598768 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:30.964020 25394 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:30.831646 27623 consensus_queue.cc:260] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:30.964181 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 160869us
I20260709 13:08:30.964923 27623 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:30.965562 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:30.800505 25143 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39582 (request call id 306) took 7863 ms. Trace:
I20260709 13:08:30.752851 27533 raft_consensus.cc:697] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:30.773945 27620 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:30.848129 25284 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:30.821182 25386 raft_consensus.cc:2410] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:08:30.773932 27687 raft_consensus.cc:697] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:30.656114 27556 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:30.825479 27728 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:30.846603 25140 catalog_manager.cc:5697] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 2 to 3, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 3 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:08:30.871901 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 122581us
W20260709 13:08:31.095975 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:30.657270 27676 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:30.870981 25388 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 13:08:30.895762 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:30.898502 27750 log.cc:927] Time spent T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
W20260709 13:08:30.899308 27749 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 13:08:30.904062 25392 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:30.911355 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:30.833362 27643 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:30.393574 25238 rpcz_store.cc:276] 0709 13:08:28.274290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:28.575983 (+301693us) service_pool.cc:224] Handling call
0709 13:08:28.576416 (+   433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:28.576723 (+   307us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:08:28.609736 (+ 33013us) log.cc:844] Serialized 14 byte log entry
0709 13:08:29.874477 (+1264741us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:29.886239 (+ 11762us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:29.892531 (+  6292us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:29.991498 (+ 98967us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:30.015383 (+ 23885us) inbound_call.cc:177] Queueing success response
0709 13:08:30.090253 (+ 74870us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
Metrics: {"mutex_wait_us":5408,"prepare.queue_time_us":14931,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":35441,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":43184,"spinlock_wait_cycles":3617024,"thread_start_us":1227477,"threads_started":2,"wal-append.queue_time_us":493936}
W20260709 13:08:30.919467 25244 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.051s	sys 0.057s
I20260709 13:08:30.871708 27624 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:30.938494 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:30.942741 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:31.115955 25318 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:30.954339 27630 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:30.955705 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:30.958632 27713 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:30.966785 25143 rpcz_store.cc:276] 0709 13:08:22.937220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:22.937408 (+   188us) service_pool.cc:224] Handling call
0709 13:08:30.800423 (+7863015us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:08:27.334789 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:08:27.357854 (+ 23065us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:08:27.357903 (+    49us) write_op.cc:435] Acquired schema lock
0709 13:08:27.357916 (+    13us) tablet.cc:662] Decoding operations
0709 13:08:27.358826 (+   910us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:08:27.358897 (+    71us) write_op.cc:641] Partition lock acquired for write op
0709 13:08:27.358923 (+    26us) tablet.cc:685] Acquiring locks for 6 operations
0709 13:08:27.359501 (+   578us) tablet.cc:701] Row locks acquired
0709 13:08:27.359543 (+    42us) write_op.cc:260] PREPARE: finished
0709 13:08:27.359703 (+   160us) write_op.cc:270] Start()
0709 13:08:27.359841 (+   138us) write_op.cc:276] Timestamp: P: 1783602507359643 usec, L: 0
0709 13:08:27.359857 (+    16us) op_driver.cc:348] REPLICATION: starting
0709 13:08:27.449138 (+ 89281us) log.cc:844] Serialized 2535 byte log entry
0709 13:08:28.522131 (+1072993us) op_driver.cc:464] REPLICATION: finished
0709 13:08:29.103956 (+581825us) spinlock_profiling.cc:243] Waited 27.8 ms on lock 0x7b5400470ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976caf32a 00007f2976c34f14 00007f2976c34ec3 00007f2976c34e38 00007f2976c34df3 00007f2976c340dc 00007f2976c5de16 00007f2976c45cbb 00007f2976c3039d 00007f2976c33d75
0709 13:08:29.205449 (+101493us) write_op.cc:301] APPLY: starting
0709 13:08:29.216954 (+ 11505us) tablet.cc:1367] starting BulkCheckPresence
0709 13:08:29.264234 (+ 47280us) tablet.cc:1370] finished BulkCheckPresence
0709 13:08:29.278847 (+ 14613us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:08:29.806628 (+527781us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:08:29.859323 (+ 52695us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0709 13:08:29.869961 (+ 10638us) write_op.cc:312] APPLY: finished
0709 13:08:29.995854 (+125893us) log.cc:844] Serialized 53 byte log entry
0709 13:08:30.499738 (+503884us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:30.499861 (+   123us) write_op.cc:454] Released schema lock
0709 13:08:30.499905 (+    44us) write_op.cc:341] FINISH: Updating metrics
0709 13:08:30.584112 (+ 84207us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:30.597182 (+ 13070us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":260163,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":621901,"apply.run_cpu_time_us":92684,"apply.run_wall_time_us":1344398,"num_ops":6,"prepare.queue_time_us":746136,"prepare.run_cpu_time_us":28177,"prepare.run_wall_time_us":608404,"raft.queue_time_us":271269,"raft.run_cpu_time_us":55640,"raft.run_wall_time_us":823166,"replication_time_us":989549,"spinlock_wait_cycles":63909504,"thread_start_us":2008625,"threads_started":5,"wal-append.queue_time_us":1175073,"wal-append.run_cpu_time_us":6393,"wal-append.run_wall_time_us":1126586}]]}
I20260709 13:08:30.877506 27562 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:30.993919 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:31.010823 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 162828us
I20260709 13:08:31.149780 25237 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:31.107329 27722 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6.
I20260709 13:08:31.038410 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:31.155682 25243 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:30.914662 27763 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:31.096812 25287 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:31.097359 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1250) took 1844 ms. Trace:
W20260709 13:08:30.873073 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 326167us
I20260709 13:08:31.101811 25395 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 13:08:31.157428 25387 rpcz_store.cc:276] 0709 13:08:29.253318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:29.253463 (+   145us) service_pool.cc:224] Handling call
0709 13:08:29.273649 (+ 20186us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:30.937826 (+1664177us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:31.097104 (+159278us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b5400282dd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:31.097298 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3502144896}
I20260709 13:08:30.954344 27716 raft_consensus.cc:3055] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:31.101598 25140 catalog_manager.cc:5697] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:08:31.159135 27562 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:31.159288 25140 catalog_manager.cc:5697] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 1 to 5, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 5 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:08:31.162559 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 65817.1us
I20260709 13:08:31.162246 25140 catalog_manager.cc:5697] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 1 to 5, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 5 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:08:31.111559 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:31.115187 25244 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.052s	sys 0.057s
I20260709 13:08:31.115852 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:31.163962 25240 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:30.947113 27652 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:30.518168 25936 rpcz_store.cc:276] 0709 13:08:29.139771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:29.147850 (+  8079us) service_pool.cc:224] Handling call
0709 13:08:29.221280 (+ 73430us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:29.303277 (+ 81997us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:29.306050 (+  2773us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:29.479131 (+173081us) log.cc:844] Serialized 26 byte log entry
0709 13:08:30.086584 (+607453us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:30.086637 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:30.086727 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:30.361537 (+274810us) raft_consensus.cc:1685] finished
0709 13:08:30.364405 (+  2868us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:30.377191 (+ 12786us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":35314,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":6977,"thread_start_us":377196,"threads_started":1,"wal-append.queue_time_us":779713}
W20260709 13:08:31.116652 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 313112us
I20260709 13:08:31.121874 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:31.124043 27335 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:31.098351 27447 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:31.173337 27773 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260709 13:08:31.173731 27447 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:31.175168 27773 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:08:31.175736 27447 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:31.123731 27630 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
W20260709 13:08:31.176266 27382 consensus_meta.cc:287] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.900s	user 0.007s	sys 0.000s
I20260709 13:08:31.176463 27773 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:08:31.177057 27533 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:31.177735 27630 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:31.166610 25140 catalog_manager.cc:5697] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:08:31.125468 27753 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.190s	user 0.001s	sys 0.000s
I20260709 13:08:31.126135 27556 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:31.037746 27587 raft_consensus.cc:697] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:31.138540 27595 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:31.111505 27443 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:31.130698 27676 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:31.180806 27620 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:31.181007 27587 consensus_queue.cc:237] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:31.182649 27676 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:31.136524 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:31.184420 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:31.143528 27765 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:31.143697 27774 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:31.144263 27777 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:31.147279 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:30.942764 27756 log.cc:927] Time spent T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
W20260709 13:08:31.138832 27674 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.042s	sys 0.056s
I20260709 13:08:31.151342 25284 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:31.142860 27642 consensus_meta.cc:287] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.890s	user 0.011s	sys 0.000s
W20260709 13:08:31.233276 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 81999.8us
I20260709 13:08:31.234731 25284 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:31.170145 27751 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.309s	user 0.000s	sys 0.001s
I20260709 13:08:31.160719 27716 raft_consensus.cc:740] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:08:31.171029 27752 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.370s	user 0.001s	sys 0.000s
I20260709 13:08:31.178323 27630 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:31.178858 27773 consensus_queue.cc:1048] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 13:08:31.180073 27725 consensus_meta.cc:287] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.822s	user 0.006s	sys 0.001s
W20260709 13:08:30.958892 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 445819us
I20260709 13:08:31.187932 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:31.189625 25240 raft_consensus.cc:2410] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:08:31.202435 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:31.215754 25315 raft_consensus.cc:2393] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 6: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:31.201427 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:31.186136 25140 catalog_manager.cc:5697] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 2 to 4, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 4 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:31.192353 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:31.226403 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 59948.8us
I20260709 13:08:31.154760 27723 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:31.202459 27762 log.cc:927] Time spent T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:08:31.169858 27764 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:31.236713 25284 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:31.240723 25392 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:08:31.246901 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:31.248482 25243 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:31.248826 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:31.185444 27720 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:31.312836 25359 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:31.308457 25394 raft_consensus.cc:2376] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6. Re-sending same reply.
I20260709 13:08:31.126605 27687 consensus_queue.cc:237] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:31.304632 25242 raft_consensus.cc:2410] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 13:08:31.309458 25393 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:31.302066 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:31.316478 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:31.232520 27765 consensus_queue.cc:1059] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.122s
I20260709 13:08:31.201511 27734 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260709 13:08:31.275127 25941 raft_consensus.cc:2393] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:31.187952 27652 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:31.166167 27763 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:31.352186 27772 log.cc:927] Time spent T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.147s	user 0.000s	sys 0.001s
I20260709 13:08:31.299369 27630 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:31.245800 27721 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:31.352864 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:31.256062 27642 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:08:31.362490 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 396995us
I20260709 13:08:31.415094 25209 leader_election.cc:400] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5.
I20260709 13:08:31.415485 25209 leader_election.cc:403] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 13:08:31.416370 25209 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:31.322121 27725 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:31.205330 27774 raft_consensus.cc:697] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:08:31.227025 27674 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.043s	sys 0.057s
W20260709 13:08:31.367973 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 51583.1us
I20260709 13:08:31.366896 25317 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:31.347590 27335 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 13:08:31.386850 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 220236us
I20260709 13:08:31.391990 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1401) took 1153 ms. Trace:
I20260709 13:08:31.364375 27706 raft_consensus.cc:1064] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:08:31.255194 27768 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.136s
W20260709 13:08:31.378636 27781 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 13:08:31.411031 27630 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:31.433467 25316 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:31.416916 25908 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:31.418884 27382 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:31.362058 27764 consensus_queue.cc:1059] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260709 13:08:31.433679 25285 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:31.396536 27533 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:31.387576 27766 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:08:31.402233 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:31.436113 27713 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:31.424953 27788 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:31.439603 25358 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:31.483980 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 50382us
W20260709 13:08:31.428171 27590 consensus_meta.cc:287] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.106s	user 0.008s	sys 0.003s
I20260709 13:08:31.415109 27622 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:31.424865 25245 rpcz_store.cc:276] 0709 13:08:30.239034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:30.330434 (+ 91400us) service_pool.cc:224] Handling call
0709 13:08:30.330727 (+   293us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:30.331011 (+   284us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:30.331046 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:30.331429 (+   383us) log.cc:844] Serialized 26 byte log entry
0709 13:08:30.515061 (+183632us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:30.515103 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:30.515709 (+   606us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:31.391758 (+876049us) raft_consensus.cc:1685] finished
0709 13:08:31.391780 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:31.391925 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1070,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":74,"thread_start_us":183252,"threads_started":1,"wal-append.queue_time_us":342551}
I20260709 13:08:31.421047 27441 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:31.423360 27765 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:08:31.470479 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:31.427996 27669 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:31.423840 27562 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:31.445887 25316 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:08:31.383785 27716 consensus_queue.cc:260] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:31.483060 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:31.434571 27670 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:31.437548 25939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 634) took 2211 ms. Trace:
W20260709 13:08:31.486954 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 250312us
I20260709 13:08:31.543653 25284 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:31.487829 27723 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:31.487977 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:31.556875 25944 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:31.446998 27643 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:31.483655 27676 consensus_queue.cc:1059] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.048s
I20260709 13:08:31.437991 27798 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:31.436309 27335 raft_consensus.cc:697] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:08:31.513346 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:31.515678 27784 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5.
I20260709 13:08:31.601400 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:31.602337 25942 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 7: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:31.480376 27713 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:31.475675 27620 raft_consensus.cc:3055] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:31.527442 25319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 325) took 1001 ms. Trace:
I20260709 13:08:31.480005 27766 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:31.475451 27777 raft_consensus.cc:697] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:31.565514 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:31.541564 25285 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:31.450563 27759 consensus_queue.cc:1059] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.083s
I20260709 13:08:31.423580 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1400) took 1251 ms. Trace:
I20260709 13:08:31.463958 27382 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:31.436228 27674 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:08:31.626838 27757 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.266s	user 0.000s	sys 0.001s
I20260709 13:08:31.583457 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:31.627589 27789 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:31.628149 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:31.628777 27804 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:31.629297 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:31.629863 25317 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:31.630568 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:31.631085 25317 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:31.631749 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 360) took 1266 ms. Trace:
I20260709 13:08:31.632081 25391 rpcz_store.cc:276] 0709 13:08:30.365740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:30.366495 (+   755us) service_pool.cc:224] Handling call
0709 13:08:30.376475 (+  9980us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:08:30.396484 (+ 20009us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:30.396531 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:08:30.451794 (+ 55263us) log.cc:844] Serialized 52 byte log entry
0709 13:08:30.833219 (+381425us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:31.106658 (+273439us) log.cc:844] Serialized 14 byte log entry
0709 13:08:31.108041 (+  1383us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:31.108110 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:31.631547 (+523437us) raft_consensus.cc:1685] finished
0709 13:08:31.631564 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:31.631698 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20694,"prepare.queue_time_us":563299,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":41167,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":8224,"thread_start_us":552369,"threads_started":2,"wal-append.queue_time_us":732820}
W20260709 13:08:31.633740 27786 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
W20260709 13:08:31.634667 27770 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.210s	user 0.000s	sys 0.001s
I20260709 13:08:31.609935 27713 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:31.638800 27706 consensus_queue.cc:237] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:31.555691 27670 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:31.642427 27785 log.cc:927] Time spent T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
I20260709 13:08:31.642880 27670 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:31.644117 27776 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 13:08:31.645049 27777 consensus_queue.cc:237] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:31.526696 27765 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:31.647434 27765 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:31.648976 27652 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:31.424079 27725 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:31.649662 27797 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:31.651831 27725 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:31.476058 27533 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:31.654956 27533 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:31.435997 27774 consensus_queue.cc:237] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:31.606074 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:31.614032 27766 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:31.657472 25943 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:31.601400 27798 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:31.659080 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:31.659776 25316 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:31.659968 27798 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:31.611877 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:31.609783 27620 raft_consensus.cc:740] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:31.662528 27620 consensus_queue.cc:260] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:31.663184 25318 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: [6.5-8.6]
I20260709 13:08:31.664094 27620 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:31.544703 27783 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:31.610098 25319 rpcz_store.cc:276] 0709 13:08:30.526392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:30.526515 (+   123us) service_pool.cc:224] Handling call
0709 13:08:30.526816 (+   301us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:30.673898 (+147082us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:30.673926 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:30.707050 (+ 33124us) log.cc:844] Serialized 26 byte log entry
0709 13:08:30.912181 (+205131us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:30.912224 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:30.912314 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:30.961615 (+ 49301us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b4800271ed8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297bbdbaa1 00007f297bbd2b37 00007f297bbd29ca 00007f297bcca26f 00007f297bcc9f37 00007f297bcde55c 00007f297bcde508 00007f297bcde488 00007f297bcde443 00007f297bcdd6ec 00007f297ac07a96
0709 13:08:31.514186 (+552571us) raft_consensus.cc:1685] finished
0709 13:08:31.514212 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:31.527370 (+ 13158us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":172858,"raft.run_cpu_time_us":1441,"raft.run_wall_time_us":249835,"spinlock_wait_cycles":27282176,"thread_start_us":220473,"threads_started":2,"wal-append.queue_time_us":445450}
I20260709 13:08:31.555843 27562 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20260709 13:08:31.671257 27783 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:31.671794 27783 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:31.671712 27768 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:08:31.672359 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:31.643985 27763 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:31.674752 27763 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:31.675503 27763 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:31.675963 27763 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:31.619204 27778 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.262s	user 0.001s	sys 0.000s
I20260709 13:08:31.672833 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:31.678356 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 321) took 1153 ms. Trace:
I20260709 13:08:31.613952 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:31.519819 27622 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:31.491381 27788 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:31.601079 27793 raft_consensus.cc:3055] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:31.624845 25394 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:31.625906 27335 consensus_queue.cc:237] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:31.543220 25939 rpcz_store.cc:276] 0709 13:08:29.226588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:29.231948 (+  5360us) service_pool.cc:224] Handling call
0709 13:08:30.313250 (+1081302us) spinlock_profiling.cc:243] Waited 8.03 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:30.313515 (+   265us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:30.314013 (+   498us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:08:30.333005 (+ 18992us) log.cc:844] Serialized 14 byte log entry
0709 13:08:30.540961 (+207956us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:30.747661 (+206700us) log.cc:844] Serialized 26 byte log entry
0709 13:08:30.803716 (+ 56055us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:30.813350 (+  9634us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:30.841023 (+ 27673us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:31.433272 (+592249us) raft_consensus.cc:1685] finished
0709 13:08:31.437321 (+  4049us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:31.437484 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34503,"prepare.queue_time_us":19289,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":26117,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":171,"spinlock_wait_cycles":18479360,"thread_start_us":161892,"threads_started":1,"wal-append.queue_time_us":327482}
I20260709 13:08:31.626631 27318 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:31.666342 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:31.538133 25320 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:31.649688 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 329) took 1122 ms. Trace:
I20260709 13:08:31.669848 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:31.660797 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:31.695899 25316 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:31.695138 25312 rpcz_store.cc:276] 0709 13:08:30.527628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:30.561441 (+ 33813us) service_pool.cc:224] Handling call
0709 13:08:30.561740 (+   299us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:30.595709 (+ 33969us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:08:30.656477 (+ 60768us) log.cc:844] Serialized 14 byte log entry
0709 13:08:30.989880 (+333403us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:30.990383 (+   503us) log.cc:844] Serialized 26 byte log entry
0709 13:08:30.993113 (+  2730us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:31.033605 (+ 40492us) log.cc:844] Serialized 14 byte log entry
0709 13:08:31.033803 (+   198us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:31.033872 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:31.649383 (+615511us) raft_consensus.cc:1685] finished
0709 13:08:31.649424 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:31.649614 (+   190us) inbound_call.cc:177] Queueing success response
0709 13:08:31.657005 (+  7391us) spinlock_profiling.cc:243] Waited 2.64 ms on lock 0x7b1800403c60. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
Metrics: {"mutex_wait_us":36431,"prepare.queue_time_us":107151,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":8480,"raft.queue_time_us":558115,"raft.run_cpu_time_us":441,"raft.run_wall_time_us":28940,"spinlock_wait_cycles":6061440,"thread_start_us":331849,"threads_started":1,"wal-append.queue_time_us":509148}
I20260709 13:08:31.700166 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 410) took 2238 ms. Trace:
I20260709 13:08:31.678776 25321 rpcz_store.cc:276] 0709 13:08:30.525064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:30.525223 (+   159us) service_pool.cc:224] Handling call
0709 13:08:30.542056 (+ 16833us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:30.751795 (+209739us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:30.751840 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:30.752391 (+   551us) log.cc:844] Serialized 26 byte log entry
0709 13:08:30.973550 (+221159us) spinlock_profiling.cc:243] Waited 5.15 ms on lock 0x7b4800271ed8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297bbdbaa1 00007f297bbd2b37 00007f297bbd29ca 00007f297bcca26f 00007f297bcc9f37 00007f297bcde55c 00007f297bcde508 00007f297bcde488 00007f297bcde443 00007f297bcdd6ec 00007f297ac07a96
0709 13:08:31.011263 (+ 37713us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:31.432203 (+420940us) log.cc:844] Serialized 14 byte log entry
0709 13:08:31.460681 (+ 28478us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:31.467801 (+  7120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:31.678008 (+210207us) raft_consensus.cc:1685] finished
0709 13:08:31.678128 (+   120us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:31.678295 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71998,"prepare.queue_time_us":103773,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":370902,"raft.run_cpu_time_us":1849,"raft.run_wall_time_us":200552,"spinlock_wait_cycles":11853696,"thread_start_us":652383,"threads_started":3,"wal-append.queue_time_us":392929}
I20260709 13:08:31.666276 25936 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:31.608336 27676 raft_consensus.cc:3055] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:31.717669 27676 raft_consensus.cc:740] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:31.718554 27676 consensus_queue.cc:260] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:31.719682 27676 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:31.679013 27624 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:31.689410 27793 raft_consensus.cc:740] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:31.679302 25393 raft_consensus.cc:2376] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5. Re-sending same reply.
I20260709 13:08:31.664944 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:31.680750 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 245986us
I20260709 13:08:31.690928 25943 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:31.710384 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:31.710618 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:31.731252 25394 raft_consensus.cc:2393] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 7: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:31.730983 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:31.732347 25392 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:31.732903 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:31.733837 25393 raft_consensus.cc:2393] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:08:31.671028 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:31.725683 27624 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:31.757503 25314 raft_consensus.cc:2368] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 4. Current term is 5.
I20260709 13:08:31.691751 27769 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:31.774013 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:31.775058 25318 raft_consensus.cc:2410] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:08:31.770347 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:31.776911 25393 raft_consensus.cc:2393] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:31.628153 27789 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:31.770953 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:31.780984 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:31.777931 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:31.629319 27804 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:31.702373 27318 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:32.300223 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1809) took 1452 ms. Trace:
I20260709 13:08:31.781702 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:32.068969 25209 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.037s	sys 0.037s
W20260709 13:08:32.313658 25209 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.887s	user 0.045s	sys 0.051s
I20260709 13:08:32.033031 27793 consensus_queue.cc:260] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:31.923022 27806 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:32.314572 27793 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:32.314472 27806 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:32.075121 27786 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
W20260709 13:08:32.241153 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 483706us
I20260709 13:08:32.139889 27763 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:31.848317 27807 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:32.015046 27766 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:32.203344 25316 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:31.864377 25392 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:32.014822 27622 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:32.319536 25242 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.004s	sys 0.016s
W20260709 13:08:32.320322 25242 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.005s	sys 0.017s
W20260709 13:08:31.887530 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:32.321777 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:32.315750 27318 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:32.300891 25320 rpcz_store.cc:276] 0709 13:08:30.847405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:31.201937 (+354532us) service_pool.cc:224] Handling call
0709 13:08:32.300051 (+1098114us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b5400087fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:32.300171 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":476430080}
I20260709 13:08:32.330559 25394 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:32.033468 25317 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
W20260709 13:08:31.789325 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:32.299755 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 758266us
I20260709 13:08:32.324043 25393 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:32.300082 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:32.336009 25285 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:32.316740 27807 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:31.820794 25358 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:32.351190 27728 consensus_meta.cc:287] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.244s	user 0.018s	sys 0.004s
I20260709 13:08:32.396304 27624 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:32.398155 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:32.400725 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 50693.5us
I20260709 13:08:32.299458 27788 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:32.326331 27777 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.001s	sys 0.015s
W20260709 13:08:32.396899 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:31.822328 27623 consensus_meta.cc:287] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.857s	user 0.012s	sys 0.000s
I20260709 13:08:32.360502 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:32.404085 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:32.405763 25240 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 13:08:32.403985 25911 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.000s	sys 0.029s
I20260709 13:08:31.700770 25246 rpcz_store.cc:276] 0709 13:08:29.461157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:30.340649 (+879492us) service_pool.cc:224] Handling call
0709 13:08:30.340956 (+   307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:30.341303 (+   347us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:08:30.649826 (+308523us) log.cc:844] Serialized 14 byte log entry
0709 13:08:31.582498 (+932672us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:31.669403 (+ 86905us) log.cc:844] Serialized 26 byte log entry
0709 13:08:31.669681 (+   278us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:08:31.669724 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:31.669794 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:31.698716 (+ 28922us) raft_consensus.cc:1685] finished
0709 13:08:31.698952 (+   236us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:31.699828 (+   876us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23219,"prepare.queue_time_us":497010,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":165159,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":35530,"thread_start_us":1147935,"threads_started":3,"wal-append.queue_time_us":699032}
W20260709 13:08:32.397966 27805 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.187s	user 0.000s	sys 0.000s
I20260709 13:08:32.433777 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:32.401916 27713 consensus_meta.cc:287] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.764s	user 0.015s	sys 0.001s
I20260709 13:08:32.415197 25320 raft_consensus.cc:2410] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 13:08:32.439221 25911 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.002s	sys 0.035s
I20260709 13:08:32.435086 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:32.413026 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:32.451906 25386 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1267) took 1911 ms (client timeout 1553 ms). Trace:
W20260709 13:08:32.452168 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 631448us
I20260709 13:08:32.452410 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:32.452769 27788 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:32.453269 25392 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:32.453315 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:32.453357 27788 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:32.454418 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:32.455492 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:32.413985 27630 consensus_meta.cc:287] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.951s	user 0.010s	sys 0.007s
I20260709 13:08:32.455999 27788 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:32.452430 25386 rpcz_store.cc:269] 0709 13:08:30.540643 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:30.554280 (+ 13637us) service_pool.cc:224] Handling call
0709 13:08:32.413220 (+1858940us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b5400482454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:32.451826 (+ 38606us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":284239232}
I20260709 13:08:32.456768 25392 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:32.456753 27630 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:32.459807 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:32.449496 25394 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:32.353360 27318 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:32.478384 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:32.479363 25392 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:32.359084 27687 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:32.396871 27624 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:32.458997 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:32.400148 27806 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:32.483714 25393 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:32.404593 27382 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:32.401948 27774 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.013s	sys 0.013s
I20260709 13:08:32.467890 25320 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260709 13:08:32.411064 27777 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.001s	sys 0.019s
I20260709 13:08:32.485080 25312 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:08:32.478654 25315 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:32.484093 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:32.434504 27794 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:32.484448 25392 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:08:32.479837 27447 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:32.500741 25320 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: [3.2-5.3]
W20260709 13:08:32.490506 27774 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.014s	sys 0.013s
I20260709 13:08:32.486759 25393 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 13:08:32.443994 27728 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:32.468848 27622 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:32.484581 25359 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.786s	user 0.027s	sys 0.081s
W20260709 13:08:32.504946 25359 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.807s	user 0.027s	sys 0.081s
W20260709 13:08:32.505363 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 821602us
I20260709 13:08:32.496352 25313 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
W20260709 13:08:32.514132 25908 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.784s	user 0.038s	sys 0.052s
I20260709 13:08:32.514278 25393 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 13:08:32.522935 25908 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.793s	user 0.038s	sys 0.052s
I20260709 13:08:32.489992 27798 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:32.523355 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 793312us
I20260709 13:08:32.523836 25393 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:32.487007 27687 raft_consensus.cc:697] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:32.471192 27713 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:32.502712 27447 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:32.504310 25386 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 13:08:32.582007 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1276) took 1338 ms. Trace:
I20260709 13:08:32.582698 25388 rpcz_store.cc:276] 0709 13:08:31.243235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:31.243584 (+   349us) service_pool.cc:224] Handling call
0709 13:08:31.243974 (+   390us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:31.244409 (+   435us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:31.244451 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:31.321569 (+ 77118us) log.cc:844] Serialized 26 byte log entry
0709 13:08:31.697052 (+375483us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:08:31.697170 (+   118us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:31.697355 (+   185us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:32.581681 (+884326us) raft_consensus.cc:1685] finished
0709 13:08:32.581715 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:32.581919 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":323,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":52462,"thread_start_us":317881,"threads_started":1,"wal-append.queue_time_us":523128}
I20260709 13:08:32.585736 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:32.586179 25945 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:32.495394 27318 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:32.496028 27794 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:32.540861 27642 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:32.544543 27447 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:32.602787 25241 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:08:32.511889 27728 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:32.631632 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:32.542775 27630 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:32.523196 27622 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:32.589565 27799 log.cc:927] Time spent T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.196s	user 0.000s	sys 0.001s
I20260709 13:08:32.589254 27811 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:32.539570 27798 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:32.642885 27818 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:32.639967 27642 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:32.653358 27766 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:32.653697 27816 log.cc:927] Time spent T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 13:08:32.653599 27818 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:32.655972 25943 raft_consensus.cc:2410] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:08:32.657075 27643 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:32.655967 27818 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:32.657586 27643 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:32.659837 27643 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:32.660043 27676 consensus_meta.cc:287] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.940s	user 0.009s	sys 0.004s
I20260709 13:08:32.660662 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 502) took 1069 ms. Trace:
W20260709 13:08:32.666561 25314 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 317) took 2143 ms (client timeout 1948 ms). Trace:
W20260709 13:08:32.667893 25313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1810) took 1820 ms (client timeout 1858 ms). Trace:
I20260709 13:08:32.670598 27318 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:32.700887 25313 rpcz_store.cc:269] 0709 13:08:30.847702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:30.955322 (+107620us) service_pool.cc:224] Handling call
0709 13:08:32.667660 (+1712338us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54000f8f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:32.667814 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3140268800}
I20260709 13:08:32.707406 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:32.709044 25941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 629) took 2203 ms (client timeout 1999 ms). Trace:
W20260709 13:08:32.709523 25941 rpcz_store.cc:269] 0709 13:08:30.505993 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:30.516677 (+ 10684us) service_pool.cc:224] Handling call
0709 13:08:32.708790 (+2192113us) spinlock_profiling.cc:243] Waited 618 ms on lock 0x7b5400690354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:32.708945 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1421202688}
I20260709 13:08:32.708992 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:32.710778 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:32.711438 25941 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:32.711830 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1804) took 1873 ms. Trace:
I20260709 13:08:32.712473 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:32.713454 25942 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 13:08:32.676664 27789 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:32.714944 27820 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.204s	user 0.001s	sys 0.000s
I20260709 13:08:32.715963 25394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 369) took 1072 ms. Trace:
I20260709 13:08:32.716320 25394 rpcz_store.cc:276] 0709 13:08:31.643234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:32.492934 (+849700us) service_pool.cc:224] Handling call
0709 13:08:32.493398 (+   464us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:32.644512 (+151114us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:32.644542 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:32.644581 (+    39us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:32.644612 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:32.715756 (+ 71144us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54003d4fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:32.715795 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:32.715912 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":347012864}
I20260709 13:08:32.699321 25937 rpcz_store.cc:276] 0709 13:08:31.591053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:31.673715 (+ 82662us) service_pool.cc:224] Handling call
0709 13:08:31.683002 (+  9287us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:31.714554 (+ 31552us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:31.721313 (+  6759us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:31.721902 (+   589us) log.cc:844] Serialized 26 byte log entry
0709 13:08:32.310676 (+588774us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:32.319134 (+  8458us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:32.352002 (+ 32868us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:32.660407 (+308405us) raft_consensus.cc:1685] finished
0709 13:08:32.660431 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:32.660580 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":146600,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":15040,"thread_start_us":470666,"threads_started":1,"wal-append.queue_time_us":780224}
I20260709 13:08:32.718245 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:32.718547 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:32.718887 25938 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:32.719138 25937 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:32.719435 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 495) took 1148 ms. Trace:
I20260709 13:08:32.719781 25938 rpcz_store.cc:276] 0709 13:08:31.571301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:31.584017 (+ 12716us) service_pool.cc:224] Handling call
0709 13:08:32.718083 (+1134066us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:32.719378 (+  1295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2790656}
I20260709 13:08:32.672289 27798 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:32.681651 27804 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:32.682224 27713 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:32.685854 27807 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:32.687901 27624 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:32.689688 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:32.698086 27642 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:32.692075 27794 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:32.692803 27687 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:32.693890 27622 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:32.645988 27811 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:32.694501 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:32.695122 27728 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:32.696331 25128 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f75b60146e7d4b9b9954b4fb2c414679 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 2329e428ebbf484fa3ea9452133df567 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:31.621747 25244 rpcz_store.cc:276] 0709 13:08:30.172647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:30.320545 (+147898us) service_pool.cc:224] Handling call
0709 13:08:30.320772 (+   227us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:30.320953 (+   181us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:08:31.164062 (+843109us) log.cc:844] Serialized 14 byte log entry
0709 13:08:31.423156 (+259094us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:31.423219 (+    63us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:08:31.423287 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:31.423373 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:31.423493 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":431957,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":4082,"raft.queue_time_us":179077,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":15431,"thread_start_us":1058969,"threads_started":2,"wal-append.queue_time_us":421812}
W20260709 13:08:32.749852 25128 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 149270us
W20260709 13:08:32.699445 25314 rpcz_store.cc:269] 0709 13:08:30.523637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:30.523783 (+   146us) service_pool.cc:224] Handling call
0709 13:08:32.666259 (+2142476us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400443c54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:32.666437 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2475407232}
W20260709 13:08:32.642979 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:32.710952 25313 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:08:32.680390 27765 consensus_meta.cc:287] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.032s	user 0.014s	sys 0.005s
I20260709 13:08:32.712174 25315 rpcz_store.cc:276] 0709 13:08:30.838674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:31.307305 (+468631us) service_pool.cc:224] Handling call
0709 13:08:31.307768 (+   463us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:32.498592 (+1190824us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:32.711602 (+213010us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b540035a7d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:32.711778 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2692307712}
I20260709 13:08:32.765736 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:32.766186 25315 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:32.720954 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:32.713071 25941 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:32.740481 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 1874) took 1127 ms. Trace:
I20260709 13:08:32.773090 25246 raft_consensus.cc:2410] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:08:32.744585 25321 raft_consensus.cc:2410] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:08:32.735620 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:32.774062 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:32.772900 25391 rpcz_store.cc:276] 0709 13:08:31.612796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:31.633932 (+ 21136us) service_pool.cc:224] Handling call
0709 13:08:32.689542 (+1055610us) spinlock_profiling.cc:243] Waited 2.78 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:32.740418 (+ 50876us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6393600}
W20260709 13:08:32.761408 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 425780us
I20260709 13:08:32.765893 25911 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:32.751430 27783 consensus_meta.cc:287] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.079s	user 0.021s	sys 0.000s
I20260709 13:08:32.652642 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:08:32.748736 27620 consensus_meta.cc:287] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.084s	user 0.010s	sys 0.003s
I20260709 13:08:32.767654 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:32.778761 27783 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:32.783633 25243 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:32.783900 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:32.784832 25314 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:32.786227 25394 raft_consensus.cc:2393] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:08:32.771345 25212 leader_election.cc:336] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.156s (SENT)
W20260709 13:08:32.795143 25909 leader_election.cc:336] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 1.948s (SENT)
I20260709 13:08:32.806464 25244 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:32.807492 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:32.811174 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:32.811546 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:32.783550 27622 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:32.811991 25244 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:32.814328 25242 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 13:08:32.815683 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 44374.4us
I20260709 13:08:32.786108 25936 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:32.778044 25938 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:32.778853 27813 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:32.816772 27817 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:32.819459 25244 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:08:32.819702 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:32.826659 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:32.827033 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:32.820019 27441 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:32.820535 27669 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:32.823333 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:32.781389 27811 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:32.824674 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:32.812960 27807 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:32.773466 27765 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:32.824086 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 1911) took 1263 ms. Trace:
I20260709 13:08:32.873339 25321 rpcz_store.cc:276] 0709 13:08:31.560723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:31.711161 (+150438us) service_pool.cc:224] Handling call
0709 13:08:32.822263 (+1111102us) spinlock_profiling.cc:243] Waited 420 ms on lock 0x7b54004399d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:32.824027 (+  1764us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":966066944}
I20260709 13:08:32.867019 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:32.875212 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:32.817538 27622 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:32.867615 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:32.838383 27669 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:32.825934 27798 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:32.880103 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:32.840161 27441 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:32.886806 25244 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:32.893044 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:32.876368 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:32.867821 27817 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:32.839406 27783 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:32.897436 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:32.835038 27774 consensus_queue.cc:1059] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20260709 13:08:32.834648 27813 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:32.885070 27622 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:32.905821 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:32.883800 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:32.906080 25244 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 3.
I20260709 13:08:32.877418 27807 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:32.920682 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:32.921856 25241 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
W20260709 13:08:32.924450 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 457865us
I20260709 13:08:32.928886 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:32.929787 25241 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:32.930464 25909 leader_election.cc:336] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 1.513s (SENT)
I20260709 13:08:32.917246 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:32.931348 25244 raft_consensus.cc:2376] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5. Re-sending same reply.
I20260709 13:08:32.931418 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 1922) took 1240 ms. Trace:
I20260709 13:08:32.931896 25312 rpcz_store.cc:276] 0709 13:08:31.699368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:32.491987 (+792619us) service_pool.cc:224] Handling call
0709 13:08:32.502014 (+ 10027us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:32.502274 (+   260us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:08:32.801167 (+298893us) log.cc:844] Serialized 14 byte log entry
0709 13:08:32.931024 (+129857us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:32.931072 (+    48us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:32.931186 (+   114us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:32.931248 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:32.931359 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":405569,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"thread_start_us":422432,"threads_started":2}
I20260709 13:08:32.915889 27774 consensus_queue.cc:1243] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer a2a77ecf2c0941618b5bc79d4c04db61 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:08:32.940717 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:32.899101 27798 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:32.917318 25937 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:32.946530 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:32.956828 27365 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:32.925478 25358 leader_election.cc:336] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.131s (SENT)
I20260709 13:08:32.966724 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:32.924763 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:32.970302 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:32.970531 25285 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:32.971060 25244 raft_consensus.cc:2376] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6. Re-sending same reply.
I20260709 13:08:32.972102 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:32.958526 27365 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:32.971249 25284 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:32.958645 25358 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:32.977026 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:32.985484 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:32.985903 25909 leader_election.cc:336] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 2.072s (SENT)
I20260709 13:08:32.986363 25909 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:32.997699 25284 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:33.004623 25284 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:33.010494 25909 leader_election.cc:336] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 2.150s (SENT)
W20260709 13:08:33.011633 25909 leader_election.cc:336] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 2.164s (SENT)
W20260709 13:08:33.012324 25909 leader_election.cc:336] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 1.863s (SENT)
I20260709 13:08:33.007040 25244 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:08:33.012782 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:33.022549 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:32.996658 25938 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:32.995728 27793 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:08:33.035961 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 110518us
I20260709 13:08:32.959710 27620 consensus_queue.cc:1059] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.056s
I20260709 13:08:32.981343 27624 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20260709 13:08:33.003335 27592 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:33.003589 27801 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:33.038681 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:33.018983 27676 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:33.053684 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:33.029842 27556 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:33.058580 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 1883) took 1307 ms. Trace:
I20260709 13:08:33.035715 27793 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:33.129266 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:33.113634 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:33.154047 25387 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:33.152240 25391 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 2.
W20260709 13:08:33.154621 25387 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 1876) took 1534 ms (client timeout 1968 ms). Trace:
I20260709 13:08:33.155176 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:33.081542 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:33.156806 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:33.154980 25387 rpcz_store.cc:269] 0709 13:08:31.620454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:31.644639 (+ 24185us) service_pool.cc:224] Handling call
0709 13:08:33.113421 (+1468782us) spinlock_profiling.cc:243] Waited 3.96 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:33.154566 (+ 41145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9118080}
I20260709 13:08:33.041483 27595 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:33.047034 27615 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:33.152565 25392 rpcz_store.cc:276] 0709 13:08:31.751600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:32.502968 (+751368us) service_pool.cc:224] Handling call
0709 13:08:32.503302 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:32.503550 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:08:32.866397 (+362847us) log.cc:844] Serialized 14 byte log entry
0709 13:08:33.058234 (+191837us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:33.058288 (+    54us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:33.058359 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:33.058427 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:33.058524 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7367,"prepare.queue_time_us":286211,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":4253,"raft.queue_time_us":26321,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":17477,"thread_start_us":525223,"threads_started":2}
I20260709 13:08:33.162242 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:33.114362 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1651) took 1449 ms. Trace:
I20260709 13:08:33.163800 25240 rpcz_store.cc:276] 0709 13:08:31.665222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:31.665648 (+   426us) service_pool.cc:224] Handling call
0709 13:08:31.665927 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:32.415453 (+749526us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:33.114070 (+698617us) spinlock_profiling.cc:243] Waited 739 ms on lock 0x7b54001f63d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:33.114287 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1700409088}
W20260709 13:08:33.164629 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:33.106011 27826 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 13:08:33.165573 25240 raft_consensus.cc:3055] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:33.165988 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:33.105398 27832 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
W20260709 13:08:33.114596 27842 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
W20260709 13:08:33.107163 27834 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 13:08:33.169852 25392 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
W20260709 13:08:33.170776 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:33.170426 25240 raft_consensus.cc:740] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:08:33.157284 27447 consensus_meta.cc:287] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.514s	user 0.010s	sys 0.001s
I20260709 13:08:33.082449 27840 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:33.170001 25244 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:08:33.076943 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:33.076717 27318 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.120s
I20260709 13:08:33.186704 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:33.187659 25940 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 498) took 1615 ms (client timeout 1607 ms). Trace:
W20260709 13:08:33.188050 25940 rpcz_store.cc:269] 0709 13:08:31.572239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:31.596596 (+ 24357us) service_pool.cc:224] Handling call
0709 13:08:32.918891 (+1322295us) spinlock_profiling.cc:243] Waited 2.48 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8deee 0000000000375939 00007f297bc011e0 00007f297bcca0c4 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:33.186611 (+267720us) spinlock_profiling.cc:243] Waited 6.78 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8df8c 00007f2973b8def6 0000000000375939 00007f297bc011e0 00007f297bcca0c4 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:33.187583 (+   972us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21295744}
I20260709 13:08:33.153483 27765 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:33.181149 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:33.173918 25209 connection.cc:256] RPC call timeout handler was delayed by -1.4926335509999262s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22210000000000002s for a response
I20260709 13:08:33.182068 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:33.197379 25387 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:33.176499 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:33.189328 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:33.199241 27764 raft_consensus.cc:3055] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:33.199683 27764 raft_consensus.cc:740] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:33.199784 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:33.201525 27556 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:33.201619 27318 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:08:33.202049 27556 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:33.202459 27556 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:33.203317 27839 log.cc:927] Time spent T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
I20260709 13:08:33.204604 25945 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 634) took 1730 ms. Trace:
I20260709 13:08:33.201028 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:33.157207 27793 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:33.215101 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:33.205032 25945 rpcz_store.cc:276] 0709 13:08:31.474466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:31.474582 (+   116us) service_pool.cc:224] Handling call
0709 13:08:31.488431 (+ 13849us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:32.586634 (+1098203us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:33.204291 (+617657us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400460fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:33.204491 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2524092160}
I20260709 13:08:33.161247 27615 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:33.166867 27721 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:33.081182 27365 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:33.247484 25241 raft_consensus.cc:2410] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:08:33.248690 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:33.249498 25241 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 2.
I20260709 13:08:33.066636 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:33.219894 25209 leader_election.cc:336] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.065s (SENT)
I20260709 13:08:33.181463 25240 consensus_queue.cc:260] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:33.233402 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:33.237558 27813 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:33.200512 27764 consensus_queue.cc:260] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:33.240957 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:33.339509 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 119651us
I20260709 13:08:33.339936 25387 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:08:33.340054 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:33.327865 27834 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 13:08:33.243269 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:33.244136 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 134251us
I20260709 13:08:33.212674 25244 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:08:33.344529 25244 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 13:08:33.345368 27824 raft_consensus.cc:3055] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:33.345772 27824 raft_consensus.cc:740] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:33.345801 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 501) took 1762 ms. Trace:
I20260709 13:08:33.341539 27774 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:33.346266 25942 rpcz_store.cc:276] 0709 13:08:31.583886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:31.624251 (+ 40365us) service_pool.cc:224] Handling call
0709 13:08:32.713214 (+1088963us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:32.713822 (+   608us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:33.345633 (+631811us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400690350. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:33.345742 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2363576704}
I20260709 13:08:33.346844 27774 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:33.347214 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:33.347755 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:33.347973 25390 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 1875) took 1735 ms (client timeout 2059 ms). Trace:
W20260709 13:08:33.348407 25390 rpcz_store.cc:269] 0709 13:08:31.613050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:31.662019 (+ 48969us) service_pool.cc:224] Handling call
0709 13:08:33.347071 (+1685052us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:33.347916 (+   845us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2826624}
I20260709 13:08:33.349826 27852 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:33.350322 27852 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:33.352345 27852 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:33.348866 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:33.353740 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:33.354236 25942 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:33.359627 27669 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:33.361431 25246 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:08:33.259281 27850 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 13:08:33.186771 27807 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:33.189286 27798 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:33.221453 27848 log.cc:927] Time spent T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 13:08:33.292447 27817 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:33.197096 27840 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:33.364032 25940 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
I20260709 13:08:33.365033 27840 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:33.304457 27844 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:08:33.312382 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:33.366253 25318 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:33.366700 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 335) took 1583 ms. Trace:
I20260709 13:08:33.366966 25318 rpcz_store.cc:276] 0709 13:08:31.783062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.312006 (+1528944us) service_pool.cc:224] Handling call
0709 13:08:33.366635 (+ 54629us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:33.314483 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:33.315963 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:33.368093 25395 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:08:33.317561 27837 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.234s	user 0.004s	sys 0.000s
I20260709 13:08:33.320533 25937 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:33.213853 27720 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:33.379058 27766 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260709 13:08:33.383442 27720 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:33.385550 27720 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:33.221529 27615 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:33.326571 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:33.190936 27676 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.087s
I20260709 13:08:33.330211 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:33.330816 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:33.329368 25359 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:33.333783 25240 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:33.389632 25388 raft_consensus.cc:2393] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 7: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:08:33.334354 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 122134us
I20260709 13:08:33.340623 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 1914) took 1779 ms. Trace:
I20260709 13:08:33.340759 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:33.391481 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:33.241375 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:33.196735 27447 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:33.369031 25942 raft_consensus.cc:2410] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260709 13:08:33.374250 25395 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 1868) took 1763 ms (client timeout 1853 ms). Trace:
W20260709 13:08:33.409994 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 80708.9us
I20260709 13:08:33.398259 25241 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 2.
I20260709 13:08:33.415832 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:33.416501 25241 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:33.398469 25393 raft_consensus.cc:2393] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:33.397054 25317 rpcz_store.cc:276] 0709 13:08:31.561484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:32.454308 (+892824us) service_pool.cc:224] Handling call
0709 13:08:32.454602 (+   294us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:32.455201 (+   599us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:32.455230 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:32.455743 (+   513us) log.cc:844] Serialized 26 byte log entry
0709 13:08:32.806376 (+350633us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:32.806434 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:32.806524 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:33.340418 (+533894us) raft_consensus.cc:1685] finished
0709 13:08:33.340437 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:33.340556 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":84,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":105,"thread_start_us":349111,"threads_started":1,"wal-append.queue_time_us":455058}
I20260709 13:08:33.392167 25945 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
W20260709 13:08:33.400804 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:33.418292 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:33.397368 25909 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:33.406342 25240 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 13:08:33.419418 25240 pending_rounds.cc:85] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 2
I20260709 13:08:33.352851 27852 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.028s
I20260709 13:08:33.271319 27721 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:33.410172 25395 rpcz_store.cc:269] 0709 13:08:31.611217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:31.614365 (+  3148us) service_pool.cc:224] Handling call
0709 13:08:33.315807 (+1701442us) spinlock_profiling.cc:243] Waited 3.95 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:33.374157 (+ 58350us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9087360}
I20260709 13:08:33.348587 27556 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:33.444584 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 47323.8us
I20260709 13:08:33.419780 25240 pending_rounds.cc:107] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 13:08:33.436177 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:33.448040 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 52416.6us
W20260709 13:08:33.443979 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 103280us
I20260709 13:08:33.436404 25317 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:08:33.449353 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:33.450246 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:33.418756 25285 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:33.348968 27774 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:33.389065 25941 consensus_meta.cc:287] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.605s	user 0.005s	sys 0.004s
I20260709 13:08:33.488900 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:33.389382 25238 raft_consensus.cc:2393] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:33.380052 27813 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:33.369891 27793 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:33.500136 25284 leader_election.cc:336] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Timed out: RequestConsensusVote RPC to 127.23.140.68:39211 timed out after 1.607s (SENT)
I20260709 13:08:33.414623 27764 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:33.544703 25390 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 6: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:33.545676 25391 raft_consensus.cc:2393] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:33.547827 25943 raft_consensus.cc:2393] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:33.346616 27824 consensus_queue.cc:260] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:33.497311 25388 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:33.549101 27676 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:08:33.444051 27804 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:33.431802 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:33.427511 25287 connection.cc:256] RPC call timeout handler was delayed by -1.4271027140000569s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1553s for a response
I20260709 13:08:34.083557 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 503) took 1286 ms. Trace:
W20260709 13:08:33.706199 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:33.849732 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 431024us
I20260709 13:08:34.101313 25240 raft_consensus.cc:2889] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:34.104409 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:33.911187 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:34.105213 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1424) took 1186 ms. Trace:
I20260709 13:08:34.105338 25209 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:33.451418 27443 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:34.102288 25942 rpcz_store.cc:276] 0709 13:08:32.796726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.500612 (+703886us) service_pool.cc:224] Handling call
0709 13:08:34.082966 (+582354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:34.083184 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:34.083212 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:34.083257 (+    45us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:34.083304 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:34.083360 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:34.083481 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:34.109524 25391 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:08:34.103245 25318 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 3.
W20260709 13:08:34.107204 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:34.105786 25240 rpcz_store.cc:276] 0709 13:08:32.919261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.165047 (+245786us) service_pool.cc:224] Handling call
0709 13:08:33.165423 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:34.104894 (+939471us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:34.105116 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:34.130666 25390 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:08:34.131110 25241 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:34.133104 25241 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:34.091799 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:34.091223 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:34.106074 25242 raft_consensus.cc:2410] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:08:33.480711 25936 raft_consensus.cc:2410] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:08:33.579772 25395 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 13:08:34.106709 25284 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:33.934296 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:33.773383 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:34.102553 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:33.699044 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 329327us
I20260709 13:08:33.451830 27802 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:34.397719 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:08:34.397295 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 254812us
I20260709 13:08:34.401866 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:34.404242 25942 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 13:08:34.404927 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 505) took 1606 ms. Trace:
I20260709 13:08:34.405326 25942 rpcz_store.cc:276] 0709 13:08:32.798439 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:34.396834 (+1598395us) service_pool.cc:224] Handling call
0709 13:08:34.403904 (+  7070us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:34.404657 (+   753us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:34.404873 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:08:34.372468 25242 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 433) took 1531 ms (client timeout 1863 ms). Trace:
I20260709 13:08:34.371794 25315 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:08:34.398820 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:33.557709 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:34.413060 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:34.415777 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:34.416260 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:34.421550 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 319090us
I20260709 13:08:34.422572 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:34.425431 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:34.420811 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:34.426281 25390 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
W20260709 13:08:34.409106 25242 rpcz_store.cc:269] 0709 13:08:32.840625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:32.868500 (+ 27875us) service_pool.cc:224] Handling call
0709 13:08:33.343998 (+475498us) spinlock_profiling.cc:243] Waited 8.12 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:33.911032 (+567034us) spinlock_profiling.cc:243] Waited 23.7 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8deee 0000000000375939 00007f297bc011e0 00007f297bcca0c4 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:34.372379 (+461347us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":73121792}
I20260709 13:08:34.413480 25394 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
W20260709 13:08:34.435521 25212 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.598s (SENT) [suppressed 43 similar messages]
I20260709 13:08:34.438621 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:33.542593 27807 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:33.713898 27855 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
I20260709 13:08:33.807636 25937 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:08:34.445189 27812 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:34.445544 27620 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:08:34.445716 27365 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:34.446465 27721 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:34.447315 27860 log.cc:927] Time spent T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.732s	user 0.001s	sys 0.001s
I20260709 13:08:34.396732 27804 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:34.452703 27615 consensus_meta.cc:287] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.066s	user 0.007s	sys 0.000s
I20260709 13:08:34.514376 27867 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:34.514218 27615 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:34.514933 27867 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:34.516394 27615 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:33.551462 25941 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
W20260709 13:08:34.517313 25941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 646) took 1882 ms (client timeout 1871 ms). Trace:
W20260709 13:08:34.517701 25941 rpcz_store.cc:269] 0709 13:08:32.634678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:32.712212 (+ 77534us) service_pool.cc:224] Handling call
0709 13:08:34.517229 (+1805017us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:34.518164 27853 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:34.518740 27853 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:34.519579 27853 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:33.673895 25938 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:34.085330 27720 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:34.061899 27669 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:34.369510 27813 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:33.557395 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1279) took 2000 ms. Trace:
I20260709 13:08:34.454843 27630 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:34.400787 27828 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:33.932335 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:34.369172 27443 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:34.543419 27443 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:34.544891 27813 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:34.545607 27813 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:34.465814 27556 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:34.467741 27793 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:34.470248 27447 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:33.494306 27815 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:33.879369 27441 consensus_meta.cc:287] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.974s	user 0.003s	sys 0.008s
I20260709 13:08:34.558724 27815 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:34.558827 27441 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:34.560734 27441 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:33.721487 27623 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.207s	user 0.015s	sys 0.011s
W20260709 13:08:34.561546 27623 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.050s	user 0.023s	sys 0.016s
I20260709 13:08:33.489583 27798 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:34.562438 27798 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:34.562857 27798 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:34.443831 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 635) took 1370 ms. Trace:
I20260709 13:08:34.564318 25942 rpcz_store.cc:276] 0709 13:08:33.036704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:34.406384 (+1369680us) service_pool.cc:224] Handling call
0709 13:08:34.406733 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:34.406923 (+   190us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:34.406951 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:34.406995 (+    44us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:34.407042 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:34.407096 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:34.407197 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:34.362703 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:33.861186 27810 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:34.438508 27859 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:34.498416 27365 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:34.503424 25945 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:08:34.475759 27824 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:34.491302 27812 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:34.516929 27867 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:34.560760 27815 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:34.500870 27622 consensus_meta.cc:287] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 1.584s	user 0.010s	sys 0.002s
I20260709 13:08:34.548470 27630 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:34.520766 27720 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:34.731879 25240 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.012s	sys 0.021s
W20260709 13:08:34.734019 25240 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.012s	sys 0.021s
I20260709 13:08:34.490733 27807 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:34.500356 25936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 642) took 1911 ms (client timeout 1518 ms). Trace:
I20260709 13:08:34.557547 27793 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:34.558183 27447 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:34.516965 27615 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:34.568600 25942 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:34.762722 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 640) took 1705 ms. Trace:
I20260709 13:08:34.566195 25386 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
W20260709 13:08:34.566048 27642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.047s	sys 0.059s
I20260709 13:08:34.553279 27669 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:08:34.571339 25940 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.048s	sys 0.057s
W20260709 13:08:34.609586 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 174080us
I20260709 13:08:34.605028 27365 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:34.532236 25389 rpcz_store.cc:276] 0709 13:08:31.557420 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:31.566694 (+  9274us) service_pool.cc:224] Handling call
0709 13:08:31.679838 (+113144us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:32.310003 (+630165us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:32.318711 (+  8708us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:32.462355 (+143644us) log.cc:844] Serialized 26 byte log entry
0709 13:08:32.873977 (+411622us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:32.917790 (+ 43813us) log.cc:844] Serialized 14 byte log entry
0709 13:08:32.938852 (+ 21062us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:32.945668 (+  6816us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:33.489730 (+544062us) raft_consensus.cc:1685] finished
0709 13:08:33.491688 (+  1958us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:33.552280 (+ 60592us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4596,"prepare.queue_time_us":60001,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":6831,"raft.queue_time_us":393567,"raft.run_cpu_time_us":357,"raft.run_wall_time_us":49317,"thread_start_us":822554,"threads_started":2,"wal-append.queue_time_us":452711,"wal-append.run_cpu_time_us":14296,"wal-append.run_wall_time_us":1613092}
I20260709 13:08:34.539489 27828 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:34.592490 27810 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:34.605540 27859 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:34.622991 27812 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:34.721308 25285 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:34.442338 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 439) took 1400 ms. Trace:
W20260709 13:08:34.742690 25211 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.000s	sys 0.033s
I20260709 13:08:34.746381 25237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1412) took 1937 ms. Trace:
W20260709 13:08:34.806807 25211 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.000s	sys 0.034s
I20260709 13:08:34.747009 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1653) took 2184 ms. Trace:
W20260709 13:08:34.753588 25209 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.037s	sys 0.042s
I20260709 13:08:34.536664 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 649) took 1875 ms. Trace:
I20260709 13:08:34.808400 25243 rpcz_store.cc:276] 0709 13:08:32.537965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:32.538151 (+   186us) service_pool.cc:224] Handling call
0709 13:08:32.588916 (+ 50765us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:32.784036 (+195120us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:34.637546 (+1853510us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b5400059454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:34.722344 (+ 84798us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":447340800}
I20260709 13:08:34.763342 25942 rpcz_store.cc:276] 0709 13:08:33.057790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:34.565248 (+1507458us) service_pool.cc:224] Handling call
0709 13:08:34.565531 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:34.762380 (+196849us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:34.762642 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:08:34.776614 25212 connection.cc:256] RPC call timeout handler was delayed by -1.0940637719999131s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21150000000000002s for a response
I20260709 13:08:34.780642 25387 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:34.782886 25242 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:08:34.733224 27815 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:34.759968 27447 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:34.737094 27622 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:34.808130 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:34.808585 25209 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.037s	sys 0.042s
W20260709 13:08:34.774447 27642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.713s	user 0.047s	sys 0.060s
W20260709 13:08:34.809274 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 201722us
I20260709 13:08:34.806980 25237 rpcz_store.cc:276] 0709 13:08:32.809078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:32.809666 (+   588us) service_pool.cc:224] Handling call
0709 13:08:32.810175 (+   509us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:32.825274 (+ 15099us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:32.825505 (+   231us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:33.206108 (+380603us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b2c000014c8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f29755aba11 00007f29755bb372 00007f2976ceda01 00007f2976cb243a 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2
0709 13:08:33.206539 (+   431us) log.cc:844] Serialized 26 byte log entry
0709 13:08:33.442554 (+236015us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:33.442608 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:33.449974 (+  7366us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:34.746002 (+1296028us) raft_consensus.cc:1685] finished
0709 13:08:34.746042 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:34.746304 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6999,"raft.queue_time_us":14820,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":12484,"spinlock_wait_cycles":26417280,"thread_start_us":235583,"threads_started":1,"wal-append.queue_time_us":507955}
W20260709 13:08:34.808986 25241 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.674s	user 0.033s	sys 0.034s
W20260709 13:08:34.818588 25944 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.004s	sys 0.012s
I20260709 13:08:34.772324 27443 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:34.774155 27615 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:34.898464 25945 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 644) took 2296 ms. Trace:
I20260709 13:08:35.030203 25321 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:34.798869 27810 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:08:34.743151 25936 rpcz_store.cc:269] 0709 13:08:32.585629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:32.588755 (+  3126us) service_pool.cc:224] Handling call
0709 13:08:34.497275 (+1908520us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:35.037357 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1662) took 1826 ms. Trace:
W20260709 13:08:34.977205 27764 consensus_meta.cc:287] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.607s	user 0.012s	sys 0.000s
I20260709 13:08:35.053170 27764 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:08:35.052809 25244 rpcz_store.cc:276] 0709 13:08:33.211357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.345993 (+134636us) service_pool.cc:224] Handling call
0709 13:08:34.258458 (+912465us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:34.606296 (+347838us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:34.606342 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:34.606404 (+    62us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:34.606467 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:34.606555 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:35.037064 (+430509us) spinlock_profiling.cc:243] Waited 912 ms on lock 0x7b5400456850. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:35.037275 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":637581,"raft.run_cpu_time_us":67,"raft.run_wall_time_us":66,"spinlock_wait_cycles":2097695616,"thread_start_us":345744,"threads_started":1}
I20260709 13:08:35.048234 25945 rpcz_store.cc:276] 0709 13:08:32.602133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.453888 (+851755us) service_pool.cc:224] Handling call
0709 13:08:33.490313 (+ 36425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:34.598304 (+1107991us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:34.898150 (+299846us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54006608d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:34.898380 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2329006976}
W20260709 13:08:34.971820 27864 log.cc:927] Time spent T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.218s	user 0.000s	sys 0.001s
I20260709 13:08:35.055704 25945 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 643) took 1951 ms. Trace:
I20260709 13:08:34.899501 27798 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.058418 27798 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:35.029798 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 171559us
W20260709 13:08:34.904371 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 465785us
I20260709 13:08:35.061920 27872 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:08:35.064504 27872 consensus_queue.cc:1059] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:08:34.944291 25387 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:34.793582 27877 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:35.020905 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:35.048020 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 642) took 1952 ms. Trace:
I20260709 13:08:35.029623 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:35.048686 25944 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.957s	user 0.016s	sys 0.024s
W20260709 13:08:35.048992 25128 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f75b60146e7d4b9b9954b4fb2c414679 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 2329e428ebbf484fa3ea9452133df567 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:08:35.047745 25241 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.913s	user 0.037s	sys 0.051s
I20260709 13:08:35.056058 25945 rpcz_store.cc:276] 0709 13:08:33.103958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:35.054895 (+1950937us) service_pool.cc:224] Handling call
0709 13:08:35.055200 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:35.055376 (+   176us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:35.055404 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:35.055449 (+    45us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:35.055490 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:35.055543 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:35.055645 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:35.087775 25312 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
W20260709 13:08:35.088495 25312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1821) took 1634 ms (client timeout 1598 ms). Trace:
I20260709 13:08:34.996419 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:35.089062 25312 rpcz_store.cc:269] 0709 13:08:33.454451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.485113 (+ 30662us) service_pool.cc:224] Handling call
0709 13:08:35.029371 (+1544258us) spinlock_profiling.cc:243] Waited 2.86 ms on lock 0x7b58000e3498. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:35.088414 (+ 59043us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6568192}
I20260709 13:08:35.052021 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1822) took 1566 ms. Trace:
I20260709 13:08:35.075359 25212 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:35.090744 25321 rpcz_store.cc:276] 0709 13:08:33.485392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.487779 (+  2387us) service_pool.cc:224] Handling call
0709 13:08:34.993913 (+1506134us) spinlock_profiling.cc:243] Waited 3.72 ms on lock 0x7b58000e3498. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:34.994143 (+   230us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:35.051742 (+ 57599us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:35.051951 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8549760}
I20260709 13:08:35.094527 25212 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:34.782051 27669 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:08:35.090059 25320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1830) took 1593 ms (client timeout 1712 ms). Trace:
W20260709 13:08:35.096154 25320 rpcz_store.cc:269] 0709 13:08:33.496252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.682297 (+186045us) service_pool.cc:224] Handling call
0709 13:08:34.992907 (+1310610us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b58000e3498. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:35.089989 (+ 97082us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":655117568}
I20260709 13:08:35.065404 27873 raft_consensus.cc:3055] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Stepping down as leader of term 8
W20260709 13:08:35.048173 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 942830us
I20260709 13:08:34.909070 27793 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:35.053867 27828 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:35.097093 25358 connection.cc:256] RPC call timeout handler was delayed by -1.796548603999895s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19840000000000002s for a response
I20260709 13:08:34.920768 25940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 643) took 2327 ms. Trace:
I20260709 13:08:35.098950 25944 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 500) took 3515 ms. Trace:
I20260709 13:08:34.971331 27620 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:35.109189 25358 connection.cc:256] RPC call timeout handler was delayed by -1.7668486039999607s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2017s for a response
I20260709 13:08:35.029706 27773 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:34.896243 27720 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:35.100764 25209 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:35.106542 25287 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.014s	sys 0.023s
W20260709 13:08:35.128773 25287 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.014s	sys 0.023s
I20260709 13:08:35.108572 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1432) took 1722 ms. Trace:
I20260709 13:08:35.129190 25242 rpcz_store.cc:276] 0709 13:08:33.386540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:34.471971 (+1085431us) service_pool.cc:224] Handling call
0709 13:08:34.480288 (+  8317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:34.814743 (+334455us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:35.108268 (+293525us) spinlock_profiling.cc:243] Waited 302 ms on lock 0x7b54002d0fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:35.108496 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":694884736}
I20260709 13:08:34.806515 25238 rpcz_store.cc:276] 0709 13:08:33.031104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.066217 (+ 35113us) service_pool.cc:224] Handling call
0709 13:08:34.430855 (+1364638us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:35.101162 25942 rpcz_store.cc:276] 0709 13:08:33.096171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:34.827443 (+1731272us) service_pool.cc:224] Handling call
0709 13:08:34.827847 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:34.882118 (+ 54271us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:34.882169 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:34.882238 (+    69us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:34.882298 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:34.882390 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:35.047962 (+165572us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":84,"raft.queue_time_us":30221,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":107541}
I20260709 13:08:35.117496 25944 rpcz_store.cc:276] 0709 13:08:31.583530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:31.607837 (+ 24307us) service_pool.cc:224] Handling call
0709 13:08:33.349433 (+1741596us) spinlock_profiling.cc:243] Waited 2.56 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:33.349629 (+   196us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:33.351835 (+  2206us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:33.351866 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:33.352337 (+   471us) log.cc:844] Serialized 26 byte log entry
0709 13:08:33.543103 (+190766us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:08:35.087805 (+1544702us) log.cc:844] Serialized 14 byte log entry
0709 13:08:35.089270 (+  1465us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:35.089363 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:35.089422 (+    59us) raft_consensus.cc:1685] finished
0709 13:08:35.098471 (+  9049us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:35.098805 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1227995,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":8638,"raft.queue_time_us":4969,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":128,"spinlock_wait_cycles":47937664,"thread_start_us":1186650,"threads_started":2,"wal-append.queue_time_us":1217851}
I20260709 13:08:35.099952 27807 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:35.098683 27877 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.164095 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:35.124933 25358 connection.cc:256] RPC call timeout handler was delayed by -1.7380486039999141s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2049s for a response
I20260709 13:08:34.816334 27878 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:35.164882 25242 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 7
W20260709 13:08:35.165120 25358 connection.cc:256] RPC call timeout handler was delayed by -1.683148603999939s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.211s for a response
W20260709 13:08:35.106917 25393 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.017s	sys 0.028s
I20260709 13:08:35.117846 27798 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:08:35.097548 27873 raft_consensus.cc:740] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:08:35.108515 25359 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.019s	sys 0.005s
I20260709 13:08:35.135387 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1825) took 1646 ms. Trace:
W20260709 13:08:35.173174 25359 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.023s	sys 0.005s
W20260709 13:08:35.173550 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 748177us
W20260709 13:08:35.135449 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 733600us
I20260709 13:08:35.173316 25316 rpcz_store.cc:276] 0709 13:08:33.488823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.488922 (+    99us) service_pool.cc:224] Handling call
0709 13:08:35.134452 (+1645530us) spinlock_profiling.cc:243] Waited 9.56 ms on lock 0x7b58000e3498. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:35.134699 (+   247us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:35.134939 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:35.134978 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:35.135033 (+    55us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:35.135094 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:35.135167 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:35.135308 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21993344}
I20260709 13:08:35.174691 25287 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:35.181205 25944 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 654) took 1842 ms. Trace:
I20260709 13:08:35.181610 25944 rpcz_store.cc:276] 0709 13:08:33.338616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:35.158784 (+1820168us) service_pool.cc:224] Handling call
0709 13:08:35.180614 (+ 21830us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:35.180814 (+   200us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:35.180847 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:35.180893 (+    46us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:35.180942 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:35.181004 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:35.181117 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:08:35.183379 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:35.136245 25909 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.041s	user 0.011s	sys 0.011s
I20260709 13:08:35.103000 27669 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:34.884709 27815 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:35.126381 27773 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:35.138607 27847 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 2.060s	user 0.011s	sys 0.015s
I20260709 13:08:34.811596 25937 rpcz_store.cc:276] 0709 13:08:32.656722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.486508 (+829786us) service_pool.cc:224] Handling call
0709 13:08:33.674183 (+187675us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:34.490955 (+816772us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:34.532142 (+ 41187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:35.037683 27858 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:35.156898 25908 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.017s	sys 0.032s
I20260709 13:08:35.145318 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:35.226083 25908 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.017s	sys 0.032s
I20260709 13:08:35.226297 25314 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
W20260709 13:08:35.226815 25314 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1823) took 1738 ms (client timeout 1844 ms). Trace:
I20260709 13:08:35.227196 25242 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:35.050930 27447 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:35.166587 25393 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.018s	sys 0.028s
W20260709 13:08:35.142378 25284 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.014s	sys 0.040s
W20260709 13:08:35.185338 25909 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.090s	user 0.011s	sys 0.011s
W20260709 13:08:35.230827 25284 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.014s	sys 0.040s
I20260709 13:08:35.168208 27798 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:35.192402 25359 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:35.130918 27620 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.217721 25245 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:35.217635 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 429) took 2394 ms. Trace:
I20260709 13:08:35.240480 25246 rpcz_store.cc:276] 0709 13:08:32.823378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:32.976792 (+153414us) service_pool.cc:224] Handling call
0709 13:08:35.217366 (+2240574us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b54003eb054. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:35.217542 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":867623808}
W20260709 13:08:35.227161 25314 rpcz_store.cc:269] 0709 13:08:33.488184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.488334 (+   150us) service_pool.cc:224] Handling call
0709 13:08:35.145109 (+1656775us) spinlock_profiling.cc:243] Waited 9.31 ms on lock 0x7b58000e3498. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:35.226758 (+ 81649us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21424384}
I20260709 13:08:35.117684 27793 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:35.229310 27642 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:35.228776 25908 leader_election.cc:336] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.513s (SENT)
I20260709 13:08:35.243366 25908 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:35.169975 27878 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:35.141637 27630 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:35.231410 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 1.73115e+06us
W20260709 13:08:35.237968 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 45626.4us
W20260709 13:08:35.248524 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:35.249652 25284 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:35.250190 25908 leader_election.cc:336] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.640s (SENT)
I20260709 13:08:35.250801 25908 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:35.239360 27798 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:08:35.245723 27720 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:35.218407 27884 consensus_queue.cc:1059] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.064s
I20260709 13:08:35.245474 27878 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:35.263448 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1668) took 1771 ms. Trace:
I20260709 13:08:35.272426 25242 rpcz_store.cc:276] 0709 13:08:33.492444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:35.237284 (+1744840us) service_pool.cc:224] Handling call
0709 13:08:35.262739 (+ 25455us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:35.263002 (+   263us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:35.263036 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:35.263092 (+    56us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:35.263156 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:35.263230 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:35.263379 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:35.231719 27858 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:35.275717 25908 leader_election.cc:336] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.836s (SENT)
W20260709 13:08:35.279528 25908 leader_election.cc:336] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.948s (SENT)
I20260709 13:08:35.280020 25908 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:35.283982 25942 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:35.266158 27720 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:35.256673 27890 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:35.247010 27642 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.265375 27443 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:35.267421 27874 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:35.265628 27879 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:35.308593 25944 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:08:35.298924 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:35.288062 27885 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:35.272135 27365 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.277827 27882 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:35.297981 27720 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:35.290901 27884 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:35.288865 27622 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:35.289219 27817 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:35.274114 27875 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:08:35.304659 27871 log.cc:927] Time spent T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
W20260709 13:08:35.296885 27862 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.009s	sys 0.016s
I20260709 13:08:35.257259 27873 consensus_queue.cc:260] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:35.315442 27879 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:08:35.322110 27824 consensus_meta.cc:287] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.679s	user 0.008s	sys 0.003s
I20260709 13:08:35.320048 27687 raft_consensus.cc:3055] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:35.325390 27884 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:35.351667 27875 consensus_queue.cc:1059] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
W20260709 13:08:35.337859 27862 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.009s	sys 0.016s
I20260709 13:08:35.352847 27365 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:35.353332 27873 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:08:35.353503 27887 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20260709 13:08:35.368686 27815 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:35.369467 27889 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 13:08:35.373553 27773 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:35.374617 27793 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:35.376523 27620 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:35.319923 27885 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:35.344875 27879 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:35.357270 27884 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:35.347927 27687 raft_consensus.cc:740] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:35.360402 27877 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:35.397964 27716 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:35.324766 27882 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:35.407320 27877 raft_consensus.cc:3055] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:35.432646 27804 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:35.433946 27859 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:35.435930 27865 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.665s	user 0.001s	sys 0.001s
I20260709 13:08:35.424479 27877 raft_consensus.cc:740] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:35.436365 27810 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:35.443979 27858 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:35.445747 27877 consensus_queue.cc:260] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.447482 27882 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:35.451323 27885 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:35.453861 27884 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.456337 27884 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:35.461414 27878 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:35.463702 27878 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:35.313040 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:35.322508 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:35.508409 25939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 641) took 2949 ms. Trace:
I20260709 13:08:35.531351 25945 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 653) took 2192 ms. Trace:
I20260709 13:08:35.491734 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1427) took 2140 ms. Trace:
W20260709 13:08:35.517174 27812 consensus_meta.cc:287] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.696s	user 0.006s	sys 0.006s
W20260709 13:08:35.493984 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 358988us
W20260709 13:08:35.490199 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 394332us
I20260709 13:08:35.523066 27642 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:35.492659 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1431) took 2107 ms. Trace:
I20260709 13:08:35.494618 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1667) took 2009 ms. Trace:
I20260709 13:08:35.543282 25212 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:35.495889 25237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1664) took 2081 ms. Trace:
W20260709 13:08:35.490849 25243 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.015s	sys 0.009s
W20260709 13:08:35.497967 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 266662us
W20260709 13:08:35.495424 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 329378us
W20260709 13:08:35.496834 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 247132us
W20260709 13:08:35.493956 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 319151us
I20260709 13:08:35.531622 25939 rpcz_store.cc:276] 0709 13:08:32.553721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:32.585731 (+ 32010us) service_pool.cc:224] Handling call
0709 13:08:33.309715 (+723984us) spinlock_profiling.cc:243] Waited 26.1 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8deee 0000000000375939 00007f297bc011e0 00007f297bcca0c4 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:33.309929 (+   214us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:33.310485 (+   556us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:33.310543 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:33.360004 (+ 49461us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b2c000476c8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f29755aba11 00007f29755bb372 00007f2976ceda01 00007f2976cb243a 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2
0709 13:08:33.360527 (+   523us) log.cc:844] Serialized 26 byte log entry
0709 13:08:33.557778 (+197251us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:33.586481 (+ 28703us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:33.678226 (+ 91745us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:35.435374 (+1757148us) spinlock_profiling.cc:243] Waited 5.05 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:35.499458 (+ 64084us) raft_consensus.cc:1685] finished
0709 13:08:35.499520 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:35.501916 (+  2396us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":156,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":271,"spinlock_wait_cycles":105524096,"thread_start_us":132241,"threads_started":1,"wal-append.queue_time_us":1193565}
I20260709 13:08:35.549837 27615 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:35.427439 27687 consensus_queue.cc:260] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.552541 27687 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:35.117404 25940 rpcz_store.cc:276] 0709 13:08:32.593828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.364810 (+770982us) service_pool.cc:224] Handling call
0709 13:08:33.365172 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:33.365425 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:08:33.855429 (+490004us) log.cc:844] Serialized 14 byte log entry
0709 13:08:34.798604 (+943175us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:34.808573 (+  9969us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:34.816597 (+  8024us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:34.858669 (+ 42072us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:34.900358 (+ 41689us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8768,"prepare.queue_time_us":1305066,"prepare.run_cpu_time_us":2508,"prepare.run_wall_time_us":50671,"raft.queue_time_us":11848,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":50995,"thread_start_us":1258005,"threads_started":2,"wal-append.queue_time_us":1242337}
I20260709 13:08:35.555773 27810 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:35.556167 25243 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.016s	sys 0.009s
I20260709 13:08:35.558156 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1827) took 2035 ms. Trace:
I20260709 13:08:35.558938 25317 rpcz_store.cc:276] 0709 13:08:33.495491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:34.279747 (+784256us) service_pool.cc:224] Handling call
0709 13:08:35.526350 (+1246603us) spinlock_profiling.cc:243] Waited 84.5 ms on lock 0x7b58000e3498. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:35.526964 (+   614us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:35.528058 (+  1094us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:35.528119 (+    61us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:35.528248 (+   129us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:35.528343 (+    95us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:35.528515 (+   172us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:35.528816 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1759,"raft.run_cpu_time_us":410,"raft.run_wall_time_us":4214,"spinlock_wait_cycles":194456832}
I20260709 13:08:35.540239 25240 rpcz_store.cc:276] 0709 13:08:33.351483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:34.113620 (+762137us) service_pool.cc:224] Handling call
0709 13:08:34.114027 (+   407us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:34.737471 (+623444us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:34.737516 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:34.741731 (+  4215us) log.cc:844] Serialized 26 byte log entry
0709 13:08:35.247363 (+505632us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:35.247421 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:35.247510 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:35.490784 (+243274us) raft_consensus.cc:1685] finished
0709 13:08:35.490878 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:35.491335 (+   457us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":781550,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":129,"thread_start_us":1053129,"threads_started":2,"wal-append.queue_time_us":566141}
I20260709 13:08:35.541227 25241 rpcz_store.cc:276] 0709 13:08:33.385521 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:34.134185 (+748664us) service_pool.cc:224] Handling call
0709 13:08:34.134516 (+   331us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:35.088175 (+953659us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:35.088201 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:35.088765 (+   564us) log.cc:844] Serialized 26 byte log entry
0709 13:08:35.265020 (+176255us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:08:35.265060 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:35.265151 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:35.491965 (+226814us) raft_consensus.cc:1685] finished
0709 13:08:35.492024 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:35.492425 (+   401us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":761336,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":65,"thread_start_us":1128396,"threads_started":2,"wal-append.queue_time_us":252776}
I20260709 13:08:35.542402 25238 rpcz_store.cc:276] 0709 13:08:33.484674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:35.180150 (+1695476us) service_pool.cc:224] Handling call
0709 13:08:35.250151 (+ 70001us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:35.261826 (+ 11675us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:35.266672 (+  4846us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:35.268951 (+  2279us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:35.493798 (+224847us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:35.494044 (+   246us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:35.494355 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:08:35.555812 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:35.533615 27881 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:35.569761 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1426) took 2464 ms. Trace:
I20260709 13:08:35.570763 25313 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 3: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:08:35.570613 25908 leader_election.cc:336] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.164s (SENT)
I20260709 13:08:35.572165 25287 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:35.572944 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:35.575630 25284 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:35.532068 25945 rpcz_store.cc:276] 0709 13:08:33.338320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:35.088284 (+1749964us) service_pool.cc:224] Handling call
0709 13:08:35.088580 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:35.088774 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:08:35.088987 (+   213us) log.cc:844] Serialized 14 byte log entry
0709 13:08:35.530078 (+441091us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:35.530223 (+   145us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:35.530493 (+   270us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:35.530696 (+   203us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:35.531042 (+   346us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":11839,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":1210,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":320,"thread_start_us":440101,"threads_started":1,"wal-append.queue_time_us":311435}
I20260709 13:08:35.578034 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:35.578428 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:35.579061 25938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 637) took 2530 ms (client timeout 1598 ms). Trace:
I20260709 13:08:35.580536 25944 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:08:35.605655 25394 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 5. Current term is 6. Ops: []
W20260709 13:08:35.605347 25938 rpcz_store.cc:269] 0709 13:08:33.048652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.076423 (+ 27771us) service_pool.cc:224] Handling call
0709 13:08:35.578527 (+2502104us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b5400644954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:35.578756 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":922672896}
W20260709 13:08:35.582952 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:35.584301 25943 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 638) took 2533 ms (client timeout 2470 ms). Trace:
W20260709 13:08:35.607611 25943 rpcz_store.cc:269] 0709 13:08:33.050860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.314159 (+263299us) service_pool.cc:224] Handling call
0709 13:08:35.583573 (+2269414us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b54006d9554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:35.583995 (+   422us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":413307648}
W20260709 13:08:35.586618 25358 connection.cc:256] RPC call timeout handler was delayed by -1.648048603999996s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2149s for a response
W20260709 13:08:35.591601 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 48721us
I20260709 13:08:35.533814 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 639) took 2480 ms. Trace:
I20260709 13:08:35.589497 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:35.590452 25245 rpcz_store.cc:276] 0709 13:08:33.056506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.202190 (+145684us) service_pool.cc:224] Handling call
0709 13:08:35.217527 (+2015337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:35.240425 (+ 22898us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:35.520568 (+280143us) spinlock_profiling.cc:243] Waited 2.01 s on lock 0x7b54003eb050. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:35.520937 (+   369us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4634100992}
I20260709 13:08:35.544760 25237 rpcz_store.cc:276] 0709 13:08:33.414101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:35.061595 (+1647494us) service_pool.cc:224] Handling call
0709 13:08:35.067066 (+  5471us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:35.067474 (+   408us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:35.067514 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:35.102973 (+ 35459us) log.cc:844] Serialized 26 byte log entry
0709 13:08:35.494907 (+391934us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:35.495042 (+   135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:35.495237 (+   195us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:35.495347 (+   110us) raft_consensus.cc:1685] finished
0709 13:08:35.495369 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:35.495638 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9943,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":17214,"thread_start_us":360815,"threads_started":1,"wal-append.queue_time_us":219408}
I20260709 13:08:35.595355 27873 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:08:35.595908 25318 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1831) took 2099 ms (client timeout 2217 ms). Trace:
I20260709 13:08:35.596427 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:35.596549 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:35.597496 27365 consensus_queue.cc:1059] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:08:35.595942 25936 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:35.626278 27813 raft_consensus.cc:3055] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:35.626868 27813 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:35.628180 27874 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:35.629655 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:35.631453 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:35.632839 25390 raft_consensus.cc:2393] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 7: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:35.633227 27873 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:35.633818 27873 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:35.637323 25908 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:35.628628 27813 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.598673 27642 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:35.651171 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1415) took 2839 ms. Trace:
I20260709 13:08:35.651046 27642 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.582000 25242 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:35.615597 25941 rpcz_store.cc:276] 0709 13:08:33.053416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:34.518605 (+1465189us) service_pool.cc:224] Handling call
0709 13:08:34.518972 (+   367us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:34.519351 (+   379us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:34.519381 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:34.519898 (+   517us) log.cc:844] Serialized 26 byte log entry
0709 13:08:34.900976 (+381078us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:34.982272 (+ 81296us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:34.982417 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:35.533210 (+550793us) raft_consensus.cc:1685] finished
0709 13:08:35.533269 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:35.533556 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8163,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":8078,"thread_start_us":337658,"threads_started":1,"wal-append.queue_time_us":694684}
I20260709 13:08:35.598546 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1663) took 2184 ms. Trace:
I20260709 13:08:35.615857 25209 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:35.651717 25239 rpcz_store.cc:276] 0709 13:08:32.811391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:32.812644 (+  1253us) service_pool.cc:224] Handling call
0709 13:08:32.923891 (+111247us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:32.924249 (+   358us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:08:32.924274 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:32.935802 (+ 11528us) log.cc:844] Serialized 26 byte log entry
0709 13:08:33.050513 (+114711us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:33.050562 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:33.050635 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:33.239757 (+189122us) raft_consensus.cc:1685] finished
0709 13:08:33.239777 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:35.650544 (+2410767us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b54003ec450. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:35.650954 (+   410us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1404,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":14817,"spinlock_wait_cycles":255036288,"thread_start_us":71859,"threads_started":1,"wal-append.queue_time_us":288815,"wal-append.run_cpu_time_us":1711,"wal-append.run_wall_time_us":1135357}
I20260709 13:08:35.644147 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:35.628970 27874 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:35.619418 25318 rpcz_store.cc:269] 0709 13:08:33.498745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:34.111925 (+613180us) service_pool.cc:224] Handling call
0709 13:08:35.572499 (+1460574us) spinlock_profiling.cc:243] Waited 2.03 ms on lock 0x7b58000e3498. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:35.595645 (+ 23146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4668288}
I20260709 13:08:35.669499 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:35.685604 27812 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:35.689270 25358 connection.cc:256] RPC call timeout handler was delayed by -1.4995486039999832s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23140000000000002s for a response
I20260709 13:08:35.689397 27615 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:08:35.689939 25358 connection.cc:256] RPC call timeout handler was delayed by -1.0792533889999731s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17020000000000002s for a response
I20260709 13:08:35.690465 25389 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:35.692059 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:35.693358 25395 raft_consensus.cc:2376] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6. Re-sending same reply.
W20260709 13:08:35.694330 25387 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1285) took 2201 ms (client timeout 1732 ms). Trace:
I20260709 13:08:35.695062 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:35.697145 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:35.698022 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:35.694880 25387 rpcz_store.cc:269] 0709 13:08:33.492882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.493197 (+   315us) service_pool.cc:224] Handling call
0709 13:08:35.693788 (+2200591us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b5400120cd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:35.694026 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":840690048}
I20260709 13:08:35.698932 25936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 651) took 2361 ms. Trace:
I20260709 13:08:35.688832 27812 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:35.699775 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:35.701504 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:35.702576 25392 raft_consensus.cc:2393] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:35.654983 27813 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:35.703279 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:35.703559 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:35.704519 25387 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:35.705550 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:35.704463 25392 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:35.706658 25391 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:35.690843 27885 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:35.705696 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:35.707837 25387 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:35.708491 25358 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:35.699558 25936 rpcz_store.cc:276] 0709 13:08:33.337252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:35.060388 (+1723136us) service_pool.cc:224] Handling call
0709 13:08:35.595289 (+534901us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:35.625914 (+ 30625us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:35.698639 (+ 72725us) spinlock_profiling.cc:243] Waited 529 ms on lock 0x7b54006d9550. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:35.698842 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1215853440}
I20260709 13:08:35.707584 27885 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.709197 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:35.709677 27812 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 13:08:35.710309 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 138399us
I20260709 13:08:35.706846 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:35.698168 25242 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:35.716868 27908 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 13:08:35.717062 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 146697us
I20260709 13:08:35.717645 27908 raft_consensus.cc:697] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:08:35.718116 25908 leader_election.cc:336] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.167s (SENT)
I20260709 13:08:35.718820 25908 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:35.719055 27908 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:35.720465 25908 leader_election.cc:336] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.236s (SENT)
I20260709 13:08:35.721545 27676 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:35.722462 27676 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:35.724349 25394 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:08:35.702495 25313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1824) took 2213 ms (client timeout 2187 ms). Trace:
I20260709 13:08:35.726028 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1280) took 2236 ms. Trace:
W20260709 13:08:35.725584 25313 rpcz_store.cc:269] 0709 13:08:33.488532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.490778 (+  2246us) service_pool.cc:224] Handling call
0709 13:08:35.702117 (+2211339us) spinlock_profiling.cc:243] Waited 988 ms on lock 0x7b54000870d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:35.702377 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2272215424}
I20260709 13:08:35.727609 25392 raft_consensus.cc:2376] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7. Re-sending same reply.
I20260709 13:08:35.726439 25393 rpcz_store.cc:276] 0709 13:08:33.447957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:33.484681 (+ 36724us) service_pool.cc:224] Handling call
0709 13:08:33.561736 (+ 77055us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:08:34.547232 (+985496us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:34.547274 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:08:34.547930 (+   656us) log.cc:844] Serialized 52 byte log entry
0709 13:08:35.230824 (+682894us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:35.519277 (+288453us) log.cc:844] Serialized 14 byte log entry
0709 13:08:35.519981 (+   704us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:35.669077 (+149096us) spinlock_profiling.cc:243] Waited 985 ms on lock 0x7b5400283cd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:35.669165 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:35.669243 (+    78us) raft_consensus.cc:1685] finished
0709 13:08:35.669333 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:35.684569 (+ 15236us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":315745,"prepare.run_cpu_time_us":130,"prepare.run_wall_time_us":25600,"raft.queue_time_us":19448,"raft.run_cpu_time_us":390,"raft.run_wall_time_us":391,"spinlock_wait_cycles":2265346944,"thread_start_us":952646,"threads_started":2,"wal-append.queue_time_us":718535}
I20260709 13:08:35.725843 25287 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:35.699047 25388 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
W20260709 13:08:35.734635 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 159333us
I20260709 13:08:35.735574 27810 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:35.737459 27810 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 13:08:35.738372 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 56166.8us
I20260709 13:08:35.720932 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:35.734310 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:35.745987 25241 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:35.656185 27642 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:35.750470 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:35.669821 25243 rpcz_store.cc:276] 0709 13:08:33.413825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:34.908800 (+1494975us) service_pool.cc:224] Handling call
0709 13:08:34.973689 (+ 64889us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:34.973992 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:08:35.557101 (+583109us) log.cc:844] Serialized 14 byte log entry
0709 13:08:35.597871 (+ 40770us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:35.597960 (+    89us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:35.598119 (+   159us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:35.598229 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:35.598462 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":443152,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":242,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":247,"thread_start_us":619691,"threads_started":2}
I20260709 13:08:35.754634 27872 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:35.754804 27764 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:35.761584 25212 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:35.762687 27872 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:35.763581 27885 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:35.767840 27798 consensus_meta.cc:287] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.503s	user 0.007s	sys 0.008s
W20260709 13:08:35.769431 27447 consensus_meta.cc:287] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.527s	user 0.014s	sys 0.006s
I20260709 13:08:35.769886 27447 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.769253 27642 consensus_queue.cc:1059] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:08:35.773886 25395 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:35.774062 27720 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:35.775471 27873 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:35.772142 27447 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:35.777962 27881 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:35.666774 27877 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:08:35.778923 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 163400us
I20260709 13:08:35.781555 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:35.780881 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 71896.3us
I20260709 13:08:35.768467 27798 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:35.787055 25287 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:35.790150 25909 connection.cc:256] RPC call timeout handler was delayed by -1.7997799470000473s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.35660000000000003s for a response
I20260709 13:08:35.790920 25319 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:35.781282 27810 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260709 13:08:35.464437 27879 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:35.805612 27873 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:35.811563 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:35.776268 27720 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:35.818795 27813 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:35.820909 27798 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:35.821429 27861 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:35.822062 27861 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:35.823506 27716 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:35.824143 27716 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:35.825440 25209 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:35.826437 27716 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:35.827452 27872 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:35.827858 27872 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:35.828210 27872 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:35.828259 27804 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:35.828749 27804 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:35.829188 27804 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:35.829504 27904 consensus_queue.cc:1059] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:08:35.830052 27764 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:35.830760 27764 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.832686 27875 raft_consensus.cc:3055] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:35.833040 27764 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:35.833076 27875 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:35.835076 27879 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:35.834965 27875 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.836477 27875 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:35.827107 27687 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:35.838066 27687 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.839008 25285 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:35.839643 27716 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:08:35.840672 27764 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:35.847705 27687 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:35.848357 27764 raft_consensus.cc:697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:35.849543 27764 consensus_queue.cc:237] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:35.862679 27642 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:35.863890 25909 leader_election.cc:336] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 1.743s (SENT)
I20260709 13:08:35.863521 27642 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.864421 25909 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:35.865448 27713 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:35.866183 27642 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:35.880262 25241 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:35.882159 27883 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:08:35.884649 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:35.885473 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:35.886480 27824 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:35.882222 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 161875us
I20260709 13:08:35.886973 27824 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:35.866626 27874 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:35.892093 27874 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:35.894127 25209 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:35.897017 27873 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:35.897096 25209 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:35.898602 25909 leader_election.cc:336] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 2.170s (SENT)
I20260709 13:08:35.898393 27875 consensus_queue.cc:1059] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:08:35.898725 27798 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:35.899075 25909 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:35.900068 27642 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:35.900120 27807 raft_consensus.cc:3055] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:35.897801 27873 raft_consensus.cc:697] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:35.900628 27807 raft_consensus.cc:740] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:08:35.900789 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 117640us
I20260709 13:08:35.901777 25287 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:35.901589 27873 consensus_queue.cc:237] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:35.901871 27807 consensus_queue.cc:260] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:35.903272 25287 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:35.903429 27807 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:35.904143 27687 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:35.904919 27687 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:08:35.901849 27669 consensus_meta.cc:287] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.683s	user 0.007s	sys 0.014s
I20260709 13:08:35.905969 27669 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.907312 25244 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:35.914201 27879 consensus_queue.cc:1059] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:08:35.914812 27824 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:35.915465 27879 consensus_queue.cc:1059] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 13:08:35.915944 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 161581us
I20260709 13:08:35.916437 27881 raft_consensus.cc:3055] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:35.916980 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:35.917243 27687 raft_consensus.cc:3055] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:35.917788 27687 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:35.917955 27720 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:08:35.918143 25319 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 1908) took 4358 ms (client timeout 2149 ms). Trace:
W20260709 13:08:35.918767 25319 rpcz_store.cc:269] 0709 13:08:31.559933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:31.665993 (+106060us) service_pool.cc:224] Handling call
0709 13:08:35.917832 (+4251839us) spinlock_profiling.cc:243] Waited 4.09 s on lock 0x7b54000f8a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:35.918024 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9397938944}
I20260709 13:08:35.918782 25358 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:35.916895 27881 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:35.929277 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:35.930169 25945 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:35.929332 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:35.916479 27883 raft_consensus.cc:3055] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:35.935575 27804 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.938588 27804 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:35.939445 27908 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:08:35.940178 25939 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:08:35.941252 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:35.934770 27883 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:35.955569 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:35.918900 27687 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:35.951350 25209 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:35.966991 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 48370.7us
I20260709 13:08:35.951087 27720 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:35.974849 25943 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:35.966295 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:35.989557 25284 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:35.980706 27720 raft_consensus.cc:697] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:08:35.996095 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 43966.9us
I20260709 13:08:35.982048 27874 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:35.998641 27720 consensus_queue.cc:237] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:36.001295 27687 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:36.002274 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:35.994004 27764 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:36.005523 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:36.008769 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 60764.4us
I20260709 13:08:35.994961 25358 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:36.005185 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:36.001873 27907 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260709 13:08:35.984107 27872 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:36.013743 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:36.014631 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:35.965970 27881 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:36.015316 27813 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:35.994325 25941 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:36.019471 25239 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 3: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:36.022945 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.020013 25287 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:36.021641 27881 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:36.024346 25284 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:36.012910 27764 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:36.034601 25395 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:36.036758 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.059149 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.038097 27764 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:36.036135 27669 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:36.072894 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.077095 25241 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:36.077967 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.079003 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:36.080735 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 99580.2us
W20260709 13:08:36.082082 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 65863.2us
I20260709 13:08:36.090384 25240 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:36.093027 25358 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:36.112109 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.113564 25243 raft_consensus.cc:2410] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:08:36.118837 27884 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:36.119266 27884 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:36.119619 27884 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:36.120012 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:36.039980 27883 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:36.122133 27883 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:36.117578 25287 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:36.124939 27720 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:36.134845 25242 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260709 13:08:36.139307 27918 log.cc:927] Time spent T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
W20260709 13:08:36.140769 27916 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 13:08:36.150939 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1433) took 1279 ms. Trace:
W20260709 13:08:36.151022 25128 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f75b60146e7d4b9b9954b4fb2c414679 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 2329e428ebbf484fa3ea9452133df567 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:36.161067 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.162925 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:36.164193 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 76132.9us
W20260709 13:08:36.166966 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 87835.8us
I20260709 13:08:36.151609 25246 rpcz_store.cc:276] 0709 13:08:34.807598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:35.241562 (+433964us) service_pool.cc:224] Handling call
0709 13:08:35.241982 (+   420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:35.788603 (+546621us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:08:35.789109 (+   506us) log.cc:844] Serialized 14 byte log entry
0709 13:08:35.942469 (+153360us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:35.957981 (+ 15512us) log.cc:844] Serialized 26 byte log entry
0709 13:08:35.958190 (+   209us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:35.958236 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:35.968301 (+ 10065us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:36.086852 (+118551us) raft_consensus.cc:1685] finished
0709 13:08:36.086872 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:36.086997 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10921,"prepare.queue_time_us":620,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":63,"raft.queue_time_us":174808,"raft.run_cpu_time_us":336,"raft.run_wall_time_us":17682,"thread_start_us":428173,"threads_started":2,"wal-append.queue_time_us":101567}
I20260709 13:08:36.162138 25239 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
W20260709 13:08:36.188678 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 54117us
W20260709 13:08:36.190584 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 96331.4us
I20260709 13:08:36.196609 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:36.164050 27858 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:36.201713 25358 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:36.203728 27447 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:36.203955 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:36.204612 27447 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:08:36.204859 27885 consensus_queue.cc:1059] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:08:36.205332 27720 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:08:36.206194 27720 raft_consensus.cc:3055] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:36.206558 27720 raft_consensus.cc:740] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:36.207618 27720 consensus_queue.cc:260] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:36.225651 25243 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:36.228634 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:36.229530 25239 raft_consensus.cc:2393] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 7: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:36.232390 27884 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:36.232795 25945 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:36.215903 27872 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:36.251191 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.234141 27764 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:36.236460 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:36.259972 25284 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:36.260356 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:36.261214 27922 log.cc:927] Time spent T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
I20260709 13:08:36.261412 27615 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:36.261554 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.262038 27615 raft_consensus.cc:697] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:08:36.262626 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:36.262516 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.263126 27615 consensus_queue.cc:237] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:36.263903 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:36.264061 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:36.264510 25314 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:36.264572 25320 raft_consensus.cc:2376] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9. Re-sending same reply.
I20260709 13:08:36.265506 25212 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:36.265641 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.266474 25312 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 3: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:36.267158 25212 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:36.268103 27807 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:36.268692 27807 raft_consensus.cc:697] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:36.269585 27872 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:36.271191 25212 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:36.272142 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.272830 25246 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:36.273509 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.273900 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:36.274551 25243 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:36.196416 27908 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:36.198246 27858 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:36.250782 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.257951 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:36.274693 25314 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:36.236639 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1666) took 2792 ms. Trace:
I20260709 13:08:36.277173 25315 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 7: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:08:36.258009 25908 leader_election.cc:336] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.816s (SENT)
W20260709 13:08:36.285187 25908 leader_election.cc:336] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 3.482s (SENT)
I20260709 13:08:36.278151 27884 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:36.285620 25908 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:36.283955 25237 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 3: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:36.287011 25237 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:36.288376 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:36.287578 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:36.289233 25937 raft_consensus.cc:2410] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:08:36.289454 25945 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:36.290380 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:36.291276 25941 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 6: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:36.292102 27908 raft_consensus.cc:697] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:36.253309 27877 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.040s
I20260709 13:08:36.288237 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:36.293088 27908 consensus_queue.cc:237] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:36.293592 25943 raft_consensus.cc:2376] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5. Re-sending same reply.
I20260709 13:08:36.276587 27807 consensus_queue.cc:237] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:36.275375 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.284327 25244 rpcz_store.cc:276] 0709 13:08:33.444765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:35.054905 (+1610140us) service_pool.cc:224] Handling call
0709 13:08:35.055265 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:35.907682 (+852417us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:36.226388 (+318706us) spinlock_profiling.cc:243] Waited 852 ms on lock 0x7b54003c2dd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:36.236585 (+ 10197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1958720256}
I20260709 13:08:36.297833 25209 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:36.282624 25241 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:36.252969 25313 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:36.279771 25285 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:36.304153 25945 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
W20260709 13:08:36.276919 27919 log.cc:927] Time spent T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.192s	user 0.000s	sys 0.001s
W20260709 13:08:36.307689 25358 connection.cc:256] RPC call timeout handler was delayed by -1.1637075999999524s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2033s for a response
I20260709 13:08:36.277812 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.307824 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:36.309442 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.313817 27669 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:36.314281 27669 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:36.314638 27669 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:36.314740 25242 raft_consensus.cc:1240] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:36.315799 27764 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:36.278422 27872 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:36.316282 25359 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:36.316532 27872 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:36.316989 27872 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:36.262528 27720 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:36.319286 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:36.319890 25939 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:36.298882 27881 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:36.323302 25285 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:36.324172 27883 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:36.324249 25285 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:36.324903 27883 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:36.325701 27883 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:36.325909 25244 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 13:08:36.326058 27883 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:36.326287 25244 pending_rounds.cc:85] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 5
I20260709 13:08:36.326413 27883 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:36.326797 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:36.327945 25936 raft_consensus.cc:2393] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 6: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:36.297283 27858 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:36.324312 27669 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:36.333834 27885 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:36.334335 27885 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:36.338166 25321 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:36.338574 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.339859 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.340764 25237 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: []
W20260709 13:08:36.342133 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 83475.2us
I20260709 13:08:36.338179 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:36.344379 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 64510.7us
I20260709 13:08:36.368682 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:36.369570 25317 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 3: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:36.371419 25314 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:36.326577 25244 pending_rounds.cc:107] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 13:08:36.337869 27669 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:36.372759 25244 raft_consensus.cc:2889] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:36.341900 27885 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:36.374579 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:36.399286 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:36.403131 25317 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:36.390230 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:36.415663 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 44707us
I20260709 13:08:36.416740 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.421415 25285 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:36.441314 27713 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:36.460211 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 72786.5us
I20260709 13:08:36.460633 25314 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
W20260709 13:08:36.479995 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 57739.6us
I20260709 13:08:36.441411 25938 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:08:36.483043 25285 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:36.488077 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.495932 25359 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:36.482558 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:36.441782 27878 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20260709 13:08:36.521327 27630 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:36.521938 25313 raft_consensus.cc:2368] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7.
I20260709 13:08:36.529999 25359 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:36.531502 27447 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:36.532238 27447 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:36.532724 27447 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:08:36.537803 27933 log.cc:927] Time spent T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 13:08:36.546629 27813 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:36.479097 27907 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.021s
I20260709 13:08:36.491952 27716 consensus_queue.cc:1059] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20260709 13:08:36.552348 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.591058 25314 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:36.591269 27716 raft_consensus.cc:3055] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:08:36.554251 27931 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.247s	user 0.005s	sys 0.000s
I20260709 13:08:36.591719 27716 raft_consensus.cc:740] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:08:36.554416 27934 log.cc:927] Time spent T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 13:08:36.555261 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:36.593632 25313 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:08:36.561316 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:36.521978 27630 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:36.598834 27630 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:36.563723 27872 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:36.565877 25209 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:36.492635 27764 consensus_queue.cc:1059] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.047s
I20260709 13:08:36.547180 27813 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:36.566722 25939 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:36.571681 27883 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:36.479247 27713 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:36.538623 27669 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 13:08:36.538859 27615 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:36.581768 27861 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:36.586223 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.548678 25317 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:36.584446 27858 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:36.642993 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 113086us
W20260709 13:08:36.647149 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:36.621209 27813 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:36.649536 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:36.650085 27935 log.cc:927] Time spent T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
W20260709 13:08:36.650620 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 84894us
I20260709 13:08:36.651372 27447 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:36.616415 27884 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 13:08:36.592765 27716 consensus_queue.cc:260] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:36.655675 25314 raft_consensus.cc:2376] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6. Re-sending same reply.
I20260709 13:08:36.656311 25318 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 6: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:36.644869 27615 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:36.640563 27877 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:36.647274 27925 raft_consensus.cc:3055] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:08:36.664698 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:36.675400 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:36.681541 25246 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
W20260709 13:08:36.682753 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 79232.1us
I20260709 13:08:36.686054 25284 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:36.667333 27615 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:36.666237 27925 raft_consensus.cc:740] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:36.670662 27861 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:36.690994 25209 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:36.682200 27907 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
W20260709 13:08:36.721200 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:36.702919 27716 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:36.728785 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.730517 27556 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:36.731137 27447 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:36.732589 27713 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:36.740409 25314 raft_consensus.cc:2393] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:36.740852 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.741700 27885 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:36.742484 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.752099 25245 raft_consensus.cc:2393] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:36.752362 25238 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:36.753093 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:36.753665 25245 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:36.754237 25238 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:08:36.737900 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.742781 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.705767 27938 consensus_queue.cc:1059] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 13:08:36.698840 27810 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260709 13:08:36.764849 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.765967 25242 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:08:36.766942 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:36.732952 27642 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:36.757628 25246 raft_consensus.cc:2393] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:36.758533 25241 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:36.746723 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.762341 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.762516 25245 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:36.773083 25319 raft_consensus.cc:2410] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:08:36.776271 25239 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:36.745155 27669 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:36.782220 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:36.788349 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.780916 27669 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:36.787796 27879 raft_consensus.cc:3055] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:36.785921 27872 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:36.814721 27885 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:36.818521 27720 consensus_queue.cc:1059] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 13:08:36.819900 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 447430us
W20260709 13:08:36.820561 25358 leader_election.cc:336] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.789s (SENT)
I20260709 13:08:36.822470 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.824345 27813 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:36.826457 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.826689 27813 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:36.827378 27813 raft_consensus.cc:3055] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:36.827919 27813 raft_consensus.cc:740] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:36.828179 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:36.829020 25241 raft_consensus.cc:2410] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:08:36.790503 27642 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:36.829495 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.831013 27642 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:36.831233 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.831913 25242 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:36.802304 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:36.803790 27929 raft_consensus.cc:3055] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:36.833439 27929 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:36.834097 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.834383 27929 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:36.808383 27879 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:36.836642 27879 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:36.837810 27879 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:36.838884 25313 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:36.840524 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:36.841446 25939 raft_consensus.cc:2393] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:36.782485 27925 consensus_queue.cc:260] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:36.768137 27556 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:36.846153 27925 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:36.847229 27556 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:36.811694 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.812678 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.848707 25314 raft_consensus.cc:2393] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:36.827319 25243 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:36.849043 25140 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53264 (request call id 204) took 10408 ms. Trace:
I20260709 13:08:36.851758 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.852523 25316 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:36.853168 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:36.853509 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.810086 27858 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:36.801384 27883 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:36.759155 27861 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:36.857061 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:36.857439 27861 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:36.858036 25143 catalog_manager.cc:5697] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 2 to 5, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 5 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:08:36.801703 27669 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:36.861413 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:36.861868 25321 raft_consensus.cc:1240] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:36.828888 27813 consensus_queue.cc:260] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:36.861413 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.867061 25314 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:36.868443 27952 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:36.869028 27952 raft_consensus.cc:697] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:36.868947 27929 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:36.870052 27952 consensus_queue.cc:237] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:36.878800 25245 raft_consensus.cc:3055] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:36.879207 25245 raft_consensus.cc:740] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:36.820956 25358 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:36.860597 25143 catalog_manager.cc:5697] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 4 to 6, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 6 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:36.849468 25140 rpcz_store.cc:276] 0709 13:08:26.440428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:26.440770 (+   342us) service_pool.cc:224] Handling call
0709 13:08:36.848841 (+10408071us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:08:32.523453 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:08:32.660585 (+137132us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:08:32.660620 (+    35us) write_op.cc:435] Acquired schema lock
0709 13:08:32.660637 (+    17us) tablet.cc:662] Decoding operations
0709 13:08:32.695845 (+ 35208us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:08:32.695919 (+    74us) write_op.cc:641] Partition lock acquired for write op
0709 13:08:32.695947 (+    28us) tablet.cc:685] Acquiring locks for 6 operations
0709 13:08:32.696507 (+   560us) tablet.cc:701] Row locks acquired
0709 13:08:32.696527 (+    20us) write_op.cc:260] PREPARE: finished
0709 13:08:32.696706 (+   179us) write_op.cc:270] Start()
0709 13:08:32.696818 (+   112us) write_op.cc:276] Timestamp: P: 1783602512696651 usec, L: 0
0709 13:08:32.696840 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 13:08:32.738289 (+ 41449us) log.cc:844] Serialized 2452 byte log entry
0709 13:08:34.670803 (+1932514us) op_driver.cc:464] REPLICATION: finished
0709 13:08:35.390322 (+719519us) write_op.cc:301] APPLY: starting
0709 13:08:35.395755 (+  5433us) tablet.cc:1367] starting BulkCheckPresence
0709 13:08:35.418056 (+ 22301us) tablet.cc:1370] finished BulkCheckPresence
0709 13:08:35.425778 (+  7722us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:08:35.440809 (+ 15031us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:08:35.444783 (+  3974us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0709 13:08:35.444860 (+    77us) write_op.cc:312] APPLY: finished
0709 13:08:35.445183 (+   323us) log.cc:844] Serialized 53 byte log entry
0709 13:08:35.728782 (+283599us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:35.729143 (+   361us) write_op.cc:454] Released schema lock
0709 13:08:35.729282 (+   139us) write_op.cc:341] FINISH: Updating metrics
0709 13:08:35.730038 (+   756us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:35.730077 (+    39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4761,"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":644330,"apply.run_cpu_time_us":8444,"apply.run_wall_time_us":347127,"num_ops":6,"prepare.queue_time_us":1096609,"prepare.run_cpu_time_us":75314,"prepare.run_wall_time_us":475631,"raft.queue_time_us":1248268,"raft.run_cpu_time_us":30043,"raft.run_wall_time_us":811358,"replication_time_us":1883776,"thread_start_us":1626870,"threads_started":5,"wal-append.queue_time_us":521503,"wal-append.run_cpu_time_us":23395,"wal-append.run_wall_time_us":2668416}]]}
I20260709 13:08:36.922228 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.926008 25909 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:36.926703 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:36.928054 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:36.928931 25242 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 6: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:08:36.929800 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 109279us
I20260709 13:08:36.932132 25239 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:36.932875 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.933202 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.934084 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.934968 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:36.939103 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:36.942975 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:36.944308 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:36.947480 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 183384us
I20260709 13:08:36.947716 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:36.941232 25245 consensus_queue.cc:260] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:36.934470 25143 catalog_manager.cc:5697] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:36.953189 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:36.958158 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:36.959476 27950 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.218s	user 0.000s	sys 0.000s
W20260709 13:08:36.961073 27946 log.cc:927] Time spent T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.221s	user 0.000s	sys 0.000s
W20260709 13:08:36.962373 27958 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260709 13:08:36.964470 25320 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:36.967394 27615 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:36.972486 25241 raft_consensus.cc:1275] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:36.978484 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:36.989902 25320 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:36.938496 27904 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:36.985637 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:36.990690 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 43886.7us
I20260709 13:08:36.957505 25359 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:36.992115 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 51329.6us
I20260709 13:08:36.994292 25395 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:37.004619 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:36.931515 27885 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:08:37.006639 25312 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:37.005740 25315 raft_consensus.cc:2393] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:37.006109 25321 raft_consensus.cc:2410] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 13:08:37.008801 25209 connection.cc:256] RPC call timeout handler was delayed by -2.3637287599999581s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15980000000000003s for a response
I20260709 13:08:37.026978 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:37.027329 25209 connection.cc:256] RPC call timeout handler was delayed by -1.5789287600000534s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24700000000000003s for a response
W20260709 13:08:37.028463 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 102531us
I20260709 13:08:37.029441 25245 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:36.945343 27642 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:37.047016 25320 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:36.976742 27669 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:37.048154 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:37.048825 25320 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:37.033360 25143 catalog_manager.cc:5697] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 2 to 5, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 5 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:37.050344 25314 raft_consensus.cc:2410] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:08:37.056492 25909 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:37.064009 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:37.066891 25245 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 13:08:37.067265 25245 pending_rounds.cc:85] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 6
I20260709 13:08:37.067512 25245 pending_rounds.cc:107] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 13:08:37.067808 25245 raft_consensus.cc:2889] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:36.964496 27447 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.091s
I20260709 13:08:37.069500 27447 consensus_queue.cc:1243] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:08:37.069609 27963 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:37.070216 27963 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:37.072419 27963 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:37.032752 27807 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:37.073280 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:37.066000 25143 catalog_manager.cc:5697] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 4 to 6, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 6 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:37.065644 27642 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:37.081897 25395 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 3: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:37.082335 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:37.080561 25143 catalog_manager.cc:5697] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 2 to 3, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:08:37.083113 25314 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 3: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:37.084272 25316 raft_consensus.cc:2410] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 13:08:37.004673 27961 consensus_queue.cc:1059] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.071s
I20260709 13:08:37.082335 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:37.048046 27881 consensus_queue.cc:1059] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 13:08:37.100314 25209 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:37.100725 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:37.102759 25314 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:08:37.103734 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 47336us
I20260709 13:08:37.104629 25909 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:37.106345 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:37.107168 25241 raft_consensus.cc:2393] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:37.035413 27904 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:37.109272 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:37.110565 27904 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:37.111203 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:37.112353 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:37.118314 27962 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:37.119369 27961 raft_consensus.cc:3055] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:37.119827 27961 raft_consensus.cc:740] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:37.120684 27961 consensus_queue.cc:260] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:37.124873 27961 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:37.127296 25209 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:37.124895 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:37.128417 27953 raft_consensus.cc:2764] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:08:37.129424 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:37.130676 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:37.112902 25315 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:37.132290 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:37.075560 27957 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:37.133850 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:37.134691 27615 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:37.135077 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:37.097589 25143 catalog_manager.cc:5697] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 2 to 4, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 4 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:08:37.153285 25243 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:37.161759 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:37.157037 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:37.101404 27556 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:37.081084 27807 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:37.150394 27962 raft_consensus.cc:697] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:37.170863 25143 catalog_manager.cc:5697] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:37.101730 27963 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.069s
I20260709 13:08:37.205801 25143 catalog_manager.cc:5697] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 3 to 4, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 4 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:37.164999 27953 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:37.156590 25320 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:37.218019 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:37.169421 27713 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:37.186542 27957 raft_consensus.cc:3055] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:37.139704 27630 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:37.225775 25143 catalog_manager.cc:5697] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:37.227087 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:37.228199 25242 pending_rounds.cc:85] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 2
I20260709 13:08:37.232242 25242 pending_rounds.cc:107] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 13:08:37.232513 25242 raft_consensus.cc:2889] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:37.234179 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:37.230257 25143 catalog_manager.cc:5697] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:37.235330 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:37.217502 27556 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:37.238759 25313 raft_consensus.cc:2368] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8.
W20260709 13:08:37.230319 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 125784us
I20260709 13:08:37.240648 25909 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:37.224786 27957 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:37.236927 27556 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:37.247375 27948 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.062s
I20260709 13:08:37.248848 27556 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:37.249830 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:37.250545 25386 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:37.251679 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:37.252291 25386 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:37.253330 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:37.253720 27642 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:37.253834 25386 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:37.254662 27972 consensus_queue.cc:1059] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:08:37.255712 27972 raft_consensus.cc:3055] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:37.256053 27972 raft_consensus.cc:740] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:37.256496 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:37.257153 25387 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:37.258080 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:37.258674 25387 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 3: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:37.259366 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:37.259779 25243 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:37.260843 25387 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:08:37.261524 27962 consensus_queue.cc:237] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:37.262288 27872 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:37.264146 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:37.274379 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:37.264997 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:37.265697 27630 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:37.266041 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:37.267712 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:37.276249 27615 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:37.276803 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:37.276886 27615 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:37.279103 27615 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:37.281031 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:37.281632 25239 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 8
I20260709 13:08:37.283823 27713 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:37.285859 27813 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:37.285295 25143 catalog_manager.cc:5697] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 3 to 4, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 4 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:08:37.288312 27669 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:37.288764 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:37.290843 25393 raft_consensus.cc:2393] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 7: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:37.291716 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:37.292446 25393 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:37.292219 27872 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:37.293411 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:37.288993 25319 raft_consensus.cc:2393] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:37.294579 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:37.295696 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:37.295648 27957 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:37.296490 25319 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:37.296831 27957 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:37.296749 27948 consensus_queue.cc:1059] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:08:37.290158 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:37.297717 27948 consensus_queue.cc:1243] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer f646dfd532974dceadfe00005d2bbf59 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:08:37.276659 25395 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:37.298370 27948 raft_consensus.cc:3055] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:37.262298 25386 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:37.299674 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:37.300191 25386 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:37.301748 25390 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:08:37.303032 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:37.309099 25319 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 13:08:37.272069 25320 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:37.293339 25388 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:08:37.298795 27948 raft_consensus.cc:740] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:37.287009 25143 catalog_manager.cc:5697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 1 to 3, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:08:37.316463 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:37.317516 25317 raft_consensus.cc:2393] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 7: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:37.318212 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:37.320062 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 192866us
I20260709 13:08:37.321967 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:37.330308 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:37.331888 25387 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
W20260709 13:08:37.334465 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 64340us
I20260709 13:08:37.323540 25143 catalog_manager.cc:5697] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 3 to 6, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 6 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:08:37.337037 25386 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
W20260709 13:08:37.338505 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 97905.5us
I20260709 13:08:37.330127 25317 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1884b7159da148dbb1c19c424d667792"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:08:37.341622 25317 raft_consensus.cc:606] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:37.341952 25317 raft_consensus.cc:614] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:08:37.342442 27945 raft_consensus.cc:3055] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:37.342823 27945 raft_consensus.cc:740] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:37.343046 25317 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:37.343595 25239 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:37.343678 27945 consensus_queue.cc:260] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:37.353677 25392 raft_consensus.cc:2410] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:08:37.350638 25315 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:37.351490 25285 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:37.355952 27881 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:37.356554 27881 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:37.256958 27972 consensus_queue.cc:260] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:37.359481 27961 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:37.360081 27961 raft_consensus.cc:697] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:37.321009 27556 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:37.344964 27945 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:37.364661 25395 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:37.350175 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:37.362594 25386 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:08:37.362659 25908 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:37.363695 25285 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:08:37.370029 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:37.370844 25315 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 13:08:37.372545 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:37.382129 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:37.382212 25314 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:08:37.383659 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:37.384655 25314 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:37.386186 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:37.386348 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:37.345041 27807 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:37.386906 25393 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:37.390182 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:37.391460 25395 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 3: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:37.387246 25244 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:37.391839 25909 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:37.373447 27615 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:37.394272 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:37.394464 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:37.395067 25386 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:37.365566 27881 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:37.369864 27929 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:37.360946 27630 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260709 13:08:37.406335 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:37.407625 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:37.386425 27812 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:37.409088 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 46505.7us
I20260709 13:08:37.410821 25908 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:37.392745 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:37.413039 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:37.414467 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:37.415088 25395 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:37.416162 25209 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:37.420972 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:37.421386 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:37.421942 25313 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
W20260709 13:08:37.422438 25212 leader_election.cc:336] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.598s (SENT)
I20260709 13:08:37.423126 27960 consensus_queue.cc:1059] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:08:37.423885 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:37.424072 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:37.424202 27713 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:37.424455 25386 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:37.424952 27929 raft_consensus.cc:697] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:37.426321 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:37.426067 27929 consensus_queue.cc:237] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:37.427546 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:37.428623 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:37.424563 25240 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:37.403082 27615 raft_consensus.cc:697] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:08:37.432371 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 54291.7us
I20260709 13:08:37.405423 27642 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:37.435854 25392 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:37.433310 25287 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:37.431242 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:37.437358 25314 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:08:37.438313 25908 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:37.428668 27949 log.cc:927] Time spent T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.636s	user 0.002s	sys 0.000s
I20260709 13:08:37.439363 25386 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:37.436383 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:37.447274 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:37.447762 25313 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:37.451833 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:37.453227 27956 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.093s
I20260709 13:08:37.444207 27881 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:37.454166 27881 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:37.454522 27881 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:37.455049 25313 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
W20260709 13:08:37.443915 27952 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.021s	sys 0.034s
I20260709 13:08:37.446288 27973 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:37.425069 27713 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:37.446805 27877 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:37.458201 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:37.486574 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:37.448856 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:37.433917 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:37.489053 25315 raft_consensus.cc:2393] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:08:37.492311 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 59067.2us
I20260709 13:08:37.495002 25313 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:37.495698 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:37.466352 27447 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:37.494356 27884 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:37.501384 27447 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:37.493321 25287 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:37.505548 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:37.464975 27952 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.021s	sys 0.035s
I20260709 13:08:37.420840 27812 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:37.503955 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 112190us
I20260709 13:08:37.444046 27872 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:37.513149 25386 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:08:37.435484 27948 consensus_queue.cc:260] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:37.515532 25909 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:37.435432 27961 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:37.517777 27976 log.cc:927] Time spent T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 13:08:37.533618 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:37.534428 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:37.466675 27807 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:37.511366 27877 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:37.596889 25386 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 6. Current term is 7. Ops: []
W20260709 13:08:37.602883 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 109651us
I20260709 13:08:37.606204 25394 raft_consensus.cc:2410] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260709 13:08:37.576089 27941 consensus_queue.cc:1175] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46de6919f85341c6a9b574dff112e35a" responder_term: 4 status { last_received { term: 4 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 4 index: 5 } }
I20260709 13:08:37.604676 25395 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:37.535691 27642 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:37.620352 25395 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 13:08:37.553803 27884 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:08:37.625351 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 112850us
W20260709 13:08:37.629668 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 177886us
I20260709 13:08:37.509366 27713 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:37.631453 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 217938us
I20260709 13:08:37.641873 25287 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:37.643113 25387 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:37.648058 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:37.648717 25386 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: []
W20260709 13:08:37.649201 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 200367us
I20260709 13:08:37.613713 27973 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:37.649901 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:37.648772 25237 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:37.653786 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:37.659938 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:37.626453 27948 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:37.616835 27872 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:37.664012 25395 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260709 13:08:37.672618 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:37.664785 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:37.674376 25387 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:37.665833 25209 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:37.675889 25359 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:37.672519 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:37.676945 25386 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:37.682380 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:08:37.684052 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 196761us
I20260709 13:08:37.661459 27877 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:37.686543 25212 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:37.669977 27952 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:37.595356 27615 consensus_queue.cc:237] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:37.676311 27447 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:37.681277 27960 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:37.676970 27630 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:37.723773 25287 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:37.745494 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 63154.7us
I20260709 13:08:37.758478 25287 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:37.691783 25212 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:37.760224 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 69201.2us
I20260709 13:08:37.688776 27877 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:37.761770 25212 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:37.763041 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:37.725276 27960 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:37.772568 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:37.776289 25287 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:37.737557 27989 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:37.789041 25287 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:37.772451 27990 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:37.784186 25945 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:37.756489 27872 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:37.761971 27971 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:08:37.834784 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:37.773607 27960 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:37.755187 27630 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:37.787870 27716 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:37.796176 27925 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:08:37.797545 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:37.864539 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:37.892001 25944 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:37.799998 27971 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:37.768656 27952 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:37.781356 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:37.863758 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:38.154476 25316 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:08:38.102098 27993 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:08:38.153944 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 364979us
I20260709 13:08:38.050954 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:38.067008 27713 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:38.168622 27989 raft_consensus.cc:697] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:38.170056 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:38.172420 25943 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:38.173316 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:38.174482 27989 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:37.969269 27872 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:38.018805 25387 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:37.858892 27956 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:08:38.224274 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 66694.7us
I20260709 13:08:38.225297 25287 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:37.876407 27812 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:38.153545 27977 log.cc:927] Time spent T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.281s	user 0.000s	sys 0.001s
I20260709 13:08:37.894106 27716 raft_consensus.cc:697] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:38.148049 27959 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:37.883680 27642 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:38.271921 25937 raft_consensus.cc:2410] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:08:37.894619 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:38.272363 27925 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:38.280776 25285 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:38.268322 27929 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
W20260709 13:08:38.284061 27982 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.258s	user 0.000s	sys 0.000s
I20260709 13:08:38.147751 27999 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:38.147586 27991 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:08:38.279963 27985 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.272s	user 0.000s	sys 0.000s
I20260709 13:08:38.284967 25241 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 13:08:38.285684 27986 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.246s	user 0.000s	sys 0.000s
I20260709 13:08:38.184234 27952 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:38.274680 27630 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:38.293582 27881 consensus_meta.cc:287] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.839s	user 0.004s	sys 0.008s
W20260709 13:08:38.294379 27957 consensus_meta.cc:287] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.997s	user 0.007s	sys 0.004s
W20260709 13:08:38.297827 27556 consensus_meta.cc:287] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.925s	user 0.011s	sys 0.000s
I20260709 13:08:38.350937 27872 raft_consensus.cc:697] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:38.345979 27959 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:38.390173 27556 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:38.390810 27872 consensus_queue.cc:237] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:38.146847 27987 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:38.309324 27813 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:38.316094 25209 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:38.414233 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 98250.5us
I20260709 13:08:38.317226 27993 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:38.184394 27971 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:37.903115 25945 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:38.329464 25944 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:38.330529 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:38.308924 25909 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:38.347869 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 67200.6us
I20260709 13:08:38.355975 27956 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:08:38.352303 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:38.339316 25908 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.778s	user 0.000s	sys 0.022s
W20260709 13:08:38.355597 25391 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.009s	sys 0.008s
W20260709 13:08:38.368861 27948 consensus_meta.cc:287] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.693s	user 0.009s	sys 0.003s
W20260709 13:08:38.353381 27615 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.012s	sys 0.015s
I20260709 13:08:38.392593 27959 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:38.298709 27953 consensus_meta.cc:287] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.075s	user 0.010s	sys 0.000s
I20260709 13:08:38.387917 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:38.384680 27991 raft_consensus.cc:697] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:38.393968 27881 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:38.428020 25911 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.000s	sys 0.025s
W20260709 13:08:38.395078 27973 consensus_meta.cc:287] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.717s	user 0.008s	sys 0.003s
I20260709 13:08:38.393426 27952 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:38.398278 27716 consensus_queue.cc:237] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:38.404217 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 179054us
I20260709 13:08:38.415813 27997 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:38.421464 27956 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:38.435096 25287 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:38.399061 27998 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.309s	user 0.005s	sys 0.004s
W20260709 13:08:38.425827 27960 consensus_meta.cc:287] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.543s	user 0.007s	sys 0.004s
I20260709 13:08:38.352164 25940 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:38.417836 25391 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.009s	sys 0.008s
I20260709 13:08:38.422590 27813 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:38.340027 27945 consensus_meta.cc:287] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.967s	user 0.007s	sys 0.004s
W20260709 13:08:38.418490 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:38.441319 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:38.421916 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 113076us
I20260709 13:08:38.386685 27999 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:38.358192 27812 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:38.423182 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:38.428443 25911 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.000s	sys 0.025s
I20260709 13:08:38.429200 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:38.429898 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:38.422830 27615 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.013s	sys 0.015s
I20260709 13:08:38.411572 27630 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:38.419292 27669 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:08:38.418061 25908 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.857s	user 0.000s	sys 0.023s
I20260709 13:08:38.427820 28000 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:38.434754 27973 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:38.442473 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:38.427789 27953 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:38.443913 25943 raft_consensus.cc:2368] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6.
I20260709 13:08:38.444195 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1855) took 1461 ms. Trace:
I20260709 13:08:38.461279 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:38.438583 27956 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:38.358978 27642 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:38.509878 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:38.605173 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 144010us
I20260709 13:08:38.447155 25937 raft_consensus.cc:2393] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 6: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:38.439069 27987 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:38.443274 27938 raft_consensus.cc:3055] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:38.425382 27993 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:38.457346 25312 rpcz_store.cc:276] 0709 13:08:36.983272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:37.355182 (+371910us) service_pool.cc:224] Handling call
0709 13:08:37.355570 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:37.365411 (+  9841us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:37.365444 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:37.392670 (+ 27226us) log.cc:844] Serialized 26 byte log entry
0709 13:08:37.520430 (+127760us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:37.520489 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:37.520590 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:38.443974 (+923384us) raft_consensus.cc:1685] finished
0709 13:08:38.443994 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:38.444138 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10236,"raft.queue_time_us":9571,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":5913,"thread_start_us":90922,"threads_started":1,"wal-append.queue_time_us":250300}
W20260709 13:08:38.622323 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 1.18398e+06us
I20260709 13:08:38.439832 27997 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:38.496991 28007 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:38.450973 27669 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:38.675987 27669 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:38.526013 27630 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:38.611403 28003 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:38.677301 27813 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:38.677615 28011 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:38.678112 27630 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:38.657480 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:38.653836 25393 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:38.631604 27881 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:38.679286 27973 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:08:38.680198 28010 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.236s	user 0.000s	sys 0.000s
I20260709 13:08:38.680518 25908 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:38.681607 27973 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:08:38.658519 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 241403us
I20260709 13:08:38.677328 25321 raft_consensus.cc:2410] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 13:08:38.440464 27960 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:38.683040 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 73988.5us
I20260709 13:08:38.684303 25908 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:38.685101 25209 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:38.675061 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 232093us
I20260709 13:08:38.686255 25287 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:38.631822 27615 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:38.627585 27938 raft_consensus.cc:740] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:08:38.705420 25212 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.911s	user 0.008s	sys 0.002s
W20260709 13:08:38.705878 25212 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.912s	user 0.008s	sys 0.002s
I20260709 13:08:38.614301 27929 consensus_queue.cc:1059] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
W20260709 13:08:38.707244 25128 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f75b60146e7d4b9b9954b4fb2c414679 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 2329e428ebbf484fa3ea9452133df567 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:08:38.708804 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:38.451471 27999 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:38.720432 25940 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
W20260709 13:08:38.711025 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 949241us
I20260709 13:08:38.709263 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:38.705427 28003 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:38.707219 27989 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:08:38.714648 28004 log.cc:927] Time spent T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:08:38.714502 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:38.717314 28014 log.cc:927] Time spent T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 13:08:38.723065 25240 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:38.612834 27812 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:38.628450 27991 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:38.744477 25285 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:38.744863 28003 raft_consensus.cc:697] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:38.759045 28003 consensus_queue.cc:237] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:38.738740 28017 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:38.764096 27962 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.425s	user 0.035s	sys 0.017s
W20260709 13:08:38.764567 27962 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.425s	user 0.035s	sys 0.017s
I20260709 13:08:38.632517 27987 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:38.767864 27987 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:38.725669 27669 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:38.768743 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 47834.1us
I20260709 13:08:38.769565 25241 raft_consensus.cc:2410] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:08:38.770123 27961 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:38.770622 27961 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:38.726709 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 1930) took 1368 ms. Trace:
I20260709 13:08:38.771790 25393 rpcz_store.cc:276] 0709 13:08:37.358543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:37.423840 (+ 65297us) service_pool.cc:224] Handling call
0709 13:08:38.726501 (+1302661us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400323cd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:38.726614 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2352819072}
W20260709 13:08:38.773182 25318 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 345) took 1921 ms (client timeout 1897 ms). Trace:
W20260709 13:08:38.773331 25212 connection.cc:256] RPC call timeout handler was delayed by -1.0001869449999958s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2034s for a response
W20260709 13:08:38.773587 25318 rpcz_store.cc:269] 0709 13:08:36.852055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:36.853236 (+  1181us) service_pool.cc:224] Handling call
0709 13:08:38.772960 (+1919724us) spinlock_profiling.cc:243] Waited 760 ms on lock 0x7b540035cad4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:38.773086 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1749084544}
I20260709 13:08:38.773984 27669 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:38.726960 28020 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:08:38.728098 25321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1856) took 1744 ms (client timeout 1594 ms). Trace:
W20260709 13:08:38.774801 25321 rpcz_store.cc:269] 0709 13:08:36.983599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:37.046846 (+ 63247us) service_pool.cc:224] Handling call
0709 13:08:38.330308 (+1283462us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b58000e3498. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:38.727887 (+397579us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b540005e954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:38.728025 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":633945344}
I20260709 13:08:38.775524 28020 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:38.775987 28020 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:38.776340 28020 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:38.782984 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:38.784195 25388 raft_consensus.cc:2410] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 13:08:38.785485 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:38.786145 25392 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:38.786942 27961 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:38.793058 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:38.728793 27960 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:38.688853 28007 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:38.730679 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:38.797510 25944 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
W20260709 13:08:38.731935 27974 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 1.394s	user 0.016s	sys 0.026s
I20260709 13:08:38.733393 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:38.800287 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 653) took 1015 ms. Trace:
I20260709 13:08:38.800830 25941 rpcz_store.cc:276] 0709 13:08:37.784917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:37.797580 (+ 12663us) service_pool.cc:224] Handling call
0709 13:08:38.733256 (+935676us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:38.800207 (+ 66951us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":45677312}
I20260709 13:08:38.801546 27877 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:38.802541 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1698) took 1698 ms. Trace:
I20260709 13:08:38.803076 25242 rpcz_store.cc:276] 0709 13:08:37.103963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:37.199035 (+ 95072us) service_pool.cc:224] Handling call
0709 13:08:37.224192 (+ 25157us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:08:37.263166 (+ 38974us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:37.263196 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:08:37.263754 (+   558us) log.cc:844] Serialized 52 byte log entry
0709 13:08:37.353159 (+ 89405us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:37.353197 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:37.353281 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:38.731205 (+1377924us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:38.802168 (+ 70963us) raft_consensus.cc:1685] finished
0709 13:08:38.802195 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:38.802468 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1175,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":78,"spinlock_wait_cycles":38709632,"thread_start_us":88969,"threads_started":1,"wal-append.queue_time_us":73581}
I20260709 13:08:38.737370 25284 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:38.710421 27957 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:38.741021 25244 raft_consensus.cc:2410] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 13:08:38.743875 25209 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:08:38.807221 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 63442us
I20260709 13:08:38.808055 25209 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:38.737208 25911 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:38.719224 25937 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:08:38.725219 27938 consensus_queue.cc:260] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:38.810199 27884 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:38.810712 27884 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:38.811743 27953 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:38.812364 27953 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:38.813342 27953 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:38.816056 28008 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:38.816705 28008 raft_consensus.cc:697] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:38.817701 28008 consensus_queue.cc:237] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:38.810832 27938 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:38.825524 25209 leader_election.cc:400] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8.
I20260709 13:08:38.826393 25209 leader_election.cc:403] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:08:38.839074 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 102014us
I20260709 13:08:38.839991 27973 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:38.724184 27904 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:38.841704 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 119558us
W20260709 13:08:38.846323 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:38.846889 25908 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:38.844641 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:38.735868 27989 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:38.864472 25940 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 13:08:38.744228 28006 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:08:38.881991 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 144726us
W20260709 13:08:38.884858 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 55055.2us
I20260709 13:08:38.844012 28007 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:38.837662 27807 consensus_queue.cc:1059] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20260709 13:08:38.897886 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:38.865854 28000 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:38.901997 25245 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:38.905635 25908 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:38.906881 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:38.840651 28024 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8.
I20260709 13:08:38.885423 27959 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:38.910462 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:38.916643 25245 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:38.917037 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:38.889604 25209 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:38.919972 25238 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:08:38.848867 27990 consensus_queue.cc:1059] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 13:08:38.921368 25359 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:38.928187 25908 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:38.889546 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 660) took 1009 ms. Trace:
I20260709 13:08:38.921823 27615 raft_consensus.cc:697] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:38.864820 27904 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:38.942678 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:38.945706 25239 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:38.912639 27713 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:38.916113 28029 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:38.922462 27881 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:38.967039 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 1919) took 1595 ms. Trace:
W20260709 13:08:38.955662 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:38.942235 28000 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:38.998332 25245 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:38.952548 27962 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:38.967624 25391 rpcz_store.cc:276] 0709 13:08:37.355407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:37.579714 (+224307us) service_pool.cc:224] Handling call
0709 13:08:37.617548 (+ 37834us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:08:37.629319 (+ 11771us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:37.629366 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:08:37.665582 (+ 36216us) log.cc:844] Serialized 52 byte log entry
0709 13:08:38.436790 (+771208us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:38.616347 (+179557us) log.cc:844] Serialized 14 byte log entry
0709 13:08:38.617121 (+   774us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:38.617459 (+   338us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:38.950828 (+333369us) raft_consensus.cc:1685] finished
0709 13:08:38.950848 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:38.950995 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":275041,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":10573,"raft.queue_time_us":17518,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":42788,"thread_start_us":875210,"threads_started":2,"wal-append.queue_time_us":746821}
I20260709 13:08:38.967576 27881 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:39.116506 27615 consensus_queue.cc:237] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
W20260709 13:08:39.120090 27971 consensus_meta.cc:287] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.695s	user 0.004s	sys 0.003s
I20260709 13:08:38.985608 28038 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:38.991407 27642 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:39.124805 27807 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:08:39.065305 28024 raft_consensus.cc:3055] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:38.936069 25938 rpcz_store.cc:276] 0709 13:08:37.878016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:38.270270 (+392254us) service_pool.cc:224] Handling call
0709 13:08:38.285144 (+ 14874us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:38.285426 (+   282us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:08:38.444074 (+158648us) log.cc:844] Serialized 14 byte log entry
0709 13:08:38.843104 (+399030us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:38.849186 (+  6082us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:38.857825 (+  8639us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:38.865251 (+  7426us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:38.886890 (+ 21639us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":271,"prepare.queue_time_us":388329,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":81187,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":9895,"spinlock_wait_cycles":117691648,"thread_start_us":258900,"threads_started":2,"wal-append.queue_time_us":378699}
I20260709 13:08:39.151544 28038 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:39.156607 27904 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:39.120769 27884 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:39.157876 28038 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:39.158874 27884 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:39.163573 27952 consensus_meta.cc:287] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.724s	user 0.007s	sys 0.000s
I20260709 13:08:39.120533 27971 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:39.150463 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 227662us
I20260709 13:08:39.062280 27962 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:38.989795 28029 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:39.179486 28029 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:39.180271 27962 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:39.181257 28032 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:39.181807 28032 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:39.183996 28032 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:39.165608 27812 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:39.124163 27881 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:39.166405 27999 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:39.146248 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 206056us
I20260709 13:08:39.151911 27807 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:39.186182 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 258061us
W20260709 13:08:39.150782 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 264701us
I20260709 13:08:39.200964 25359 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:39.155254 28024 raft_consensus.cc:740] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:39.209057 25317 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:39.205111 25209 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:39.175781 28036 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:39.213004 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:39.202968 27807 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:08:39.220832 25942 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:39.199488 28032 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:39.215054 25321 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:08:39.227289 25321 raft_consensus.cc:606] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:39.185638 27884 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:08:39.205590 25939 consensus_meta.cc:287] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.528s	user 0.010s	sys 0.000s
I20260709 13:08:39.215140 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:39.222129 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:39.227864 25209 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:39.201781 27812 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:39.224484 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:39.226610 25318 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:08:39.240381 25313 raft_consensus.cc:2393] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 4: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:39.240835 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 658) took 1435 ms. Trace:
I20260709 13:08:39.236893 28029 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:08:39.241346 25942 rpcz_store.cc:276] 0709 13:08:37.786819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:37.787207 (+   388us) service_pool.cc:224] Handling call
0709 13:08:38.735842 (+948635us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:39.222132 (+486290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28693888}
I20260709 13:08:39.237283 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:39.237908 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:39.238260 27971 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:39.225582 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:39.223022 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:39.236083 25359 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:39.185333 27952 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:39.208367 27716 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:39.198693 28020 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:39.252576 25316 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:08:39.439268 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 211457us
I20260709 13:08:39.435415 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:39.250914 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:39.290144 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:39.251442 25312 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 6: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:39.442477 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:39.246798 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:39.443099 25245 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:39.443691 25317 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
W20260709 13:08:39.379889 28052 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
W20260709 13:08:39.436807 28017 consensus_meta.cc:287] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.676s	user 0.011s	sys 0.000s
W20260709 13:08:39.442827 25209 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.68:39211 timed out after 2.132s (SENT) [suppressed 34 similar messages]
I20260709 13:08:39.448726 25314 raft_consensus.cc:2368] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8.
W20260709 13:08:39.380857 28046 log.cc:927] Time spent T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.158s	user 0.000s	sys 0.000s
I20260709 13:08:39.239557 27642 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:39.452729 25320 raft_consensus.cc:2368] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7.
I20260709 13:08:39.239817 27872 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:39.435588 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:39.443104 25393 raft_consensus.cc:1275] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:39.316877 27948 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:39.239701 25939 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:39.316661 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:39.446566 27716 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:39.473886 27716 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:39.474467 25319 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:39.475306 27642 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:39.441505 27991 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:39.476817 27991 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:39.477648 27642 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:39.478987 27991 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:39.471225 27952 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:39.471879 28017 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:39.244485 27884 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:39.483906 27884 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:39.228720 28036 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:39.486445 28036 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:39.244379 27812 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:39.487524 25388 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:39.488646 27812 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:39.497987 25245 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:39.446895 28055 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:08:39.550148 28052 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
W20260709 13:08:39.551312 25939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 649) took 1787 ms (client timeout 1657 ms). Trace:
W20260709 13:08:39.551956 25939 rpcz_store.cc:269] 0709 13:08:37.763960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:37.766658 (+  2698us) service_pool.cc:224] Handling call
0709 13:08:39.551077 (+1784419us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b54001b2bd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:39.551222 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1029233920}
I20260709 13:08:39.553926 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:39.554404 28020 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:39.556252 28007 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:39.469041 27948 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:39.556746 28007 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:39.557446 27948 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:39.559576 27713 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:39.560127 27713 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:39.250916 28032 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:39.562235 28022 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.589s	user 0.000s	sys 0.002s
I20260709 13:08:39.563479 28032 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:39.563603 25909 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:39.565804 28050 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 13:08:39.567338 25936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 532) took 1285 ms. Trace:
I20260709 13:08:39.567873 25936 rpcz_store.cc:276] 0709 13:08:38.282370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:38.718037 (+435667us) service_pool.cc:224] Handling call
0709 13:08:38.718378 (+   341us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:38.721754 (+  3376us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:38.723967 (+  2213us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:38.882815 (+158848us) log.cc:844] Serialized 26 byte log entry
0709 13:08:39.151380 (+268565us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:39.154197 (+  2817us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:39.165470 (+ 11273us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:39.567042 (+401572us) raft_consensus.cc:1685] finished
0709 13:08:39.567068 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:39.567264 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":93,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"thread_start_us":148815,"threads_started":1,"wal-append.queue_time_us":274488}
I20260709 13:08:39.571064 27999 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:08:39.572316 28040 log.cc:927] Time spent T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.363s	user 0.000s	sys 0.000s
I20260709 13:08:39.571863 27999 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:39.574446 28034 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.367s	user 0.001s	sys 0.000s
I20260709 13:08:39.578890 27993 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:39.536700 27991 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:39.600703 25388 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:39.551230 28028 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:39.581758 25936 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:39.588747 27630 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:39.640733 27630 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:39.642851 27630 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:39.617009 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:39.643675 28016 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:39.618477 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:39.644219 28016 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:39.644982 25245 raft_consensus.cc:2368] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 6. Current term is 7.
I20260709 13:08:39.601652 27971 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:39.646359 28016 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:39.647030 27971 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:39.649163 27971 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:39.649822 27881 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:39.652144 27881 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:39.655359 25242 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.712s	user 0.039s	sys 0.067s
W20260709 13:08:39.630434 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:39.590095 28055 raft_consensus.cc:697] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:39.364305 28024 consensus_queue.cc:260] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:39.591691 28032 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:39.593492 28020 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:39.614114 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:39.657512 25242 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.039s	sys 0.068s
I20260709 13:08:39.595134 28011 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:39.619691 27991 raft_consensus.cc:697] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:08:39.655575 27956 consensus_meta.cc:287] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.028s	user 0.011s	sys 0.001s
I20260709 13:08:39.630630 28000 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:08:39.683014 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 69026.7us
I20260709 13:08:39.657323 27872 raft_consensus.cc:697] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:39.710637 28024 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:39.710901 27872 consensus_queue.cc:237] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:39.712559 27962 consensus_meta.cc:287] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.532s	user 0.008s	sys 0.000s
I20260709 13:08:39.709278 28055 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:39.713277 27955 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:39.713967 27955 raft_consensus.cc:697] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 6 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:39.714783 27807 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:39.714952 27955 consensus_queue.cc:237] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:39.673246 28020 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:39.677541 28011 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:39.719713 28020 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:39.720247 28011 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:39.720770 27991 consensus_queue.cc:237] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:39.724274 27956 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:39.655853 28028 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:39.731966 28028 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:39.670528 28032 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:39.735123 28032 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:39.661595 27938 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:39.717027 27807 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:39.674333 27948 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:39.753247 28066 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:08:39.776041 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 90803us
W20260709 13:08:39.827160 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 153314us
I20260709 13:08:39.927647 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:39.928542 25393 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:39.938234 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:39.921707 28005 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 1.236s	user 0.001s	sys 0.003s
I20260709 13:08:39.939743 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:39.940495 25387 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.023s	sys 0.020s
W20260709 13:08:40.011250 25387 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.024s	sys 0.020s
I20260709 13:08:40.011265 25393 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:40.012987 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:40.013990 25387 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:40.015141 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:40.007310 25386 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:08:40.017067 25387 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:40.031212 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:40.032058 25390 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:40.032171 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:40.032213 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 104165us
I20260709 13:08:40.035043 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:40.035666 25387 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:40.049078 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:40.049770 25387 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:39.713016 27962 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:40.031245 27990 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.059s	user 0.027s	sys 0.042s
I20260709 13:08:39.918817 28020 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.125s
I20260709 13:08:40.064847 25389 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260709 13:08:40.061699 27884 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:40.065289 28011 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:40.041620 27953 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:40.072412 25390 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:39.921277 27938 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:40.065811 28060 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:40.075922 28015 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 13:08:40.076287 28068 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.148s
W20260709 13:08:40.062752 27990 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.120s	user 0.027s	sys 0.042s
I20260709 13:08:40.061344 28028 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:40.114425 25393 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:40.117022 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:40.119096 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:40.121038 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:40.124940 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:40.257805 25317 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:40.125264 28030 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:40.125200 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:40.259275 25321 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:40.125954 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:40.125000 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:40.128263 25386 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:08:40.261202 25315 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:40.130465 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:40.130816 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:40.051435 28066 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:40.139890 25391 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:08:40.110411 27807 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:40.083649 27884 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:40.108283 27953 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:40.107815 28011 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:40.127619 28028 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:40.111032 28060 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:40.211527 27956 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:40.217167 27948 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:40.240301 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:40.241722 25909 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.033s	sys 0.052s
W20260709 13:08:40.279843 25909 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.033s	sys 0.052s
W20260709 13:08:40.267674 25938 consensus_meta.cc:287] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.728s	user 0.012s	sys 0.000s
I20260709 13:08:40.125159 28056 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:40.262612 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 132214us
I20260709 13:08:40.264425 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:40.265282 25318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:08:40.265826 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1329) took 1018 ms. Trace:
I20260709 13:08:40.256831 27938 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:40.272090 27807 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:40.276336 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 244634us
W20260709 13:08:40.280180 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 716586us
I20260709 13:08:40.279217 25316 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
W20260709 13:08:40.252944 25942 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.033s	sys 0.047s
W20260709 13:08:40.299944 25942 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.034s	sys 0.048s
I20260709 13:08:40.292330 25287 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:40.301939 25287 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:40.292655 25313 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:40.293066 25318 raft_consensus.cc:606] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:40.293212 25391 rpcz_store.cc:276] 0709 13:08:39.247162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:39.249774 (+  2612us) service_pool.cc:224] Handling call
0709 13:08:40.139191 (+889417us) spinlock_profiling.cc:243] Waited 4.16 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:40.139447 (+   256us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:40.263590 (+124143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:40.265763 (+  2173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9563008}
I20260709 13:08:40.307562 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 670) took 1078 ms. Trace:
I20260709 13:08:40.307891 25942 rpcz_store.cc:276] 0709 13:08:39.229351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:39.454872 (+225521us) service_pool.cc:224] Handling call
0709 13:08:39.455303 (+   431us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:08:39.464358 (+  9055us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:39.464403 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:08:39.472827 (+  8424us) log.cc:844] Serialized 52 byte log entry
0709 13:08:39.629244 (+156417us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:08:40.300360 (+671116us) log.cc:844] Serialized 14 byte log entry
0709 13:08:40.300714 (+   354us) log.cc:844] Serialized 14 byte log entry
0709 13:08:40.307321 (+  6607us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:40.307395 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:40.307422 (+    27us) raft_consensus.cc:1685] finished
0709 13:08:40.307429 (+     7us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:40.307521 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6383,"prepare.queue_time_us":440906,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":3839,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":127,"thread_start_us":747994,"threads_started":2,"wal-append.queue_time_us":262717}
W20260709 13:08:40.266280 28024 consensus_meta.cc:287] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.555s	user 0.009s	sys 0.001s
W20260709 13:08:40.274499 28062 log.cc:927] Time spent T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 13:08:40.308837 25317 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
W20260709 13:08:40.276657 27991 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.023s	sys 0.097s
I20260709 13:08:40.305825 25908 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:40.290980 28072 log.cc:927] Time spent T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
W20260709 13:08:40.297546 28073 log.cc:927] Time spent T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 13:08:40.278206 27962 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:40.272907 28035 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:40.285064 27884 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:40.290870 27953 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:40.286990 28028 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:40.326287 25313 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:40.302084 28068 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:40.291370 25938 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:40.294281 28060 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:40.309803 27990 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:40.314492 27991 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.023s	sys 0.098s
I20260709 13:08:40.309700 28020 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.025s
I20260709 13:08:40.311340 28027 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:40.334949 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:40.272202 28066 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:40.265563 28030 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:40.348097 28066 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:40.349407 28030 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:40.315302 28016 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:40.350131 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1332) took 1102 ms. Trace:
I20260709 13:08:40.350484 28016 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:40.350551 25388 rpcz_store.cc:276] 0709 13:08:39.248149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:39.248293 (+   144us) service_pool.cc:224] Handling call
0709 13:08:40.349944 (+1101651us) spinlock_profiling.cc:243] Waited 46.4 ms on lock 0x7b54003d4ad4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:40.350061 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":106627584}
I20260709 13:08:40.351349 28016 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:40.319002 28035 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:40.351989 28035 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:40.355639 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1705) took 1459 ms. Trace:
I20260709 13:08:40.356055 25246 rpcz_store.cc:276] 0709 13:08:38.896243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:38.897102 (+   859us) service_pool.cc:224] Handling call
0709 13:08:38.897390 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:38.997717 (+100327us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:38.997762 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:38.997832 (+    70us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:38.997896 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:40.355405 (+1357509us) spinlock_profiling.cc:243] Waited 99.9 ms on lock 0x7b5400457254. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:40.355448 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:40.355578 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":178821,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":13764,"spinlock_wait_cycles":229775232}
W20260709 13:08:40.358083 28039 log.cc:927] Time spent T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 1.148s	user 0.002s	sys 0.015s
I20260709 13:08:40.340827 28068 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:40.359427 27999 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:40.291770 28056 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:40.362540 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:40.363116 28056 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:40.363873 25319 raft_consensus.cc:2393] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:40.364312 25358 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:40.339720 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 291939us
I20260709 13:08:40.341073 25942 raft_consensus.cc:2376] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4. Re-sending same reply.
W20260709 13:08:40.365720 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 59971us
I20260709 13:08:40.316394 28070 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:40.342746 28027 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:40.346560 28007 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:40.365154 27938 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:40.328527 27955 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:40.363123 25943 raft_consensus.cc:2376] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6. Re-sending same reply.
I20260709 13:08:40.335944 27807 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:40.374915 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:40.375504 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:40.375818 25241 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:40.376143 25943 raft_consensus.cc:2376] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4. Re-sending same reply.
I20260709 13:08:40.376806 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:40.367852 25908 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:40.369019 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:40.370322 27948 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:40.335048 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:40.327852 28011 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:40.373756 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:40.381481 25246 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:40.377403 25943 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:40.377730 25942 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 7: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:40.379180 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:40.383852 25237 raft_consensus.cc:2410] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:08:40.385908 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:40.392233 25314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1864) took 1168 ms. Trace:
I20260709 13:08:40.392846 25314 rpcz_store.cc:276] 0709 13:08:39.224132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:39.224260 (+   128us) service_pool.cc:224] Handling call
0709 13:08:40.391952 (+1167692us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b540005e954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:40.392129 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":478073600}
I20260709 13:08:40.342051 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 668) took 1113 ms. Trace:
I20260709 13:08:40.343745 27990 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:40.400128 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:40.599834 25284 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:40.401311 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:40.598788 25241 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:40.499399 25246 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:40.416747 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:40.548355 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:40.600876 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 228172us
I20260709 13:08:40.372638 28027 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:40.600330 25393 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:40.603572 25390 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:40.602311 25943 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:40.457194 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:40.567284 25209 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:40.446426 25909 leader_election.cc:400] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 6. Current term is 7.
I20260709 13:08:40.481942 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:40.605472 25909 leader_election.cc:403] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:08:40.606127 25209 leader_election.cc:400] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8.
I20260709 13:08:40.606055 25942 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:40.606557 25209 leader_election.cc:403] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20260709 13:08:40.508899 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:40.457765 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:40.600881 25392 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
W20260709 13:08:40.615926 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 86770.2us
W20260709 13:08:40.599427 25212 leader_election.cc:336] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.660s (SENT)
I20260709 13:08:40.623844 25212 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:40.608006 27807 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:40.638324 25394 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:08:40.641443 27884 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:40.633488 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:40.642014 27884 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:40.658499 25358 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:40.727878 25393 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:40.632050 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1485) took 1814 ms. Trace:
I20260709 13:08:40.401551 25936 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:40.602536 28030 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:40.443152 27955 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:08:40.516885 28079 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.198s	user 0.001s	sys 0.000s
I20260709 13:08:40.615993 28003 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:08:40.742892 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 109515us
I20260709 13:08:40.761816 25287 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:40.757731 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:40.762604 27945 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:40.762928 25937 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:40.763167 27945 raft_consensus.cc:697] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:40.763422 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:40.764194 27945 consensus_queue.cc:237] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:40.743955 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 121875us
I20260709 13:08:40.765972 25359 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:40.766839 25359 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:40.769330 27961 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:40.769502 28055 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:40.769970 27961 raft_consensus.cc:697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:40.770198 28055 raft_consensus.cc:697] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:40.770938 27961 consensus_queue.cc:237] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:40.771190 28055 consensus_queue.cc:237] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:40.772145 28020 consensus_queue.cc:1059] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:08:40.749513 28085 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:40.772991 28085 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:40.773310 28028 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:40.773501 28085 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:40.773931 28028 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:40.774854 28028 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:40.703658 27807 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8.
I20260709 13:08:40.483144 28086 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:40.749655 27872 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:40.743813 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 154856us
W20260709 13:08:40.643738 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 44319.5us
I20260709 13:08:40.602579 27972 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:40.787170 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:40.402680 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:40.812207 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:40.482870 28068 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:40.694306 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:40.382084 27938 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:40.760761 25391 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:40.764773 28066 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:40.564584 28000 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:40.834236 28000 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:40.836699 28000 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:40.757006 27955 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:40.794073 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:40.714887 27989 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:40.783473 25945 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:08:40.783977 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:40.845963 25392 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:40.778420 27997 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:40.848997 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:40.757165 28087 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:40.850271 25391 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 7: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:40.850109 28087 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:40.851393 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:40.626713 27960 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.640s	user 0.012s	sys 0.027s
W20260709 13:08:40.852552 28016 consensus_meta.cc:287] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.500s	user 0.005s	sys 0.005s
W20260709 13:08:40.852753 27960 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.886s	user 0.012s	sys 0.036s
W20260709 13:08:40.853703 28035 consensus_meta.cc:287] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.501s	user 0.009s	sys 0.000s
I20260709 13:08:40.694875 28011 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:40.786820 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:40.854166 28035 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:40.854236 28011 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:40.855516 28011 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:40.856590 28035 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:40.782526 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 124156us
I20260709 13:08:40.857695 25358 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:40.858999 25358 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:40.859921 27948 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:40.860638 27948 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:40.777067 28056 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.067s
I20260709 13:08:40.854349 25389 raft_consensus.cc:2410] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 10 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 13:08:40.863622 27990 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:40.864319 28011 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:40.375566 28070 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:40.817305 25942 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:40.741305 25244 rpcz_store.cc:276] 0709 13:08:38.817837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:38.818069 (+   232us) service_pool.cc:224] Handling call
0709 13:08:38.818536 (+   467us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:38.819018 (+   482us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:08:38.819399 (+   381us) log.cc:844] Serialized 14 byte log entry
0709 13:08:38.961085 (+141686us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:38.961655 (+   570us) log.cc:844] Serialized 26 byte log entry
0709 13:08:38.961826 (+   171us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:38.961868 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:38.961953 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:40.629752 (+1667799us) spinlock_profiling.cc:243] Waited 33.2 ms on lock 0x7b18002c5a20. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:08:40.631756 (+  2004us) raft_consensus.cc:1685] finished
0709 13:08:40.631856 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:40.631991 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":794,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":112831,"raft.run_cpu_time_us":405,"raft.run_wall_time_us":10104,"spinlock_wait_cycles":76420992,"thread_start_us":141173,"threads_started":2,"wal-append.queue_time_us":178785}
W20260709 13:08:40.824988 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:40.885461 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:40.607215 25938 rpcz_store.cc:276] 0709 13:08:39.228626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:39.239285 (+ 10659us) service_pool.cc:224] Handling call
0709 13:08:40.339809 (+1100524us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:08:40.663981 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 64218.8us
I20260709 13:08:40.687291 28045 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:40.670118 25939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 671) took 1385 ms. Trace:
W20260709 13:08:40.840057 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 75688.7us
I20260709 13:08:40.749977 27669 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:40.842252 25394 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:40.797571 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:40.757956 28030 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:40.893869 28011 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:40.895125 28030 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:40.812799 25940 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:08:40.897665 27938 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:40.895224 25939 rpcz_store.cc:276] 0709 13:08:39.229676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:39.586367 (+356691us) service_pool.cc:224] Handling call
0709 13:08:39.640754 (+ 54387us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:08:39.641726 (+   972us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:39.641816 (+    90us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:08:39.643001 (+  1185us) log.cc:844] Serialized 52 byte log entry
0709 13:08:39.676664 (+ 33663us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:08:40.318351 (+641687us) log.cc:844] Serialized 14 byte log entry
0709 13:08:40.366809 (+ 48458us) log.cc:844] Serialized 14 byte log entry
0709 13:08:40.373579 (+  6770us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:40.385922 (+ 12343us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:40.395121 (+  9199us) raft_consensus.cc:1685] finished
0709 13:08:40.395137 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:40.614959 (+219822us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":359661,"prepare.queue_time_us":442074,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":8045,"raft.queue_time_us":41972,"raft.run_cpu_time_us":21912,"raft.run_wall_time_us":562796,"spinlock_wait_cycles":19533696,"thread_start_us":422459,"threads_started":2,"wal-append.queue_time_us":74144}
I20260709 13:08:40.895268 27884 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:40.891846 25287 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:40.899633 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:40.784240 28086 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:40.853040 28016 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:40.903053 28086 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:40.903234 28016 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:40.906687 25320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 1956) took 1697 ms (client timeout 2149 ms). Trace:
I20260709 13:08:40.907481 28009 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:40.907793 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:40.908176 28009 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:40.908622 28020 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 13:08:40.907166 25320 rpcz_store.cc:269] 0709 13:08:39.209387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:39.209654 (+   267us) service_pool.cc:224] Handling call
0709 13:08:40.906463 (+1696809us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b54000f8a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:40.906619 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":481057152}
I20260709 13:08:40.910669 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:40.848917 28043 raft_consensus.cc:3055] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:40.913066 28043 raft_consensus.cc:740] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:40.913482 25318 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 13:08:40.917169 25318 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 13:08:40.866066 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:40.918857 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:40.919473 25318 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:40.919523 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:40.921469 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:40.922068 25318 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:40.811856 25943 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:40.788034 28003 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:40.878659 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:40.933265 25241 raft_consensus.cc:2393] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 10: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:40.785609 27872 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:40.843384 27955 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:40.923524 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:40.914045 28043 consensus_queue.cc:260] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:40.924456 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:40.925086 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:40.925235 27713 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:40.926082 25321 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:08:40.926777 28070 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:40.927395 25287 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:40.928030 28030 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:40.852557 28087 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:40.924858 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:40.936026 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:40.884742 27990 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:40.946239 25395 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:41.232065 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:41.233363 25395 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:41.234118 25315 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:08:41.081480 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:41.235366 25392 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:40.940954 28003 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:40.940663 27669 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:41.239343 28030 raft_consensus.cc:697] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:08:41.209303 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:41.241353 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 313990us
I20260709 13:08:41.241531 27884 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:41.242368 27884 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:41.230938 25312 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
W20260709 13:08:41.232093 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:41.242914 27884 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:41.003671 27941 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:41.243340 28016 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:08:41.243979 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:41.019416 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:41.245615 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 383356us
I20260709 13:08:41.245887 25391 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
W20260709 13:08:41.007097 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 45722us
I20260709 13:08:41.232895 25321 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 13:08:41.248116 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:08:41.250761 25128 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f75b60146e7d4b9b9954b4fb2c414679 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 2329e428ebbf484fa3ea9452133df567 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:41.128189 28085 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:41.252343 25395 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:41.257225 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:41.257736 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1342) took 1325 ms. Trace:
I20260709 13:08:41.244912 25287 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:41.264678 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:41.258386 25386 rpcz_store.cc:276] 0709 13:08:39.932308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:40.016773 (+ 84465us) service_pool.cc:224] Handling call
0709 13:08:40.049750 (+ 32977us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:40.261078 (+211328us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:41.249825 (+988747us) spinlock_profiling.cc:243] Waited 78.2 ms on lock 0x7b54001ec654. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:41.257667 (+  7842us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":181742080}
I20260709 13:08:41.249344 25209 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:41.261482 25315 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:08:41.265743 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:41.271023 28070 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:41.276072 28070 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:41.276582 28070 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:41.283004 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:41.283656 25392 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:08:41.284183 25320 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:41.284991 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:41.056604 28068 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.214s
W20260709 13:08:41.286324 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 364461us
I20260709 13:08:41.070549 28066 raft_consensus.cc:697] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:41.293226 27993 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:08:41.293905 25940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 538) took 2564 ms. Trace:
I20260709 13:08:41.017877 28045 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.285s
I20260709 13:08:41.294272 25940 rpcz_store.cc:276] 0709 13:08:38.729816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:38.881177 (+151361us) service_pool.cc:224] Handling call
0709 13:08:38.891566 (+ 10389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:40.895844 (+2004278us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:41.293649 (+397805us) spinlock_profiling.cc:243] Waited 1.89 s on lock 0x7b5400460fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:41.293849 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4355990400}
I20260709 13:08:41.295653 27872 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:41.240504 28030 consensus_queue.cc:237] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:41.296204 27872 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:41.296671 27872 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:41.298699 27872 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:41.298930 28085 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:41.264472 27990 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:41.301291 25312 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
W20260709 13:08:41.301910 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:41.270663 28094 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:41.273311 27987 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:41.332047 28066 consensus_queue.cc:237] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:41.287393 27925 raft_consensus.cc:3055] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:41.351006 27925 raft_consensus.cc:740] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:41.313900 27938 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:08:41.373821 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:41.371938 27925 consensus_queue.cc:260] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:41.383363 25312 raft_consensus.cc:606] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:41.383888 25312 raft_consensus.cc:614] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:08:41.355123 27959 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260709 13:08:41.336659 27987 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:08:41.388809 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 123188us
I20260709 13:08:41.390025 27960 consensus_queue.cc:1059] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:08:41.391533 27962 raft_consensus.cc:3055] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:41.392038 27962 raft_consensus.cc:740] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:41.393188 27962 consensus_queue.cc:260] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:41.355768 28094 raft_consensus.cc:697] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:41.397876 25388 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:41.402546 28056 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:41.405017 28027 consensus_meta.cc:287] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.709s	user 0.007s	sys 0.002s
I20260709 13:08:41.408946 25358 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:41.413683 25319 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: []
W20260709 13:08:41.419824 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:41.395601 27987 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:41.424157 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:41.425817 25312 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:08:41.426406 25312 raft_consensus.cc:606] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:41.426744 25312 raft_consensus.cc:614] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:08:41.427903 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:41.436240 25241 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:41.438176 28045 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:41.442744 28045 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:41.422830 27938 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:41.445151 28045 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:41.447110 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:41.448391 28009 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:41.427670 28035 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 13:08:41.417966 28056 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:41.458904 28025 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 2.512s	user 0.062s	sys 0.062s
I20260709 13:08:41.459823 28056 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:41.438024 28094 consensus_queue.cc:237] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:41.462378 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:41.463053 25390 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:41.463434 27990 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:41.467317 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:41.467659 27884 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:41.434362 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:41.440652 25244 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 13:08:41.468012 25391 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:41.441995 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:41.419689 28027 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:41.471140 25244 pending_rounds.cc:85] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 3
I20260709 13:08:41.489552 25244 pending_rounds.cc:107] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 13:08:41.489486 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:41.489935 25244 raft_consensus.cc:2889] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:41.494160 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1351) took 1093 ms. Trace:
I20260709 13:08:41.494643 25389 rpcz_store.cc:276] 0709 13:08:40.400386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:40.402322 (+  1936us) service_pool.cc:224] Handling call
0709 13:08:41.493947 (+1091625us) spinlock_profiling.cc:243] Waited 39.3 ms on lock 0x7b54003e3d54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:41.494098 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":90363648}
I20260709 13:08:41.497538 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 370) took 1380 ms. Trace:
I20260709 13:08:41.497943 25316 rpcz_store.cc:276] 0709 13:08:40.117198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:40.117350 (+   152us) service_pool.cc:224] Handling call
0709 13:08:41.497377 (+1380027us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b5400275e54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:41.497484 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":48504704}
I20260709 13:08:41.501955 28043 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:41.504030 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:41.504448 28070 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:41.504910 25389 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:41.505821 25239 raft_consensus.cc:2393] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:41.513706 25245 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 13:08:41.516301 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 1.06998e+06us
I20260709 13:08:41.516544 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:08:41.517316 28069 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 7
I20260709 13:08:41.521914 28088 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:41.522352 28089 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:41.522517 28088 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:41.522876 28089 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:41.525817 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 739310us
I20260709 13:08:41.526914 25908 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:41.528827 27999 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:41.529583 25944 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 672) took 2299 ms. Trace:
I20260709 13:08:41.529748 27999 raft_consensus.cc:697] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:41.530828 27999 consensus_queue.cc:237] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:41.530004 25944 rpcz_store.cc:276] 0709 13:08:39.229997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:39.557767 (+327770us) service_pool.cc:224] Handling call
0709 13:08:39.558044 (+   277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:39.558235 (+   191us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:08:39.680966 (+122731us) log.cc:844] Serialized 14 byte log entry
0709 13:08:40.928588 (+1247622us) spinlock_profiling.cc:243] Waited 2.84 ms on lock 0x7b18002f4a00. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f297685a4bc 00007f2973ba848b 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f297685223a
0709 13:08:41.529204 (+600616us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:41.529261 (+    57us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:08:41.529313 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:41.529385 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:41.529525 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":600339,"prepare.queue_time_us":371893,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":600496,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"spinlock_wait_cycles":6522240,"thread_start_us":456826,"threads_started":2,"wal-append.queue_time_us":573141,"wal-append.run_cpu_time_us":1370,"wal-append.run_wall_time_us":1091843}
I20260709 13:08:41.535902 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:41.485167 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:41.537499 25911 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:41.542618 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:41.543499 25943 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:41.544889 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:41.545496 25943 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:41.546761 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:41.486708 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:41.548207 25390 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:41.552703 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:41.469965 27884 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:41.556246 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:41.550067 28088 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:41.557842 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:41.558480 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:41.566498 25942 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:41.561028 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:41.567310 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:41.567696 25315 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:41.561302 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:41.562887 25944 raft_consensus.cc:2410] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 8 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:08:41.565627 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:41.561964 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:41.569940 25319 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
W20260709 13:08:41.571246 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 302083us
W20260709 13:08:41.571606 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 146187us
I20260709 13:08:41.572940 25358 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:41.573310 25209 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:41.575560 27990 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:08:41.580951 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 160119us
I20260709 13:08:41.583465 25944 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:41.585412 25388 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:41.500968 27987 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:41.590479 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:41.587455 25392 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:08:41.591679 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:41.592317 25392 raft_consensus.cc:2376] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7. Re-sending same reply.
I20260709 13:08:41.592633 27941 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:41.596819 25316 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:41.470098 27669 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:41.528852 27973 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:41.576117 27990 raft_consensus.cc:697] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:41.542244 28089 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:41.479633 28027 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:41.551491 27997 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:41.604013 25313 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 13:08:41.609887 25358 leader_election.cc:400] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7.
W20260709 13:08:41.621411 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:41.558902 28087 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:41.625948 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:41.670801 25911 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:41.626811 25908 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:41.604045 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:41.488559 28003 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:41.661306 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:41.661667 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:41.662621 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:41.678459 25391 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:41.565770 28044 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:41.662631 25358 leader_election.cc:403] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 13:08:41.486169 27955 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:41.558847 28106 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
I20260709 13:08:41.551715 28000 consensus_queue.cc:1059] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.078s
I20260709 13:08:41.599346 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:41.672673 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:41.694262 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 84566.9us
I20260709 13:08:41.661190 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:41.696390 25942 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:41.674602 25911 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:41.674750 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:41.698263 25239 raft_consensus.cc:2393] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:41.675388 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
W20260709 13:08:41.675491 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 48774us
I20260709 13:08:41.699767 25908 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:41.675410 25944 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 13:08:41.551129 28056 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.079s
I20260709 13:08:41.677947 25245 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:41.624073 27973 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:41.683230 28007 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:41.656477 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 670) took 1320 ms. Trace:
I20260709 13:08:41.657152 28115 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:41.707692 25938 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
W20260709 13:08:41.688428 28109 log.cc:927] Time spent T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 13:08:41.716361 28115 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:41.552011 28095 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.110s
I20260709 13:08:41.717969 28086 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:41.718711 28115 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:41.707415 27955 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:41.714116 25937 rpcz_store.cc:276] 0709 13:08:40.309485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:40.763658 (+454173us) service_pool.cc:224] Handling call
0709 13:08:40.764023 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:41.525390 (+761367us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:41.525426 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:41.526055 (+   629us) log.cc:844] Serialized 26 byte log entry
0709 13:08:41.574444 (+ 48389us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:41.574494 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:41.574571 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:41.587283 (+ 12712us) raft_consensus.cc:1685] finished
0709 13:08:41.587318 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:41.629553 (+ 42235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":760894,"raft.queue_time_us":761090,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":82,"thread_start_us":47955,"threads_started":1,"wal-append.queue_time_us":48309}
I20260709 13:08:41.657346 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:41.691486 28087 raft_consensus.cc:697] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:41.694370 25312 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:41.673036 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:41.703179 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:41.706559 25246 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 13:08:41.683487 27713 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:41.700372 25944 pending_rounds.cc:85] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 6
W20260709 13:08:41.683368 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 101062us
I20260709 13:08:41.708760 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:41.736997 25945 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:41.737325 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:41.709931 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:41.714740 25209 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:41.695276 28094 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:41.711980 27973 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:41.704562 27938 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7.
I20260709 13:08:41.739096 25209 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:41.692812 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:41.714628 28007 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:41.774181 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:41.775358 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:41.750455 25942 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:41.715755 28107 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:41.750187 28094 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:41.758011 25358 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:41.758929 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:41.789765 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:41.735630 25944 pending_rounds.cc:107] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 6.7
I20260709 13:08:41.790576 25944 raft_consensus.cc:2917] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Aborting config change with OpId 6.7: Aborted: Op aborted by new leader
I20260709 13:08:41.742456 27713 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:41.741175 27990 consensus_queue.cc:237] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:41.765764 28087 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:41.750263 25936 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:41.770452 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:41.735479 28000 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:41.785879 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:41.855803 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 97859.9us
I20260709 13:08:41.787504 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:41.746789 28029 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:41.790967 25944 raft_consensus.cc:2889] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 7 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:41.794837 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:41.801565 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:41.805511 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:41.817062 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:41.841444 27973 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:41.846024 28003 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:41.859941 25285 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:41.785429 28007 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:41.846998 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:41.837553 25242 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:41.849903 28070 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:41.851492 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1719) took 1061 ms. Trace:
I20260709 13:08:41.870390 25241 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:41.870548 25246 rpcz_store.cc:276] 0709 13:08:40.790239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:40.790369 (+   130us) service_pool.cc:224] Handling call
0709 13:08:40.790728 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:41.734538 (+943810us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:41.851232 (+116694us) spinlock_profiling.cc:243] Waited 916 ms on lock 0x7b5400457254. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:41.851431 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2105648512}
I20260709 13:08:41.785049 28089 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:41.847878 27960 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:41.785157 28108 raft_consensus.cc:2764] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 13:08:41.853193 25237 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:08:41.854897 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:41.856600 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:41.806525 27713 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:41.788604 28107 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:41.859512 28000 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:41.862828 28029 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:41.906553 25212 leader_election.cc:336] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.506s (SENT)
W20260709 13:08:41.878542 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 178821us
I20260709 13:08:41.806747 28045 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20260709 13:08:41.896088 25358 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:41.882201 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 168273us
I20260709 13:08:41.886576 27962 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:41.889725 25241 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:41.864174 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:41.876742 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:41.847678 27872 consensus_queue.cc:1059] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.029s
I20260709 13:08:41.909258 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:41.879640 28063 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:41.878824 28044 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:41.925154 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:41.916234 25285 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:41.925853 25386 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:41.923842 25241 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:08:41.906682 28000 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:41.911118 28029 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:41.905807 27960 raft_consensus.cc:697] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:41.910254 27945 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:08:41.906790 28107 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:41.918924 27941 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:08:41.915285 28069 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 6. Current term is 7.
I20260709 13:08:41.910709 28016 consensus_queue.cc:1059] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 13:08:41.916975 27987 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:41.928002 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:41.949302 25388 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:41.936895 25943 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:41.951485 28011 raft_consensus.cc:993] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: : Instructing follower 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:08:41.951938 28011 raft_consensus.cc:1081] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Signalling peer 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:08:41.952212 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:41.953465 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:41.953992 25937 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:41.954118 28016 raft_consensus.cc:3055] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:41.954545 28016 raft_consensus.cc:740] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:41.955255 28016 consensus_queue.cc:260] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:41.956454 28016 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:41.939162 27872 consensus_queue.cc:1243] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer 2329e428ebbf484fa3ea9452133df567 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:08:41.961331 27960 consensus_queue.cc:237] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:41.927707 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:41.936961 25939 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 7: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:41.946237 25285 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:41.977774 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1870) took 1070 ms. Trace:
I20260709 13:08:41.978647 25285 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:41.978379 25313 rpcz_store.cc:276] 0709 13:08:40.907482 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:40.909232 (+  1750us) service_pool.cc:224] Handling call
0709 13:08:40.909564 (+   332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:41.662534 (+752970us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:41.977385 (+314851us) spinlock_profiling.cc:243] Waited 694 ms on lock 0x7b54000f8a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:41.977680 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1596432128}
I20260709 13:08:41.879732 27948 consensus_queue.cc:1059] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20260709 13:08:41.980455 27948 raft_consensus.cc:3055] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:41.980813 27948 raft_consensus.cc:740] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:41.981652 27948 consensus_queue.cc:260] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:41.982817 27948 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:41.910440 27956 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260709 13:08:41.985327 27884 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:41.986043 27884 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:08:41.986537 27884 raft_consensus.cc:697] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:41.987421 27884 consensus_queue.cc:237] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:41.918747 27999 consensus_queue.cc:1059] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260709 13:08:41.937178 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:08:41.974711 28105 log.cc:927] Time spent T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.564s	user 0.000s	sys 0.002s
I20260709 13:08:41.983946 27997 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:41.921936 28030 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.026s
I20260709 13:08:41.946414 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:42.016273 25942 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:42.018263 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:41.879176 28089 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:42.029268 25937 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 13:08:42.029479 28094 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:42.031526 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1883) took 1108 ms. Trace:
I20260709 13:08:42.031982 25321 rpcz_store.cc:276] 0709 13:08:40.923315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:41.231916 (+308601us) service_pool.cc:224] Handling call
0709 13:08:41.232478 (+   562us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:41.258505 (+ 26027us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:42.031196 (+772691us) spinlock_profiling.cc:243] Waited 5.75 ms on lock 0x7b540021faa0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976c29dc0 00007f2976cb5a02 00007f2976cb1dd6 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2
0709 13:08:42.031464 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13233536}
I20260709 13:08:42.033802 25209 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:41.996742 25386 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:41.942099 28063 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:08:42.001827 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 74203.3us
I20260709 13:08:42.005275 25242 raft_consensus.cc:3055] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:42.040850 25242 raft_consensus.cc:740] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:42.037599 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:42.041561 25313 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:42.007148 25239 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:08:42.009013 25939 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:08:42.036882 25287 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:42.039671 27813 raft_consensus.cc:2764] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260709 13:08:42.008519 28007 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:42.044191 25358 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:42.052625 25313 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
W20260709 13:08:42.087564 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 50648.3us
W20260709 13:08:42.090749 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 53979.6us
W20260709 13:08:42.160426 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 116294us
I20260709 13:08:42.161000 25287 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:42.162549 25358 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:42.160140 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:42.163522 25287 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:42.164177 25937 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 7: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:42.041630 25242 consensus_queue.cc:260] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:42.165273 25242 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:42.172384 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:42.180320 25242 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:08:42.182284 25242 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:42.050221 28085 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:42.157223 27987 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:42.160977 28016 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:42.174340 28027 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:42.171705 28011 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:08:42.176393 27938 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:08:42.212328 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 49865.4us
W20260709 13:08:42.226214 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:42.181288 27945 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:42.160868 27872 consensus_queue.cc:1059] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20260709 13:08:42.226117 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:42.217154 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:42.195060 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:42.229705 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:42.199193 28124 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.348s	user 0.001s	sys 0.000s
I20260709 13:08:42.195977 28063 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:42.015867 28029 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:42.202621 27987 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:42.223651 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:42.181818 27941 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.044s
I20260709 13:08:42.233319 25937 raft_consensus.cc:2376] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5. Re-sending same reply.
I20260709 13:08:42.210152 28016 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:42.214964 28027 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:42.243475 25237 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:42.183518 27713 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:42.197009 28085 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:42.251992 25239 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:08:42.230456 27973 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:42.252923 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:08:42.251296 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 78990.3us
I20260709 13:08:42.209429 28107 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:42.309295 25941 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:42.310690 28107 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:42.312589 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:42.316745 25941 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:42.317862 25244 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:42.251132 28016 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:08:42.323582 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 97529.2us
I20260709 13:08:42.344739 25358 leader_election.cc:400] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7.
I20260709 13:08:42.345064 25358 leader_election.cc:403] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:08:42.253315 28027 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:42.351924 25908 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:42.246943 25936 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:42.353734 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:42.354508 25313 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 7: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:42.258090 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:42.437332 27713 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:42.438006 28086 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:42.438542 28086 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:08:42.440376 28088 raft_consensus.cc:670] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
W20260709 13:08:42.246744 28043 consensus_meta.cc:287] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.744s	user 0.008s	sys 0.003s
I20260709 13:08:42.327054 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:42.443020 28070 raft_consensus.cc:3055] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Stepping down as leader of term 4
W20260709 13:08:42.443104 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 116130us
I20260709 13:08:42.443517 28070 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:42.444093 25909 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:42.445494 25243 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 13:08:42.445891 25243 pending_rounds.cc:85] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 6
I20260709 13:08:42.446116 25243 pending_rounds.cc:107] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 6.7
I20260709 13:08:42.446396 25243 raft_consensus.cc:2917] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Aborting config change with OpId 6.7: Aborted: Op aborted by new leader
I20260709 13:08:42.446781 25243 raft_consensus.cc:2889] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:42.446882 27997 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:42.334445 28009 raft_consensus.cc:3055] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 13:08:42.336596 28129 log.cc:927] Time spent T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.299s	user 0.000s	sys 0.001s
I20260709 13:08:42.449309 28009 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:42.249264 27987 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:42.451620 27925 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:42.324872 28133 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
I20260709 13:08:42.324160 28011 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:08:42.456799 28011 raft_consensus.cc:697] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:42.457770 28011 consensus_queue.cc:237] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:42.317322 28063 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:42.461755 28063 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:42.248870 27938 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:42.466693 27938 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:42.467173 27938 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:42.403128 27973 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:42.406673 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:42.406725 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:42.473806 25313 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:42.472602 25314 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 13:08:42.434146 28131 log.cc:927] Time spent T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 13:08:42.403108 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 58493.8us
I20260709 13:08:42.434803 28029 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:42.404085 27948 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7.
W20260709 13:08:42.471194 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 246163us
I20260709 13:08:42.486094 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:42.487212 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:42.488644 25209 leader_election.cc:336] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Timed out: RequestConsensusVote RPC to 127.23.140.68:39211 timed out after 1.536s (SENT)
I20260709 13:08:42.489246 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:42.503011 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:42.503413 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:42.505085 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:42.505875 25317 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:42.481827 25316 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:42.482342 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:42.484114 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:42.480283 25908 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:42.507952 25314 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 7: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:08:42.517639 25358 connection.cc:256] RPC call timeout handler was delayed by -3.373202042999992s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18430000000000002s for a response
W20260709 13:08:42.518307 25358 connection.cc:256] RPC call timeout handler was delayed by -3.0609020429999418s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.219s for a response
I20260709 13:08:42.519091 25358 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:42.521675 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:42.523885 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 43690.7us
I20260709 13:08:42.527050 25909 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:42.471629 27713 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:42.586838 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:42.484752 28089 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:42.592558 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:42.444469 28070 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:42.600973 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 84305.1us
I20260709 13:08:42.617909 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:42.618716 25316 raft_consensus.cc:2393] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 10: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:42.624130 25317 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:42.546029 27945 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:42.586674 27872 consensus_queue.cc:1243] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer 2329e428ebbf484fa3ea9452133df567 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 13:08:42.545724 27993 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:42.546082 28029 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:42.628314 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:42.522714 27973 raft_consensus.cc:697] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:42.514848 28115 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:42.522491 28088 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:42.641206 25320 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:42.644265 25315 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:08:42.546355 28055 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:42.598685 27955 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:08:42.625429 28028 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.208s	user 0.003s	sys 0.025s
I20260709 13:08:42.684616 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:42.450328 28009 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:42.628925 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:08:42.699316 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 152487us
I20260709 13:08:42.598873 27972 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.078s
I20260709 13:08:42.701876 25358 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:42.654870 28070 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:42.706678 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:42.619642 28089 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:42.677716 28085 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:42.689438 28028 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.275s	user 0.004s	sys 0.025s
I20260709 13:08:42.676810 27872 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:42.736260 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:42.654099 27945 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:42.738034 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:42.738839 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:42.739269 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:42.684252 27955 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:42.705694 27990 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.674s	user 0.026s	sys 0.049s
W20260709 13:08:42.745406 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 55352.8us
I20260709 13:08:42.761016 25209 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:42.746448 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:42.679992 25936 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 13:08:42.736733 25386 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
W20260709 13:08:42.762615 27990 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.731s	user 0.027s	sys 0.050s
I20260709 13:08:42.763269 27990 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:42.763731 27990 raft_consensus.cc:697] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:42.764539 27990 consensus_queue.cc:237] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:42.769274 27973 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:42.770689 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:42.771884 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:42.687330 28088 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:42.714943 27941 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:42.752717 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:42.752945 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:42.780047 25387 raft_consensus.cc:2368] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8.
I20260709 13:08:42.780953 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:42.753196 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:42.753779 25390 raft_consensus.cc:2368] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 3. Current term is 4.
W20260709 13:08:42.752099 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 225129us
I20260709 13:08:42.758937 25392 raft_consensus.cc:2410] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:08:42.783895 25909 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:42.784005 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:42.784169 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:42.785451 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:42.786289 25392 raft_consensus.cc:2368] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5.
I20260709 13:08:42.787075 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:42.787681 25392 raft_consensus.cc:2376] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5. Re-sending same reply.
I20260709 13:08:42.679280 28115 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:42.757290 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:08:42.742992 28009 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:42.773470 27955 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:42.774219 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:42.775297 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:42.828322 25242 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:42.775561 25239 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:42.777343 25237 raft_consensus.cc:3055] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:42.829092 25237 raft_consensus.cc:740] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:42.790413 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:42.676509 27993 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:42.761966 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:42.803550 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:42.832906 25390 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:08:42.814993 28027 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:42.759068 28045 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.042s
I20260709 13:08:42.750005 28146 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.067s
I20260709 13:08:42.817291 27945 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:42.801770 28128 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.017s
I20260709 13:08:42.818835 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:42.824723 25318 raft_consensus.cc:2410] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 13:08:42.825941 28144 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 13:08:42.826414 28151 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:42.835009 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:42.836171 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 74281.2us
I20260709 13:08:42.874531 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
W20260709 13:08:42.826807 28145 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 13:08:42.789669 25238 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:32992
I20260709 13:08:42.797338 28089 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:42.815999 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:42.830183 25237 consensus_queue.cc:260] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:42.773042 27872 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:42.876139 25239 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 7: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:42.876956 25238 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:08:42.878161 25313 raft_consensus.cc:2368] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8.
I20260709 13:08:42.883117 25237 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:42.886346 25238 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:42.883956 25387 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: []
I20260709 13:08:42.849514 28029 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:08:42.874961 28154 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.054s	user 0.002s	sys 0.000s
I20260709 13:08:42.890019 25239 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:08:42.866055 27945 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:42.825063 27987 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:42.888291 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:42.886319 28151 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:42.894876 28029 raft_consensus.cc:697] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:43.061064 25390 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:08:42.898319 27945 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:42.901412 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:43.093530 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:43.265599 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:42.906234 27669 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260709 13:08:43.016808 25386 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
W20260709 13:08:43.331267 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:42.929073 25943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 551) took 1364 ms. Trace:
I20260709 13:08:42.888832 28153 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:42.927295 28148 log.cc:927] Time spent T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
W20260709 13:08:43.092432 27962 consensus_meta.cc:287] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.170s	user 0.012s	sys 0.000s
I20260709 13:08:42.918473 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:42.904817 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:42.920497 25212 leader_election.cc:336] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.592s (SENT)
W20260709 13:08:42.922729 28016 consensus_meta.cc:287] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.525s	user 0.008s	sys 0.000s
W20260709 13:08:42.920881 28000 consensus_meta.cc:287] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.978s	user 0.008s	sys 0.004s
I20260709 13:08:42.967062 25237 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
W20260709 13:08:42.904712 28143 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20260709 13:08:43.332546 25943 rpcz_store.cc:276] 0709 13:08:41.565257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:41.565370 (+   113us) service_pool.cc:224] Handling call
0709 13:08:42.928869 (+1363499us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b5400504e54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:42.929001 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":841075456}
I20260709 13:08:43.386042 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:43.090276 28152 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.223s	user 0.001s	sys 0.000s
W20260709 13:08:43.336022 25945 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.020s	sys 0.022s
W20260709 13:08:43.389802 25945 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.020s	sys 0.023s
I20260709 13:08:43.033636 25238 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:43.390935 25239 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:43.359443 25909 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:43.337721 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:43.397923 25386 raft_consensus.cc:2410] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20260709 13:08:43.339660 28070 consensus_meta.cc:287] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.624s	user 0.011s	sys 0.000s
I20260709 13:08:43.229254 28087 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:42.934309 28128 consensus_queue.cc:1059] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.451s
W20260709 13:08:43.284590 28086 consensus_meta.cc:287] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.846s	user 0.012s	sys 0.005s
W20260709 13:08:43.377182 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 456668us
W20260709 13:08:43.444283 25212 leader_election.cc:336] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.594s (SENT)
W20260709 13:08:43.445546 25212 leader_election.cc:336] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.708s (SENT)
I20260709 13:08:42.914892 27993 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:43.381343 25237 pending_rounds.cc:85] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 3
I20260709 13:08:43.448696 25237 pending_rounds.cc:107] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260709 13:08:43.022636 28146 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:42.918092 28088 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:42.919917 27872 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:43.333521 27997 consensus_meta.cc:287] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.886s	user 0.025s	sys 0.007s
W20260709 13:08:43.383710 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 427626us
I20260709 13:08:43.023905 28045 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:43.386845 25943 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:43.389171 25388 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
W20260709 13:08:43.376430 27941 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.026s	sys 0.027s
W20260709 13:08:43.439353 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:43.448944 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:43.490272 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:43.448724 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:43.448982 25237 raft_consensus.cc:2889] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:43.492174 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:43.451903 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:43.402339 28069 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:43.501693 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:43.502431 25943 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
W20260709 13:08:43.439915 27938 consensus_meta.cc:287] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.972s	user 0.013s	sys 0.004s
W20260709 13:08:43.504895 25212 leader_election.cc:336] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.034s (SENT)
W20260709 13:08:43.506054 25212 leader_election.cc:336] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.132s (SENT)
I20260709 13:08:43.509339 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:43.384944 28016 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:43.337833 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:43.518462 25390 raft_consensus.cc:2410] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:08:43.484306 25358 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:43.391474 28000 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:43.559218 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:43.375284 28153 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:43.562041 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:43.502362 27941 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.027s	sys 0.028s
I20260709 13:08:43.514945 28141 raft_consensus.cc:2764] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 13:08:43.439169 28087 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:43.618604 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 70253.4us
I20260709 13:08:43.549491 28070 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.070s
I20260709 13:08:43.550021 28056 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:43.515890 27669 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:43.501870 27997 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:43.619421 28127 raft_consensus.cc:3055] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:43.508275 28045 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:43.501566 27884 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:43.619910 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:43.620009 28156 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:43.477349 28088 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:43.477797 28027 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:43.489516 28115 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:43.567781 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:43.562779 25390 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:43.502542 27872 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:43.654639 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:43.739473 25395 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:43.456103 28029 consensus_queue.cc:237] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:43.652900 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 1960) took 1168 ms. Trace:
I20260709 13:08:43.757463 25313 rpcz_store.cc:276] 0709 13:08:42.484438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:42.485854 (+  1416us) service_pool.cc:224] Handling call
0709 13:08:43.652624 (+1166770us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b540005df54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:43.652815 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":899878144}
I20260709 13:08:43.679519 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:43.517105 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:43.690327 25940 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 8: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:08:43.622812 28009 consensus_meta.cc:287] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.818s	user 0.007s	sys 0.003s
I20260709 13:08:43.466499 28086 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:43.766460 28087 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:43.767724 28086 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:43.769610 27807 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
W20260709 13:08:43.769795 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:43.770124 27807 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:43.691592 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:43.770643 27807 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:43.771333 25939 consensus_meta.cc:287] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.878s	user 0.008s	sys 0.000s
I20260709 13:08:43.772017 25245 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:43.548516 28128 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:43.473752 28146 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:43.626441 27941 raft_consensus.cc:697] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:43.719658 25392 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:43.677659 27925 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.086s
I20260709 13:08:43.698945 25237 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:43.737087 25386 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
W20260709 13:08:43.682613 27955 consensus_meta.cc:287] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.866s	user 0.012s	sys 0.005s
I20260709 13:08:43.656558 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:43.548516 27972 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:43.743175 28045 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:43.743264 27872 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
W20260709 13:08:43.745780 28158 log.cc:927] Time spent T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.415s	user 0.000s	sys 0.001s
W20260709 13:08:43.746495 28159 log.cc:927] Time spent T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.404s	user 0.001s	sys 0.000s
I20260709 13:08:43.746606 28163 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:43.719913 27884 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:43.774978 25936 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:43.691061 28056 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:43.686684 27669 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:43.807876 27669 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:43.809962 25244 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 1512) took 2039 ms (client timeout 1965 ms). Trace:
W20260709 13:08:43.810552 25244 rpcz_store.cc:269] 0709 13:08:41.770640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:41.775071 (+  4431us) service_pool.cc:224] Handling call
0709 13:08:43.809633 (+2034562us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b5400363154. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:43.809832 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1028422400}
I20260709 13:08:43.691350 28127 raft_consensus.cc:740] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:43.748363 25395 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:43.813503 25395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 1953) took 1073 ms. Trace:
I20260709 13:08:43.813843 25395 rpcz_store.cc:276] 0709 13:08:42.740194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:42.759040 (+ 18846us) service_pool.cc:224] Handling call
0709 13:08:43.654485 (+895445us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8deee 0000000000375939 00007f297bc011e0 00007f297bcca0c4 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:43.813443 (+158958us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":35603072}
I20260709 13:08:43.813894 28127 consensus_queue.cc:260] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:43.750847 28007 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:43.814960 28127 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:43.815284 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:43.816370 25937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 674) took 1290 ms (client timeout 1582 ms). Trace:
I20260709 13:08:43.761516 25387 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:43.822634 28153 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:43.824090 27884 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:43.824739 28016 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:43.826341 28056 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:43.691744 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:43.699434 25942 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:43.781705 25391 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:43.787904 25393 raft_consensus.cc:2376] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10. Re-sending same reply.
I20260709 13:08:43.789391 25386 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:08:43.840186 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 1955) took 1099 ms. Trace:
I20260709 13:08:43.517608 27938 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:43.777344 27987 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:43.802574 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:43.840611 25393 rpcz_store.cc:276] 0709 13:08:42.741157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:42.758444 (+ 17287us) service_pool.cc:224] Handling call
0709 13:08:43.656347 (+897903us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:43.840112 (+183765us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37478656}
I20260709 13:08:43.842682 27938 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:43.747401 28151 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:43.777813 27945 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:43.774551 25939 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:43.845722 25939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 677) took 1279 ms. Trace:
I20260709 13:08:43.763798 28027 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260709 13:08:43.760639 25390 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5. Ops: []
W20260709 13:08:43.816896 25937 rpcz_store.cc:269] 0709 13:08:42.525800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:42.525938 (+   138us) service_pool.cc:224] Handling call
0709 13:08:43.815087 (+1289149us) spinlock_profiling.cc:243] Waited 62.2 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:43.816297 (+  1210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":142998784}
I20260709 13:08:43.760413 28088 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:43.848572 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:43.852042 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:43.855583 25390 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:08:43.856940 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1536) took 1078 ms. Trace:
I20260709 13:08:43.857367 25239 rpcz_store.cc:276] 0709 13:08:42.778242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:42.895692 (+117450us) service_pool.cc:224] Handling call
0709 13:08:42.895966 (+   274us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:43.402594 (+506628us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:43.856716 (+454122us) spinlock_profiling.cc:243] Waited 495 ms on lock 0x7b54002d0fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:43.856882 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1137803648}
I20260709 13:08:43.859256 25393 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:08:43.860008 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 1957) took 1106 ms. Trace:
I20260709 13:08:43.860414 25388 rpcz_store.cc:276] 0709 13:08:42.753695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:42.783258 (+ 29563us) service_pool.cc:224] Handling call
0709 13:08:43.859865 (+1076607us) spinlock_profiling.cc:243] Waited 523 ms on lock 0x7b5400283cd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:43.859956 (+    91us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1202185472}
I20260709 13:08:43.860913 25390 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:43.778995 28128 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:43.854686 25937 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:08:43.863389 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:43.865270 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 381035us
I20260709 13:08:43.796003 27993 consensus_queue.cc:1059] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.054s
I20260709 13:08:43.792829 27955 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:43.870497 25911 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:43.846115 25939 rpcz_store.cc:276] 0709 13:08:42.566240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:42.797683 (+231443us) service_pool.cc:224] Handling call
0709 13:08:43.845656 (+1047973us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:43.874577 25393 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260709 13:08:43.874676 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1528) took 1101 ms. Trace:
I20260709 13:08:43.875398 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 688) took 1348 ms. Trace:
I20260709 13:08:43.881916 25937 rpcz_store.cc:276] 0709 13:08:42.526734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:43.853974 (+1327240us) service_pool.cc:224] Handling call
0709 13:08:43.854294 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:43.868569 (+ 14275us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:43.875309 (+  6740us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:43.876216 25390 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:08:43.879217 25911 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:43.797397 27972 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:43.881870 25243 rpcz_store.cc:276] 0709 13:08:42.773186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:42.775741 (+  2555us) service_pool.cc:224] Handling call
0709 13:08:42.776028 (+   287us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:42.776405 (+   377us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:42.776440 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:42.776751 (+   311us) log.cc:844] Serialized 26 byte log entry
0709 13:08:43.064449 (+287698us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:43.064514 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:43.064620 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:43.874435 (+809815us) raft_consensus.cc:1685] finished
0709 13:08:43.874457 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:43.874611 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7822,"raft.queue_time_us":2769,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":8041,"thread_start_us":287374,"threads_started":1,"wal-append.queue_time_us":300855}
I20260709 13:08:43.887737 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:43.759863 28156 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:43.872777 25358 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:43.886508 27904 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:43.888975 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:43.894424 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:43.895282 25395 raft_consensus.cc:2410] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:08:43.896183 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:43.896862 25395 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:43.898375 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:43.899152 25390 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:08:43.904044 25937 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:43.877000 28153 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:43.892907 25389 raft_consensus.cc:2410] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 13:08:43.940069 25395 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:08:43.941180 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:43.908454 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:43.943555 25395 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:43.908560 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:43.909252 27997 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:08:43.909914 27872 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:43.909992 27972 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:43.910960 28146 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:43.911737 28088 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:43.912779 28156 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:43.900178 27904 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:43.913800 28167 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:43.913117 27938 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:43.915582 28128 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:43.846935 27941 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:43.916445 28157 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 13:08:43.919993 25944 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:43.921180 27807 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:43.922580 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:43.925915 28127 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:08:43.926266 28044 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:43.927265 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:43.928237 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:43.962432 25939 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:43.892941 28086 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:08:43.932363 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:43.883694 28009 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:43.904865 25209 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:44.017633 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 131115us
I20260709 13:08:43.978020 27904 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:43.954414 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 331080us
I20260709 13:08:43.955557 25395 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
W20260709 13:08:43.958168 25361 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.001s	sys 0.019s
W20260709 13:08:44.025344 25361 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.001s	sys 0.020s
I20260709 13:08:43.965543 25911 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:43.972640 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:44.026561 25388 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:43.951334 25936 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 7. Ops: []
I20260709 13:08:43.958550 28056 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:43.997578 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 44205.9us
I20260709 13:08:43.941676 28153 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:44.009073 28151 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:08:44.010715 25387 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:43.931962 25358 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:43.965222 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 1967) took 1050 ms. Trace:
W20260709 13:08:44.040578 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:44.013109 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:44.041249 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:44.024930 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:44.022301 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:44.041908 25312 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:08:44.031349 25388 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:08:44.033495 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:44.036027 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 70607.8us
I20260709 13:08:44.036234 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:44.044117 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 112233us
I20260709 13:08:44.044277 25244 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:44.017278 28009 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:44.055768 28009 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:44.058533 28035 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:44.059170 28035 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:44.042438 25239 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:43.995436 28167 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:44.061880 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:44.062451 25321 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:44.063414 28167 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:44.063349 28027 consensus_queue.cc:1059] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:08:44.064347 28029 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:44.064870 28029 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:44.034672 27904 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:44.065944 27904 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:44.066324 27904 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:44.067520 28029 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:44.068219 28153 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:44.040724 25389 rpcz_store.cc:276] 0709 13:08:42.914355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:43.564249 (+649894us) service_pool.cc:224] Handling call
0709 13:08:43.960778 (+396529us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:44.035519 28176 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:44.070537 28095 raft_consensus.cc:3055] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:44.070363 28176 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:44.071044 28095 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:44.072800 28176 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:44.045589 25388 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:08:44.048080 25312 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:08:44.048768 25317 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:44.050053 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:44.050750 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 376733us
I20260709 13:08:44.051157 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:44.036278 28164 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:44.033474 27807 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.020s
I20260709 13:08:44.053411 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:44.053555 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:44.022125 25938 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:44.054010 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:44.054157 28088 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:44.009841 27669 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:44.054315 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:44.158336 25315 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:44.074470 25911 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:44.078444 25314 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
W20260709 13:08:44.038713 28162 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.200s	user 0.000s	sys 0.001s
W20260709 13:08:44.167248 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 92853us
I20260709 13:08:44.167644 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:44.042011 25395 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 8
W20260709 13:08:44.158531 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 80359.6us
I20260709 13:08:44.169358 25358 leader_election.cc:400] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8.
I20260709 13:08:44.068856 28153 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.013s
I20260709 13:08:44.072230 28095 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:44.160558 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 126179us
I20260709 13:08:44.042369 28086 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:08:44.180001 25395 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:44.042616 27925 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.127s
I20260709 13:08:44.167644 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:44.159108 25320 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:44.169706 25358 leader_election.cc:403] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:08:44.091228 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:44.185802 25314 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:44.186218 25317 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:44.090219 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:44.015297 28044 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:44.090727 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:44.085728 25212 connection.cc:256] RPC call timeout handler was delayed by -1.0298457110000072s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30560000000000004s for a response
I20260709 13:08:44.128159 25908 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:44.077076 25936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 675) took 1511 ms. Trace:
W20260709 13:08:44.202680 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 74642.8us
I20260709 13:08:44.201799 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:44.127638 27713 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:44.121901 25284 connection.cc:256] RPC call timeout handler was delayed by -2.0057443819999889s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17s for a response
W20260709 13:08:44.198772 25212 leader_election.cc:336] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.895s (SENT)
I20260709 13:08:44.038213 28056 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:44.206087 25284 connection.cc:256] RPC call timeout handler was delayed by -1.8545443819999718s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18680000000000002s for a response
I20260709 13:08:44.205945 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:44.209671 28172 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:44.207857 28056 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:44.198294 25313 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:44.210537 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:44.211692 25284 connection.cc:256] RPC call timeout handler was delayed by -1.784344381999972s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1946s for a response
I20260709 13:08:44.212452 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:44.205928 27713 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:44.213328 27955 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:44.213765 28127 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:44.231882 25314 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:44.174664 27997 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:44.181144 27948 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.036s
I20260709 13:08:44.202499 25939 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:44.218227 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 563) took 1745 ms. Trace:
I20260709 13:08:44.173538 28167 consensus_queue.cc:1059] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.059s
I20260709 13:08:44.246392 25941 rpcz_store.cc:276] 0709 13:08:42.473157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:42.473502 (+   345us) service_pool.cc:224] Handling call
0709 13:08:42.473809 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:42.626775 (+152966us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:42.626821 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:42.626883 (+    62us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:42.626938 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:44.217993 (+1591055us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b5400702bd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:44.218041 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:44.218165 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":351132416}
I20260709 13:08:44.218596 27669 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:44.201790 25936 rpcz_store.cc:276] 0709 13:08:42.565472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:43.877104 (+1311632us) service_pool.cc:224] Handling call
0709 13:08:43.902456 (+ 25352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:44.033422 (+130966us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:44.073349 (+ 39927us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:44.169400 27813 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:08:44.170354 28035 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:44.266450 28009 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:44.160977 28164 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:44.269869 28170 consensus_queue.cc:1059] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 13:08:44.270985 28164 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:44.165432 28088 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:44.273186 25936 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 13:08:44.273455 27962 consensus_queue.cc:1059] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:08:44.274284 27962 consensus_queue.cc:1243] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:08:44.274343 28167 raft_consensus.cc:3055] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:44.274601 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:44.274785 28087 raft_consensus.cc:3055] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:44.181228 28179 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:44.275317 28087 raft_consensus.cc:740] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:08:44.222325 25284 connection.cc:256] RPC call timeout handler was delayed by -1.7672443819999444s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1965s for a response
W20260709 13:08:44.275967 25284 connection.cc:256] RPC call timeout handler was delayed by -1.7069443819999606s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20320000000000002s for a response
I20260709 13:08:44.275540 28179 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:44.276620 25284 connection.cc:256] RPC call timeout handler was delayed by -1.68804438199993s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2053s for a response
W20260709 13:08:44.222149 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 181535us
W20260709 13:08:44.277060 25284 connection.cc:256] RPC call timeout handler was delayed by -1.6646443820000059s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20790000000000003s for a response
W20260709 13:08:44.277379 25284 connection.cc:256] RPC call timeout handler was delayed by -1.662844382000003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2081s for a response
W20260709 13:08:44.277640 25284 connection.cc:256] RPC call timeout handler was delayed by -1.6241443819999404s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21240000000000003s for a response
W20260709 13:08:44.277902 25284 connection.cc:256] RPC call timeout handler was delayed by -1.5728443819999711s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21810000000000002s for a response
W20260709 13:08:44.278159 25284 connection.cc:256] RPC call timeout handler was delayed by -1.5305443820000164s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22280000000000003s for a response
I20260709 13:08:44.227854 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:44.232885 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:44.279464 28178 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:08:44.279986 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:44.280577 25314 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:44.281589 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:44.282212 25314 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:44.214268 28086 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:08:44.248100 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:44.221535 25908 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:44.274886 28167 raft_consensus.cc:740] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:44.275426 25237 raft_consensus.cc:2393] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 7: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:44.220840 25320 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:08:44.350732 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:44.350955 25313 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:44.285534 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:44.351902 25239 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:44.311326 25941 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:44.353392 25393 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:08:44.353618 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:44.354301 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:44.311898 25212 leader_election.cc:336] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.220s (SENT)
I20260709 13:08:44.354327 25313 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:44.355871 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 44012.1us
I20260709 13:08:44.332563 25321 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:08:44.340534 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:44.340735 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 62264us
W20260709 13:08:44.341977 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 120512us
W20260709 13:08:44.343525 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 174289us
I20260709 13:08:44.358258 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:44.241803 28151 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8.
I20260709 13:08:44.360399 25312 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:44.360700 25239 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:44.360893 25393 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:08:44.361017 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:44.344885 25314 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:44.226846 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:44.238875 27997 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:44.278414 25284 connection.cc:256] RPC call timeout handler was delayed by -1.5240171730000611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1836s for a response
I20260709 13:08:44.364280 25314 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: []
W20260709 13:08:44.366098 25284 connection.cc:256] RPC call timeout handler was delayed by -1.4511171730000569s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1917s for a response
I20260709 13:08:44.367203 25314 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:44.325583 25317 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:44.357211 25244 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:08:44.361650 25395 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:44.364581 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:44.363838 25245 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 13:08:44.368476 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:44.118621 27904 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:44.342442 28127 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:44.376910 28179 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:44.349754 28009 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:44.371031 28153 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:44.378520 28044 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:44.377079 28127 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:44.378737 28153 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:44.379400 28127 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:44.363479 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:44.384536 25284 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:44.385471 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:44.380896 28153 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:44.274636 25936 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:44.391968 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:44.393529 28167 consensus_queue.cc:260] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:44.373960 25239 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:08:44.374543 27962 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:08:44.375247 28177 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.149s	user 0.000s	sys 0.000s
I20260709 13:08:44.375754 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:44.276494 28087 consensus_queue.cc:260] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:44.376636 27904 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:44.392738 25359 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:44.362215 27813 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:44.369598 25358 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:44.361500 28171 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:44.401057 25313 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:44.372131 27997 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:44.361635 28156 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:44.363610 27945 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:44.457652 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 73226.7us
I20260709 13:08:44.468283 25284 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:44.460353 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 71560.4us
I20260709 13:08:44.399012 25938 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:44.461381 27956 consensus_queue.cc:1059] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:08:44.470294 25287 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:44.484777 25287 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:44.397557 28181 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:44.381887 28174 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:44.403813 28167 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:08:44.460868 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 63474.4us
I20260709 13:08:44.460456 28178 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.009s
I20260709 13:08:44.450999 28088 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:44.481487 25284 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:44.364313 25939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 683) took 1079 ms. Trace:
W20260709 13:08:44.471904 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 102358us
I20260709 13:08:44.509959 25358 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:44.463747 27813 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:44.514542 27813 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:44.490108 28186 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:44.521337 28186 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:08:44.494170 25938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 687) took 1963 ms (client timeout 2203 ms). Trace:
W20260709 13:08:44.525380 25938 rpcz_store.cc:269] 0709 13:08:42.526421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:42.545613 (+ 19192us) service_pool.cc:224] Handling call
0709 13:08:43.932186 (+1386573us) spinlock_profiling.cc:243] Waited 7.34 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:44.489724 (+557538us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16880000}
I20260709 13:08:44.527051 25938 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:08:44.528294 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 479) took 1111 ms. Trace:
I20260709 13:08:44.468578 27948 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:44.493534 27962 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:44.515930 27813 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:44.528740 25246 rpcz_store.cc:276] 0709 13:08:43.416711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:43.668701 (+251990us) service_pool.cc:224] Handling call
0709 13:08:43.669245 (+   544us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:43.945941 (+276696us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:08:43.960552 (+ 14611us) log.cc:844] Serialized 14 byte log entry
0709 13:08:44.387762 (+427210us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:44.397141 (+  9379us) log.cc:844] Serialized 26 byte log entry
0709 13:08:44.397290 (+   149us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:44.397325 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:44.397389 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:44.528108 (+130719us) raft_consensus.cc:1685] finished
0709 13:08:44.528125 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:44.528241 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37628,"prepare.queue_time_us":12114,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":3307,"raft.queue_time_us":144349,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":16563,"thread_start_us":614987,"threads_started":2,"wal-append.queue_time_us":418151}
I20260709 13:08:44.490662 28035 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:44.494931 28087 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:44.505686 25284 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:44.473701 28030 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.036s
I20260709 13:08:44.464707 28171 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:44.544211 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 151527us
I20260709 13:08:44.470000 28156 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:44.470747 28187 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.074s
W20260709 13:08:44.553385 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 59175.7us
I20260709 13:08:44.541940 25284 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:44.476078 27945 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:44.550962 25359 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:44.557098 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:44.557716 25246 raft_consensus.cc:2376] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6. Re-sending same reply.
I20260709 13:08:44.532368 27948 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:44.567365 25237 raft_consensus.cc:3055] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:08:44.542229 28035 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:44.496805 28181 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:44.542071 27813 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:44.559820 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:44.569977 25246 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:44.567757 25237 raft_consensus.cc:740] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:44.561292 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:44.571431 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:44.576457 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:44.579742 25246 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:44.578785 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:44.581836 25239 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
W20260709 13:08:44.586908 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:44.565667 28189 log.cc:927] Time spent T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 13:08:44.589043 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:44.590549 28198 consensus_queue.cc:1059] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.004s
I20260709 13:08:44.592325 28174 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:08:44.575402 27813 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:44.569101 28035 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:44.595677 28055 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:44.596266 28055 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:44.593238 25938 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:44.598460 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:44.599965 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:44.507450 25939 rpcz_store.cc:276] 0709 13:08:43.280073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:44.040281 (+760208us) service_pool.cc:224] Handling call
0709 13:08:44.169270 (+128989us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:44.249088 (+ 79818us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:44.359575 (+110487us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:44.604401 25237 consensus_queue.cc:260] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:44.604956 25940 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 558) took 2749 ms (client timeout 3011 ms). Trace:
I20260709 13:08:44.605394 25237 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:44.568554 27948 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:08:44.605589 25940 rpcz_store.cc:269] 0709 13:08:41.855751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:41.855883 (+   132us) service_pool.cc:224] Handling call
0709 13:08:44.604769 (+2748886us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b54001bf8d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:44.604900 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4078360832}
I20260709 13:08:44.609824 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 432) took 1837 ms. Trace:
I20260709 13:08:44.654251 25391 rpcz_store.cc:276] 0709 13:08:42.772026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:42.792279 (+ 20253us) service_pool.cc:224] Handling call
0709 13:08:43.752301 (+960022us) spinlock_profiling.cc:243] Waited 8.44 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:08:43.781456 (+ 29155us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:43.841808 (+ 60352us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:44.609635 (+767827us) spinlock_profiling.cc:243] Waited 28.9 ms on lock 0x7b5400147b50. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:44.609762 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":85881984}
I20260709 13:08:44.567916 27713 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:44.617934 25240 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:08:44.668638 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:44.587314 27962 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:44.682737 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:44.685505 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:44.687211 25388 raft_consensus.cc:2410] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:08:44.587692 28204 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:44.688982 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:44.690366 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:44.751706 25237 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260709 13:08:44.752418 25237 pending_rounds.cc:85] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 5
I20260709 13:08:44.752776 25237 pending_rounds.cc:107] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 13:08:44.685586 25938 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:44.772179 25237 raft_consensus.cc:2889] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:44.787554 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:44.793907 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:44.677667 27962 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:44.792788 25391 raft_consensus.cc:2376] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7. Re-sending same reply.
I20260709 13:08:44.694033 28030 raft_consensus.cc:2764] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260709 13:08:44.792714 28189 log.cc:927] Time spent T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 13:08:44.818367 27962 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:44.863843 28127 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:44.795256 28108 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:44.807768 28178 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:08:44.916898 28192 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.350s	user 0.000s	sys 0.001s
W20260709 13:08:44.921883 28199 log.cc:927] Time spent T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 13:08:44.928099 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:44.930462 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:44.931082 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:44.932183 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:44.932358 25395 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:44.928907 25392 raft_consensus.cc:2376] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10. Re-sending same reply.
I20260709 13:08:44.933450 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:44.934751 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:44.933480 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:44.869561 28055 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:44.936996 25395 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: [5.5-7.6]
W20260709 13:08:44.938184 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 396287us
I20260709 13:08:44.935870 25387 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:44.944258 25936 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:44.946244 25284 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:44.947714 28030 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:44.948901 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:44.945209 28193 log.cc:927] Time spent T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.414s	user 0.000s	sys 0.001s
I20260709 13:08:44.953827 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:44.957614 28156 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:44.958292 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:44.958820 28171 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:44.959401 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:44.960620 25944 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 560) took 3028 ms (client timeout 1955 ms). Trace:
I20260709 13:08:44.924228 28108 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:44.963459 28108 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:44.973157 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:44.974260 25209 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.68:39211 timed out after 2.203s (SENT) [suppressed 33 similar messages]
I20260709 13:08:44.973783 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:44.979534 25391 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:44.975572 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:44.980562 25390 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 8: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:44.977403 28181 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:44.977562 27945 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:44.985909 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:44.985986 28127 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:44.957803 28196 log.cc:927] Time spent T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.285s	user 0.001s	sys 0.000s
I20260709 13:08:44.986596 25938 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:44.986734 28095 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:44.979034 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1732) took 1487 ms. Trace:
I20260709 13:08:44.972463 27813 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:44.981343 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:44.990554 25391 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
W20260709 13:08:44.990931 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:44.992499 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 434991us
I20260709 13:08:44.983759 25284 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:44.984115 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:44.993602 25942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 679) took 2339 ms (client timeout 1708 ms). Trace:
I20260709 13:08:44.946242 28000 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:44.961054 25944 rpcz_store.cc:269] 0709 13:08:41.926574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:41.926730 (+   156us) service_pool.cc:224] Handling call
0709 13:08:43.802413 (+1875683us) spinlock_profiling.cc:243] Waited 1.82 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:44.954588 (+1152175us) spinlock_profiling.cc:243] Waited 71.3 ms on lock 0x7b540037ee54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:44.954678 (+    90us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":168177280}
I20260709 13:08:44.992130 25391 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:08:44.995935 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:44.996299 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:44.988704 25943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 694) took 1589 ms. Trace:
I20260709 13:08:44.988754 25241 rpcz_store.cc:276] 0709 13:08:43.491417 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:43.750091 (+258674us) service_pool.cc:224] Handling call
0709 13:08:44.220915 (+470824us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:44.221090 (+   175us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:44.221113 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:44.221181 (+    68us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:08:44.221211 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:44.221265 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:44.978866 (+757601us) spinlock_profiling.cc:243] Waited 471 ms on lock 0x7b5400456850. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:44.978979 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1082173312}
W20260709 13:08:44.993980 25942 rpcz_store.cc:269] 0709 13:08:42.654225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:42.682803 (+ 28578us) service_pool.cc:224] Handling call
0709 13:08:44.993422 (+2310619us) spinlock_profiling.cc:243] Waited 992 ms on lock 0x7b54006d9554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:44.993536 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2280859392}
I20260709 13:08:44.994531 25284 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:44.957121 27997 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:44.981639 28035 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:44.996474 25944 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:45.211894 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:45.213238 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:45.210108 27960 raft_consensus.cc:3055] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:45.213824 27960 raft_consensus.cc:740] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:45.214149 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:45.214687 27960 consensus_queue.cc:260] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:45.215876 27960 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:45.213826 25239 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:45.217383 25945 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 561) took 2745 ms. Trace:
I20260709 13:08:45.217321 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1533) took 2442 ms. Trace:
I20260709 13:08:45.219725 25936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 698) took 1333 ms. Trace:
I20260709 13:08:45.029716 25238 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:44.994066 25390 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:45.220263 25936 rpcz_store.cc:276] 0709 13:08:43.885934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:44.393850 (+507916us) service_pool.cc:224] Handling call
0709 13:08:44.394144 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:44.982107 (+587963us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:45.219500 (+237393us) spinlock_profiling.cc:243] Waited 550 ms on lock 0x7b540062e3d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:45.219667 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1264510848}
I20260709 13:08:45.214726 25940 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:45.218369 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:45.223263 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:45.071774 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 678) took 2456 ms. Trace:
I20260709 13:08:45.223906 25940 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 8: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:08:45.224188 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:45.221696 25943 rpcz_store.cc:276] 0709 13:08:43.399698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:44.350216 (+950518us) service_pool.cc:224] Handling call
0709 13:08:44.350572 (+   356us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:08:44.350920 (+   348us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:44.350949 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:08:44.374366 (+ 23417us) log.cc:844] Serialized 52 byte log entry
0709 13:08:44.910594 (+536228us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:44.910659 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:44.910755 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:44.988472 (+ 77717us) raft_consensus.cc:1685] finished
0709 13:08:44.988494 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:44.988637 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":618,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":9969,"thread_start_us":457172,"threads_started":1,"wal-append.queue_time_us":579185}
W20260709 13:08:45.226989 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:45.100268 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:45.231161 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:45.222200 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:45.218631 25242 rpcz_store.cc:276] 0709 13:08:42.774445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:42.852419 (+ 77974us) service_pool.cc:224] Handling call
0709 13:08:42.852808 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:42.869693 (+ 16885us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:08:42.885953 (+ 16260us) log.cc:844] Serialized 14 byte log entry
0709 13:08:43.383119 (+497166us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:43.391003 (+  7884us) log.cc:844] Serialized 26 byte log entry
0709 13:08:43.391156 (+   153us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:43.835565 (+444409us) log.cc:844] Serialized 14 byte log entry
0709 13:08:43.842925 (+  7360us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:45.216888 (+1373963us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b54003aa7d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:45.216933 (+    45us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:45.217108 (+   175us) raft_consensus.cc:1685] finished
0709 13:08:45.217141 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:45.217269 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8034,"prepare.queue_time_us":537808,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":73,"raft.queue_time_us":20368,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":26304,"spinlock_wait_cycles":37568768,"thread_start_us":892429,"threads_started":2,"wal-append.queue_time_us":880645}
I20260709 13:08:45.017459 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:45.224963 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:45.232841 25242 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:44.998919 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:45.223842 25937 rpcz_store.cc:276] 0709 13:08:42.615562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:43.883133 (+1267571us) service_pool.cc:224] Handling call
0709 13:08:43.883396 (+   263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:43.908150 (+ 24754us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:45.071438 (+1163288us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x7b5400014d54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:45.071686 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46951296}
W20260709 13:08:45.233680 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 234828us
I20260709 13:08:45.234138 25239 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
W20260709 13:08:45.138547 25361 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.009s	sys 0.033s
I20260709 13:08:45.234864 25911 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:45.235406 25361 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.010s	sys 0.038s
I20260709 13:08:45.095965 25209 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:44.996891 25243 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:08:45.217979 25945 rpcz_store.cc:276] 0709 13:08:42.472192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:42.473276 (+  1084us) service_pool.cc:224] Handling call
0709 13:08:42.521922 (+ 48646us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:08:43.415415 (+893493us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:43.415456 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:08:43.416281 (+   825us) log.cc:844] Serialized 52 byte log entry
0709 13:08:43.848256 (+431975us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:43.849697 (+  1441us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:45.210685 (+1360988us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b5400460fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:45.210735 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:45.210785 (+    50us) raft_consensus.cc:1685] finished
0709 13:08:45.210806 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:45.217328 (+  6522us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7118,"raft.queue_time_us":99840,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"spinlock_wait_cycles":640771456,"thread_start_us":923969,"threads_started":2,"wal-append.queue_time_us":554751,"wal-append.run_cpu_time_us":2212,"wal-append.run_wall_time_us":1053731}
I20260709 13:08:45.031551 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:45.000877 25317 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.010s	sys 0.015s
W20260709 13:08:45.238224 25317 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.773s	user 0.011s	sys 0.019s
I20260709 13:08:45.036067 25388 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:08:45.229745 25285 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:44.999892 25909 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:45.244724 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1901) took 1153 ms. Trace:
I20260709 13:08:45.238703 25944 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:45.232841 25237 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:08:45.239531 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 143690us
W20260709 13:08:45.224772 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 230299us
I20260709 13:08:45.250645 25284 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:45.239918 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:32992 (request call id 1526) took 2448 ms. Trace:
I20260709 13:08:45.236752 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:45.241379 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:45.241459 25245 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:45.252655 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:45.253281 25245 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:45.269500 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:45.270443 25244 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:45.274586 25943 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:08:45.276055 25284 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:45.244355 25285 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:45.251286 25238 rpcz_store.cc:276] 0709 13:08:42.772573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:42.774085 (+  1512us) service_pool.cc:224] Handling call
0709 13:08:45.002131 (+2228046us) spinlock_profiling.cc:243] Waited 907 us on lock 0x7b2400120ab8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973a48cb9 00007f2973a46dcb 00007f2973b452b1 00007f2973b40848 00007f2973b407ad 00007f2974f819a7 00007f2976c0e85c 00007f2976c0e699 00007f2976c5f012 00007f2976ca9a55
0709 13:08:45.220856 (+218725us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2085760}
I20260709 13:08:45.277046 25237 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7. Ops: []
W20260709 13:08:45.279121 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 52615.4us
W20260709 13:08:45.280251 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:45.296469 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:45.041747 28056 raft_consensus.cc:3055] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:45.001623 27884 raft_consensus.cc:2764] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
W20260709 13:08:45.331423 25287 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.837s	user 0.036s	sys 0.022s
I20260709 13:08:45.046399 27813 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:45.339018 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:45.339238 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 339420us
I20260709 13:08:45.340112 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:45.298909 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 63143.8us
I20260709 13:08:45.245862 25317 rpcz_store.cc:276] 0709 13:08:44.091567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:44.369358 (+277791us) service_pool.cc:224] Handling call
0709 13:08:44.369596 (+   238us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:44.369756 (+   160us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:08:44.465097 (+ 95341us) log.cc:844] Serialized 14 byte log entry
0709 13:08:45.244250 (+779153us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:45.244324 (+    74us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:08:45.244387 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:45.244482 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:45.244634 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31817,"prepare.queue_time_us":95884,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":60,"thread_start_us":841654,"threads_started":2}
W20260709 13:08:45.299882 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 55599.8us
I20260709 13:08:45.301201 25284 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:45.033571 28000 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:45.277372 28070 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
W20260709 13:08:45.332055 25287 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.837s	user 0.036s	sys 0.022s
I20260709 13:08:45.017022 27945 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.305s
I20260709 13:08:45.360472 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:45.041870 27948 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:45.278344 27925 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:45.274206 28086 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:45.275852 28128 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:45.276691 28032 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:45.350541 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:45.303369 28056 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:08:45.356642 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 55575.4us
W20260709 13:08:45.355057 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 870270us
I20260709 13:08:45.100440 25939 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 13:08:45.358852 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:45.391430 25238 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:45.354207 28178 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260709 13:08:45.297559 28176 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:45.331817 28186 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:45.364916 27884 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:08:45.332020 28164 raft_consensus.cc:3055] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:45.331560 28204 raft_consensus.cc:2764] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 13:08:45.087265 27962 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:45.370893 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:45.277143 28107 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:45.372304 28205 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.003s
I20260709 13:08:45.290323 27807 consensus_queue.cc:1059] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.078s
I20260709 13:08:45.239877 28210 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:45.439858 28210 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:45.440330 28210 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:45.442639 27948 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:45.393277 25395 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 13:08:45.446008 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:45.298360 28108 consensus_queue.cc:1059] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.070s
I20260709 13:08:45.396001 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:45.363233 28211 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:45.398356 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:45.370618 28086 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:45.369858 28032 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:45.370333 28128 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:45.335983 28208 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.091s
I20260709 13:08:45.390448 28127 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:45.415868 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:45.422468 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:45.410801 28107 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:45.360852 28000 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:45.479244 27997 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:45.480553 28000 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:45.482141 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:45.482795 25394 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 1949) took 2744 ms (client timeout 1963 ms). Trace:
I20260709 13:08:45.404125 28186 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:45.462378 25944 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: []
W20260709 13:08:45.483371 25394 rpcz_store.cc:269] 0709 13:08:42.738642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:42.738740 (+    98us) service_pool.cc:224] Handling call
0709 13:08:45.482517 (+2743777us) spinlock_profiling.cc:243] Waited 2.65 s on lock 0x7b54003237d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:45.482709 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6105077504}
I20260709 13:08:45.463057 28056 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:45.458554 25390 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:45.484431 28035 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:45.465233 27962 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:45.465585 25386 raft_consensus.cc:2393] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:45.470403 27807 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:45.402196 28164 raft_consensus.cc:740] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:45.394550 28181 consensus_queue.cc:1059] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.063s
I20260709 13:08:45.399586 28030 consensus_queue.cc:1059] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
I20260709 13:08:45.427742 25937 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:45.462528 27973 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:45.466460 28086 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:45.459590 25938 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:45.477605 27925 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:45.468598 28032 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:45.463301 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:45.464015 25942 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:08:45.484476 28056 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:45.483594 28186 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:08:45.464851 28213 log.cc:927] Time spent T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 13:08:45.472506 25389 raft_consensus.cc:2410] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:08:45.485737 27807 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:45.482990 25393 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 8
I20260709 13:08:45.468793 28128 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:45.471738 25387 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:45.471433 28127 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:45.487524 28107 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:08:45.618705 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:45.619788 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:45.548969 27973 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:45.549487 28087 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:45.630900 27945 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:45.554119 27925 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:45.554427 28035 raft_consensus.cc:3055] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:45.469120 28211 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:45.640857 28127 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:45.684252 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1546) took 1111 ms. Trace:
I20260709 13:08:45.684782 25246 rpcz_store.cc:276] 0709 13:08:44.572829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:44.590309 (+ 17480us) service_pool.cc:224] Handling call
0709 13:08:44.602481 (+ 12172us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:44.666709 (+ 64228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:44.666744 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:44.667265 (+   521us) log.cc:844] Serialized 26 byte log entry
0709 13:08:44.996787 (+329522us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:44.996830 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:44.996921 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:45.683950 (+687029us) raft_consensus.cc:1685] finished
0709 13:08:45.683977 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:45.684175 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3153,"raft.queue_time_us":7878,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":9634,"thread_start_us":329238,"threads_started":1,"wal-append.queue_time_us":700222}
W20260709 13:08:45.672430 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 332409us
I20260709 13:08:45.657325 27973 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:45.675555 25393 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:08:45.697410 25359 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:45.698334 25911 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:45.702256 25246 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:45.703228 25359 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:45.676185 28035 raft_consensus.cc:740] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:45.694335 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:45.694980 25237 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:45.705750 25359 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:45.665658 28087 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:45.692476 25909 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:45.637522 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 256456us
I20260709 13:08:45.712109 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:45.721922 25244 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:45.702209 27813 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:45.725324 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 331394us
I20260709 13:08:45.675863 27945 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:45.725981 25359 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:45.727687 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 350850us
W20260709 13:08:45.729681 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:45.708858 27962 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:45.707206 28220 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:45.734524 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:45.715126 27948 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:45.742002 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 696) took 2337 ms. Trace:
I20260709 13:08:45.728969 27938 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:08:45.749498 25941 rpcz_store.cc:276] 0709 13:08:43.404990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:44.353738 (+948748us) service_pool.cc:224] Handling call
0709 13:08:44.353993 (+   255us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:44.354332 (+   339us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:44.354360 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:44.390509 (+ 36149us) log.cc:844] Serialized 26 byte log entry
0709 13:08:44.941949 (+551440us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:08:45.032788 (+ 90839us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b4800270258. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297bbdbaa1 00007f297bbd2b37 00007f297bbd29ca 00007f297bcca26f 00007f297bcc9f37 00007f297bcde55c 00007f297bcde508 00007f297bcde488 00007f297bcde443 00007f297bcdd6ec 00007f297ac07a96
0709 13:08:45.400294 (+367506us) log.cc:844] Serialized 14 byte log entry
0709 13:08:45.732382 (+332088us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:45.732450 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:45.732487 (+    37us) raft_consensus.cc:1685] finished
0709 13:08:45.741768 (+  9281us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:45.741945 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":620159,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":806,"raft.queue_time_us":79409,"raft.run_cpu_time_us":13903,"raft.run_wall_time_us":670929,"spinlock_wait_cycles":34896640,"thread_start_us":1221496,"threads_started":3,"wal-append.queue_time_us":640592}
I20260709 13:08:45.736872 27962 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:45.738807 28170 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:45.743251 27948 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:45.757808 28214 log.cc:927] Time spent T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.289s	user 0.000s	sys 0.001s
I20260709 13:08:45.743482 28176 raft_consensus.cc:697] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:45.775729 28170 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:45.746497 28201 consensus_queue.cc:1059] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
W20260709 13:08:45.786378 28216 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.226s	user 0.001s	sys 0.001s
I20260709 13:08:45.777544 28221 raft_consensus.cc:3055] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:45.773018 28224 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:08:45.799926 28218 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 13:08:45.821909 28176 consensus_queue.cc:237] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:45.802309 28170 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:45.678948 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 404230us
I20260709 13:08:46.049224 28070 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:08:45.668861 25212 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:45.731034 25287 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:46.052698 28070 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:08:46.053520 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 384609us
W20260709 13:08:46.053799 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 322722us
I20260709 13:08:46.055080 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:46.055455 25212 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:45.891909 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1552) took 1317 ms. Trace:
W20260709 13:08:45.899341 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:45.768782 27962 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:08:45.909314 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 180166us
I20260709 13:08:45.910693 27807 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:46.058898 28238 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:45.778172 27997 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:46.060034 27997 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:46.060494 27997 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:45.752630 28035 consensus_queue.cc:260] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:43.910815 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:08:45.783197 28108 raft_consensus.cc:3055] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:46.077656 28108 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:46.078091 27945 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:45.837049 28210 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:45.834414 28211 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:45.808504 28222 raft_consensus.cc:3055] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:08:45.815634 28221 raft_consensus.cc:740] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:46.056397 25240 rpcz_store.cc:276] 0709 13:08:44.574471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:44.675162 (+100691us) service_pool.cc:224] Handling call
0709 13:08:44.675443 (+   281us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:44.796572 (+121129us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:08:44.870652 (+ 74080us) log.cc:844] Serialized 14 byte log entry
0709 13:08:45.712764 (+842112us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:45.713267 (+   503us) log.cc:844] Serialized 26 byte log entry
0709 13:08:45.713433 (+   166us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:45.713467 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:45.713534 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:45.890785 (+177251us) raft_consensus.cc:1685] finished
0709 13:08:45.890884 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:45.891488 (+   604us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44367,"prepare.queue_time_us":75010,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":56,"raft.queue_time_us":511811,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":16021,"thread_start_us":770358,"threads_started":2,"wal-append.queue_time_us":538479}
I20260709 13:08:46.094828 28221 consensus_queue.cc:260] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.096505 28221 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:08:46.097330 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 80597.9us
I20260709 13:08:46.098512 28016 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:08:45.956156 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 208440us
W20260709 13:08:45.973697 28095 consensus_meta.cc:287] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.986s	user 0.009s	sys 0.003s
I20260709 13:08:45.757455 27938 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:46.107232 27938 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:08:45.982059 28087 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:45.982795 28223 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:46.109911 28223 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:45.731752 27941 consensus_queue.cc:1059] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 13:08:45.826924 28220 raft_consensus.cc:697] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:08:45.997861 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 533997us
I20260709 13:08:46.008273 25939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 690) took 1424 ms. Trace:
I20260709 13:08:46.111923 25939 rpcz_store.cc:276] 0709 13:08:44.583551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:44.679066 (+ 95515us) service_pool.cc:224] Handling call
0709 13:08:45.001768 (+322702us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:45.393974 (+392206us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:46.007805 (+613831us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b54006d9550. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:46.008056 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":473879936}
I20260709 13:08:46.012833 28174 raft_consensus.cc:3055] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:46.113390 28174 raft_consensus.cc:740] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:46.030164 27884 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:45.780622 27948 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:46.043946 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:45.737511 27813 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:46.056910 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:46.071666 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:46.080461 28108 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.089948 28222 raft_consensus.cc:740] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:46.105702 28234 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:46.105822 28236 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:08:46.111296 25245 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:46.181357 25284 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:46.173291 27973 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:46.182914 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 61828.9us
I20260709 13:08:46.184459 27973 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:46.184566 27813 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:46.124698 25315 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:46.185506 28016 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:08:46.185492 27925 consensus_meta.cc:287] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.507s	user 0.008s	sys 0.008s
I20260709 13:08:46.186041 28016 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:46.190062 28016 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:46.177862 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 65052.3us
I20260709 13:08:46.188627 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:46.189265 28179 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:46.195207 28179 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.177755 28226 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:46.196825 28226 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.114828 28174 consensus_queue.cc:260] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.125804 25287 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:46.200955 28174 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:46.201049 25284 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:46.201830 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:46.202271 28228 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:46.112829 28220 consensus_queue.cc:237] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.202924 28228 raft_consensus.cc:697] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:08:46.206770 28186 consensus_meta.cc:287] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.570s	user 0.011s	sys 0.005s
I20260709 13:08:46.207345 28186 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:46.209514 28210 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:46.214632 28164 consensus_queue.cc:260] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:46.216388 28164 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:46.204190 28228 consensus_queue.cc:237] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:46.225478 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:46.080523 28205 consensus_queue.cc:1059] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.027s
W20260709 13:08:46.242453 25128 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f75b60146e7d4b9b9954b4fb2c414679 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 2329e428ebbf484fa3ea9452133df567 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:46.244839 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:46.247990 28210 raft_consensus.cc:3055] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:08:46.248615 28210 raft_consensus.cc:740] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:46.250226 28226 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:46.124544 27956 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:46.249967 28210 consensus_queue.cc:260] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:46.267382 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 141787us
W20260709 13:08:46.274395 28086 consensus_meta.cc:287] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.713s	user 0.011s	sys 0.006s
I20260709 13:08:46.274936 28086 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:46.277525 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 468690us
I20260709 13:08:46.279078 28223 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:46.280280 28128 consensus_meta.cc:287] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.622s	user 0.014s	sys 0.000s
I20260709 13:08:46.186316 27925 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.254807 27956 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:46.280808 28128 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.281458 27956 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:46.283064 27925 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:46.123664 27884 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:46.276973 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:46.240132 28234 raft_consensus.cc:697] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:46.126281 28095 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.143s
W20260709 13:08:46.291671 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:46.295519 28222 consensus_queue.cc:260] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.297361 28222 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:46.270524 28108 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:46.308668 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 87791.2us
I20260709 13:08:46.311539 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:46.287808 28128 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:46.322827 28201 consensus_queue.cc:1059] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:08:46.323846 28201 raft_consensus.cc:3055] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:08:46.324172 28201 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:08:46.325855 28107 consensus_meta.cc:287] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.618s	user 0.011s	sys 0.002s
I20260709 13:08:46.326498 28107 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:46.329308 25244 consensus_meta.cc:287] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.562s	user 0.010s	sys 0.003s
I20260709 13:08:46.329708 28210 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:08:46.330487 28250 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 13:08:46.331772 28217 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:08:46.333285 28032 consensus_meta.cc:287] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.702s	user 0.022s	sys 0.000s
W20260709 13:08:46.340348 28127 consensus_meta.cc:287] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.639s	user 0.002s	sys 0.009s
I20260709 13:08:46.343551 28035 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:46.346944 27884 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:46.351460 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:46.352254 25940 raft_consensus.cc:2376] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6. Re-sending same reply.
I20260709 13:08:46.352733 28179 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:46.353649 28186 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:46.354426 25315 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 401) took 2213 ms (client timeout 1656 ms). Trace:
W20260709 13:08:46.406028 25315 rpcz_store.cc:269] 0709 13:08:44.140698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:44.203882 (+ 63184us) service_pool.cc:224] Handling call
0709 13:08:46.354152 (+2150270us) spinlock_profiling.cc:243] Waited 1.9 s on lock 0x7b54000429d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:46.354296 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4374052736}
I20260709 13:08:46.355125 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:46.407461 25939 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:46.344830 28170 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:46.410917 25140 catalog_manager.cc:5697] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 3 to 4, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 4 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:08:46.412362 28170 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:46.355695 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 154906us
I20260709 13:08:46.340973 28127 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.416488 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:46.417490 25240 raft_consensus.cc:2410] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:08:46.418543 28127 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:46.418685 25240 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: []
W20260709 13:08:46.358171 25143 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39596 (request call id 284) took 11262 ms (client timeout 15000 ms). Trace:
I20260709 13:08:46.362546 28232 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:46.365379 25237 consensus_meta.cc:287] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.650s	user 0.008s	sys 0.008s
I20260709 13:08:46.423928 28232 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.426656 28232 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:46.362345 27962 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:46.366240 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 81634.9us
I20260709 13:08:46.428805 27962 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:46.367045 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:46.369433 28211 consensus_queue.cc:1059] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:08:46.370285 27941 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:08:46.430384 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:46.369455 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:46.373395 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1555) took 1101 ms. Trace:
I20260709 13:08:46.432075 25241 rpcz_store.cc:276] 0709 13:08:45.271930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:45.289537 (+ 17607us) service_pool.cc:224] Handling call
0709 13:08:45.346090 (+ 56553us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:45.673014 (+326924us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:45.673060 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:45.673678 (+   618us) log.cc:844] Serialized 26 byte log entry
0709 13:08:46.093949 (+420271us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:46.094221 (+   272us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:46.094606 (+   385us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:46.372873 (+278267us) raft_consensus.cc:1685] finished
0709 13:08:46.372901 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:46.373160 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8137,"raft.queue_time_us":399326,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":515,"thread_start_us":687403,"threads_started":2,"wal-append.queue_time_us":595668}
I20260709 13:08:46.434074 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:46.434239 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:46.434973 25241 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:46.435393 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:46.435884 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:46.436149 25242 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:46.381744 28070 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:46.353018 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:46.384371 27948 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:46.385529 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 64445.9us
I20260709 13:08:46.329741 25244 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:46.386183 28217 raft_consensus.cc:697] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:46.386966 28030 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:08:46.352411 25942 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:46.333940 28032 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:46.387952 27938 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:46.309556 27813 consensus_queue.cc:1059] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20260709 13:08:46.390846 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:46.316525 28086 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:46.391656 27960 consensus_meta.cc:287] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.175s	user 0.008s	sys 0.013s
I20260709 13:08:46.391489 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:46.392076 27973 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:46.391518 25392 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:46.394479 28087 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:46.395316 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:46.395898 28201 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.365463 28156 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:46.398764 28056 consensus_meta.cc:287] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.768s	user 0.013s	sys 0.005s
I20260709 13:08:46.405197 28107 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:46.415832 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:46.422969 28016 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.366400 28234 consensus_queue.cc:237] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:46.422969 25240 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260709 13:08:46.412648 25140 catalog_manager.cc:5697] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 1 to 6, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 6 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:08:46.441303 25239 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:46.442713 25939 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:46.450662 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:46.420495 27997 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.466442 25285 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:46.464001 25287 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:46.472424 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:46.475186 25243 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:46.475881 25943 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:46.459800 28232 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:46.485843 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:46.488857 25391 raft_consensus.cc:2410] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:08:46.485424 28087 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.489569 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:46.491078 27938 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:46.490588 25140 catalog_manager.cc:5697] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:08:46.501432 25940 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:46.493777 27997 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:46.495601 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:46.480844 28174 raft_consensus.cc:3055] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:46.475078 28212 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:46.484236 27960 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:46.502301 25942 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:46.470381 27962 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:46.421406 25143 rpcz_store.cc:269] 0709 13:08:35.095423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:35.095596 (+   173us) service_pool.cc:224] Handling call
0709 13:08:35.699102 (+603506us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b5400470ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1c090 00007f2976cbaf3a 00007f297ca711cc 00007f297cb6f37e 00007f297b0302a4 00007f297b030232 00007f297b030161 00007f297b0300dc 00007f297b02f392 00007f2974fed48c 00007f2974fec8c6
0709 13:08:46.357958 (+10658856us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:08:37.890728 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:08:38.011292 (+120564us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:08:38.011357 (+    65us) write_op.cc:435] Acquired schema lock
0709 13:08:38.011384 (+    27us) tablet.cc:662] Decoding operations
0709 13:08:38.315868 (+304484us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:08:38.315970 (+   102us) write_op.cc:641] Partition lock acquired for write op
0709 13:08:38.316007 (+    37us) tablet.cc:685] Acquiring locks for 14 operations
0709 13:08:38.317469 (+  1462us) tablet.cc:701] Row locks acquired
0709 13:08:38.317513 (+    44us) write_op.cc:260] PREPARE: finished
0709 13:08:38.317740 (+   227us) write_op.cc:270] Start()
0709 13:08:38.317899 (+   159us) write_op.cc:276] Timestamp: P: 1783602518317664 usec, L: 0
0709 13:08:38.317930 (+    31us) op_driver.cc:348] REPLICATION: starting
0709 13:08:38.318781 (+   851us) log.cc:844] Serialized 5594 byte log entry
0709 13:08:39.185062 (+866281us) op_driver.cc:464] REPLICATION: finished
0709 13:08:39.564368 (+379306us) write_op.cc:301] APPLY: starting
0709 13:08:39.564494 (+   126us) tablet.cc:1367] starting BulkCheckPresence
0709 13:08:39.564878 (+   384us) tablet.cc:1370] finished BulkCheckPresence
0709 13:08:39.565041 (+   163us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:08:39.748011 (+182970us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:08:39.748433 (+   422us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0709 13:08:39.748504 (+    71us) write_op.cc:312] APPLY: finished
0709 13:08:39.749059 (+   555us) log.cc:844] Serialized 101 byte log entry
0709 13:08:39.920947 (+171888us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:39.938092 (+ 17145us) write_op.cc:454] Released schema lock
0709 13:08:40.030965 (+ 92873us) write_op.cc:341] FINISH: Updating metrics
0709 13:08:40.069738 (+ 38773us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:40.074219 (+  4481us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":235886,"reported_tablets":35,"spinlock_wait_cycles":681557888,"child_traces":[["op",{"apply.queue_time_us":342271,"apply.run_cpu_time_us":5384,"apply.run_wall_time_us":487531,"num_ops":14,"prepare.queue_time_us":106330,"prepare.run_cpu_time_us":18149,"prepare.run_wall_time_us":570815,"raft.queue_time_us":467004,"raft.run_cpu_time_us":19611,"raft.run_wall_time_us":123412,"replication_time_us":866991,"thread_start_us":415893,"threads_started":4,"wal-append.queue_time_us":396624,"wal-append.run_cpu_time_us":53730,"wal-append.run_wall_time_us":2392916}]]}
I20260709 13:08:46.518167 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2004) took 1168 ms. Trace:
W20260709 13:08:46.521181 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 57541.5us
I20260709 13:08:46.521466 27941 raft_consensus.cc:3055] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:46.522053 28181 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:46.525316 27925 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:46.525940 27884 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:46.526785 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:46.527633 25209 leader_election.cc:336] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.554s (SENT)
I20260709 13:08:46.528646 28201 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:46.528276 28217 consensus_queue.cc:237] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.529217 27945 consensus_queue.cc:1059] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:08:46.593531 28211 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:46.594295 28211 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:46.529660 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:46.535854 28016 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:46.559863 25285 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:46.597333 27938 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:46.597980 27938 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:46.600286 28016 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:46.600713 28016 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:46.601068 28016 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:46.558596 25140 catalog_manager.cc:5697] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 3 to 5, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 5 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:08:46.498617 27813 consensus_queue.cc:1059] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
I20260709 13:08:46.511853 25395 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:46.558830 25391 rpcz_store.cc:276] 0709 13:08:45.350116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:45.350252 (+   136us) service_pool.cc:224] Handling call
0709 13:08:46.517875 (+1167623us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400145d54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:46.518075 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2500526976}
I20260709 13:08:46.562785 25939 raft_consensus.cc:2393] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 8: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:46.569077 27925 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:46.570957 27962 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.561714 27941 raft_consensus.cc:740] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:46.571882 28227 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:46.572005 27960 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:46.571890 28212 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:46.572871 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 45310.3us
I20260709 13:08:46.574556 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:46.576426 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:46.576609 25318 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:08:46.501453 27973 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.582309 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:46.505714 28174 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:08:46.591150 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:46.643963 25209 leader_election.cc:336] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.652s (SENT)
I20260709 13:08:46.643731 28174 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.646904 28070 consensus_queue.cc:1059] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 13:08:46.569821 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 216894us
I20260709 13:08:46.630307 25140 catalog_manager.cc:5697] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:08:46.656687 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 191445us
I20260709 13:08:46.667028 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:46.673830 27941 consensus_queue.cc:260] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.631806 27925 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:46.679942 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:46.726284 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 63101.8us
I20260709 13:08:46.649292 28227 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:46.747440 25315 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20260709 13:08:46.762787 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 95906.1us
W20260709 13:08:46.768692 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:46.774921 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:46.775985 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 385184us
W20260709 13:08:46.774205 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 323362us
I20260709 13:08:46.744633 28221 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.006s
I20260709 13:08:46.780359 28108 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:46.783529 28265 raft_consensus.cc:3055] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:46.784533 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:46.577792 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:46.751281 25140 catalog_manager.cc:5697] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 3 to 5, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 5 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:08:46.795910 25392 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:46.787657 27948 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:46.503715 28156 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.805307 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:46.791486 28265 raft_consensus.cc:740] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:46.803845 28174 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:46.812167 25318 raft_consensus.cc:2376] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6. Re-sending same reply.
I20260709 13:08:46.803259 28032 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:46.805328 28108 raft_consensus.cc:697] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:08:46.826483 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 62990.1us
I20260709 13:08:46.825917 25911 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:46.834143 27962 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:46.833767 28108 consensus_queue.cc:237] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.828891 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:46.821020 27884 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:46.817070 28087 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:46.830598 25941 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
W20260709 13:08:46.827091 28259 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 13:08:46.850678 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:46.851449 25359 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:46.831537 27960 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:46.855767 27962 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:46.856287 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:46.856801 28204 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:08:46.856989 25140 catalog_manager.cc:5697] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:08:46.856812 27884 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.856990 28156 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:46.889600 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 86219.8us
I20260709 13:08:46.890530 25212 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:46.890930 28267 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:46.891510 28267 raft_consensus.cc:697] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:46.891937 25212 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:46.893275 27962 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:46.893436 28265 consensus_queue.cc:260] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.894483 28107 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:46.894867 28265 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:46.895144 28107 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:46.892531 28267 consensus_queue.cc:237] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:46.897265 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:46.901316 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:46.902654 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:46.903367 25244 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:46.904469 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:46.905591 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:46.845988 28220 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:46.906359 25238 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:46.906589 28220 raft_consensus.cc:697] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:46.895155 25140 catalog_manager.cc:5697] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:08:46.907640 28220 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:46.902714 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:46.905004 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:46.910713 25240 raft_consensus.cc:2393] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:46.890641 27884 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:46.924856 25312 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
W20260709 13:08:46.935593 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 80059.2us
I20260709 13:08:46.925582 25244 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:46.908180 25140 catalog_manager.cc:5697] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 3 to 4, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 4 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:46.940774 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:46.941835 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:46.942124 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:46.860042 28210 consensus_meta.cc:287] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.530s	user 0.012s	sys 0.018s
I20260709 13:08:46.942202 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:46.944144 25238 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:46.942550 25246 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:46.849758 28212 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:46.946724 25284 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:46.896309 27956 consensus_meta.cc:287] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.614s	user 0.018s	sys 0.013s
W20260709 13:08:46.948347 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:46.948727 28207 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:46.951568 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:46.953629 25395 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
W20260709 13:08:46.955258 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:46.853190 27948 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:46.883258 28128 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:46.959782 25237 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:46.959795 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:46.961902 25140 catalog_manager.cc:5697] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 4 to 5, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 5 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:46.974884 25239 raft_consensus.cc:2410] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:08:46.937748 28211 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:46.978251 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:46.955472 28070 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:46.974769 25140 catalog_manager.cc:5697] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:08:46.972409 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:08:47.002998 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 56352.3us
I20260709 13:08:47.006099 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:47.014966 25942 raft_consensus.cc:2393] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 5: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:47.014865 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:46.921448 28221 consensus_queue.cc:1059] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20260709 13:08:46.960861 27938 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:47.035244 25940 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:47.015870 25943 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:47.032213 25209 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:08:47.032507 28035 consensus_meta.cc:287] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.688s	user 0.010s	sys 0.020s
I20260709 13:08:47.040911 25140 catalog_manager.cc:5697] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 5 to 6, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 6 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:47.017000 28227 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:47.048591 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:47.036257 28210 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:47.043344 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:46.965251 27948 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:46.957067 27956 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:47.063515 25939 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:47.056118 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:47.064946 25312 raft_consensus.cc:2376] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6. Re-sending same reply.
I20260709 13:08:46.972568 28128 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:47.070320 28210 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:47.071048 25241 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:47.072638 28211 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:47.073292 28211 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:47.074532 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:47.075250 25316 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:47.075989 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:47.076712 25945 raft_consensus.cc:2393] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:47.072757 28210 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:47.077486 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:47.078111 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:47.078219 25242 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:47.079319 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:47.079978 27956 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:47.077486 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:47.081360 25945 raft_consensus.cc:2376] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6. Re-sending same reply.
I20260709 13:08:47.082432 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:47.034274 28032 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:47.034595 28174 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:47.009836 28207 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:47.082847 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:47.148561 25943 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:47.285593 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:47.288623 25284 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:47.256117 25237 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
W20260709 13:08:47.214669 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:47.284672 25939 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:08:47.171546 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:47.101482 25945 raft_consensus.cc:2393] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:47.294466 25313 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:47.267808 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:47.268402 25314 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 13:08:47.246495 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:47.298687 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:47.088492 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:47.306290 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:47.152702 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 257279us
W20260709 13:08:47.269886 28275 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.224s	user 0.000s	sys 0.001s
I20260709 13:08:47.295436 25942 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:47.265581 25317 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
W20260709 13:08:47.286012 28279 log.cc:927] Time spent T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
W20260709 13:08:47.296936 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 295249us
I20260709 13:08:47.288837 25943 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:47.113657 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:47.324342 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:47.186954 25140 catalog_manager.cc:5697] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 2 to 5, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 5 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:47.314275 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:47.315829 28207 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:47.118342 28108 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:47.321444 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:47.322566 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:47.125205 28016 consensus_meta.cc:287] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.524s	user 0.017s	sys 0.008s
I20260709 13:08:47.334813 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:47.050441 27925 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:47.328451 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:47.227146 28070 raft_consensus.cc:697] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:47.118803 28221 raft_consensus.cc:3055] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:47.336576 25944 raft_consensus.cc:2410] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 9 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260709 13:08:47.340057 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 51510.6us
I20260709 13:08:47.344652 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:47.348302 25321 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:47.343693 25140 catalog_manager.cc:5697] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 2 to 5, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 5 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:47.357152 25313 raft_consensus.cc:2368] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8.
I20260709 13:08:47.359182 25319 raft_consensus.cc:2368] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7.
I20260709 13:08:47.362473 25140 catalog_manager.cc:5697] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 2 to 5, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 5 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:47.366391 25284 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:47.360030 25240 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:47.367573 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:47.368336 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 74048.1us
I20260709 13:08:47.352391 28207 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:47.349388 28221 raft_consensus.cc:740] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:47.371045 25140 catalog_manager.cc:5697] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:47.353621 28032 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:47.373852 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:47.375788 25939 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 8: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:47.347530 28217 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:47.376292 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:47.378453 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:47.358707 28108 raft_consensus.cc:697] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:47.354514 28174 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:47.382040 28201 consensus_meta.cc:287] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.780s	user 0.013s	sys 0.006s
I20260709 13:08:47.382722 25944 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 6: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:47.370101 28207 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:47.391340 25321 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:47.342715 28016 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:47.397238 28221 consensus_queue.cc:260] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:47.423367 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:47.408656 25942 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:47.391340 28276 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:47.375005 28030 consensus_queue.cc:1059] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260709 13:08:47.412261 27925 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:47.413735 28108 consensus_queue.cc:237] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:47.416720 25212 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:47.391005 28228 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:08:47.441596 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:47.445807 25387 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:47.450906 27973 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:47.446656 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:47.452979 25939 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:08:47.440105 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:47.455080 25393 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 8: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:47.457773 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:47.458673 25943 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 7
I20260709 13:08:47.462364 25313 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:47.429530 28276 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:47.474916 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:47.476524 25359 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:47.477593 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:47.478492 25359 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:47.483026 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 84960.6us
I20260709 13:08:47.485955 25359 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:47.488804 25390 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 13:08:47.489976 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:47.490813 25390 raft_consensus.cc:2410] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 11 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 13:08:47.491863 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:47.492569 25394 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:47.493355 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:47.470851 27973 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:47.494156 28016 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:47.494287 27973 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:47.495345 27962 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:47.495738 28285 log.cc:927] Time spent T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 13:08:47.498437 27925 consensus_queue.cc:1059] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:08:47.499738 27925 raft_consensus.cc:3055] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:47.500063 27925 raft_consensus.cc:740] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:47.466828 28107 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:47.501539 25313 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8. Ops: [5.7-7.8]
I20260709 13:08:47.442646 28217 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:47.502584 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:47.503680 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:47.504603 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:47.505789 25314 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:47.506251 28032 raft_consensus.cc:3055] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:47.506639 28032 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:47.506734 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:47.507277 25129 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet eda233ace2a24b379fef320a5f610c7a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 46de6919f85341c6a9b574dff112e35a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:47.507731 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:47.507567 28032 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:47.508838 27941 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:47.509531 28286 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260709 13:08:47.510663 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:47.511444 25320 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 8: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:47.512042 28174 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:47.484038 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:47.508857 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:47.508883 28128 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:47.482965 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:47.539047 25387 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:47.514169 25209 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:47.539911 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:47.461512 28228 raft_consensus.cc:697] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:47.540548 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:47.496830 27973 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:47.534978 25393 raft_consensus.cc:2410] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260709 13:08:47.543324 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 57478.3us
I20260709 13:08:47.544855 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:47.535952 25394 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:47.536604 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:47.547676 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:47.547853 25394 raft_consensus.cc:1240] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260709 13:08:47.538308 25321 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
W20260709 13:08:47.549734 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:47.548357 25237 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 8: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:47.512992 28087 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:47.552438 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:47.552563 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 65486.3us
I20260709 13:08:47.508882 28032 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:47.554929 25242 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:47.555796 25359 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:47.560597 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:47.561367 25243 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:47.564363 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:47.542959 28232 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:47.500903 27925 consensus_queue.cc:260] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:47.469391 28070 consensus_queue.cc:237] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:48.274986 25317 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1914) took 1701 ms (client timeout 2048 ms). Trace:
I20260709 13:08:47.582739 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:47.821862 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 106700us
I20260709 13:08:47.551795 27962 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:47.643354 25388 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 13:08:47.604759 25240 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260709 13:08:47.659776 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 707) took 1280 ms. Trace:
I20260709 13:08:48.377576 25240 pending_rounds.cc:85] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 3
W20260709 13:08:47.662000 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:47.579252 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 65138.6us
W20260709 13:08:48.378882 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:47.604527 25390 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:08:48.089218 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1913) took 1515 ms. Trace:
I20260709 13:08:48.371536 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:48.061661 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 505875us
I20260709 13:08:48.372339 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:48.372866 25243 consensus_meta.cc:287] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.768s	user 0.006s	sys 0.003s
W20260709 13:08:48.368412 25943 consensus_meta.cc:287] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.900s	user 0.007s	sys 0.004s
W20260709 13:08:48.367172 25317 rpcz_store.cc:269] 0709 13:08:46.573923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:46.574089 (+   166us) service_pool.cc:224] Handling call
0709 13:08:48.274698 (+1700609us) spinlock_profiling.cc:243] Waited 421 ms on lock 0x7b5400045954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:48.274856 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":969039744}
I20260709 13:08:47.551919 28271 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:47.554623 28221 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:47.643095 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:48.452136 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:48.466600 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:48.467263 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:48.454100 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 895768us
W20260709 13:08:48.026074 25908 leader_election.cc:336] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.656s (SENT)
I20260709 13:08:48.453229 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.468568 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:48.469269 25243 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
W20260709 13:08:47.986146 28108 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.013s	sys 0.028s
W20260709 13:08:48.454775 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 67665.1us
I20260709 13:08:48.469403 25318 raft_consensus.cc:2410] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:08:48.377628 25941 rpcz_store.cc:276] 0709 13:08:46.351068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:46.368907 (+ 17839us) service_pool.cc:224] Handling call
0709 13:08:47.631433 (+1262526us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b5400460fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:47.631601 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":916748160}
W20260709 13:08:47.974519 28267 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.021s	sys 0.018s
I20260709 13:08:48.383971 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:48.380271 25387 consensus_meta.cc:287] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.829s	user 0.011s	sys 0.000s
I20260709 13:08:48.378245 25240 pending_rounds.cc:107] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260709 13:08:48.483244 25240 raft_consensus.cc:2889] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:48.380661 25315 rpcz_store.cc:276] 0709 13:08:46.573499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:46.575046 (+  1547us) service_pool.cc:224] Handling call
0709 13:08:46.575398 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:46.747930 (+172532us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:48.088837 (+1340907us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b54000f8554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:48.089088 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":390363904}
I20260709 13:08:48.471454 25943 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:47.563102 28087 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:48.484182 25387 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:48.487322 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1411) took 1072 ms. Trace:
W20260709 13:08:47.884295 28278 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.185s	user 0.000s	sys 0.003s
I20260709 13:08:47.820393 28222 raft_consensus.cc:3055] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:47.934738 28276 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:47.644549 28151 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:48.485430 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.509852 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1407) took 1096 ms. Trace:
W20260709 13:08:48.450726 27941 consensus_meta.cc:287] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.873s	user 0.010s	sys 0.001s
I20260709 13:08:48.382056 25320 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:08:48.489344 25313 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 8: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:47.619148 28286 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:47.642967 28280 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:47.642786 27948 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:47.641202 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:47.646158 27960 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:48.535543 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:48.534056 25284 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:48.375906 28207 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:48.490945 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.496169 28222 raft_consensus.cc:740] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:47.643460 28212 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:48.487932 25387 rpcz_store.cc:276] 0709 13:08:47.415158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:47.467716 (+ 52558us) service_pool.cc:224] Handling call
0709 13:08:48.487212 (+1019496us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:48.484149 25943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 581) took 1061 ms. Trace:
I20260709 13:08:48.502595 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.545861 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:47.816134 28211 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:48.546303 25317 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:08:48.479282 28108 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.028s	user 0.021s	sys 0.043s
W20260709 13:08:48.478524 28220 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.832s	user 0.014s	sys 0.007s
I20260709 13:08:47.642292 28170 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:48.507179 28276 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:48.491035 28174 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:48.490535 27813 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:48.480496 28267 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.098s	user 0.023s	sys 0.039s
W20260709 13:08:48.510962 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 130801us
I20260709 13:08:48.516403 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.563052 25318 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:48.516667 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1579) took 1014 ms. Trace:
I20260709 13:08:48.563486 25240 rpcz_store.cc:276] 0709 13:08:47.501911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:47.513071 (+ 11160us) service_pool.cc:224] Handling call
0709 13:08:47.542302 (+ 29231us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:48.516432 (+974130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:48.516613 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:48.480072 28030 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:48.450809 28287 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:48.490594 28095 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:48.494628 28087 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:48.529348 28181 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:47.644245 28228 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:48.530026 28292 log.cc:927] Time spent T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.450s	user 0.004s	sys 0.000s
I20260709 13:08:48.529762 28151 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:48.510241 25389 rpcz_store.cc:276] 0709 13:08:47.413855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:47.457070 (+ 43215us) service_pool.cc:224] Handling call
0709 13:08:48.509793 (+1052723us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:48.531217 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.520524 25321 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:48.537701 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.538327 27925 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:48.538408 25394 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:48.537883 25393 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 13:08:48.539106 28217 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:48.539332 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:48.591926 25395 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:48.532995 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 1995) took 1020 ms. Trace:
I20260709 13:08:47.578908 28210 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:48.451272 28271 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:48.599897 28271 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:48.545782 25943 rpcz_store.cc:276] 0709 13:08:47.422389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:47.457453 (+ 35064us) service_pool.cc:224] Handling call
0709 13:08:48.484081 (+1026628us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:48.384953 28232 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:48.531749 28286 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:48.550091 25212 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:48.550604 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:48.605064 25313 raft_consensus.cc:2393] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:48.531240 28265 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:48.530685 28186 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:48.559424 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.568248 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:48.542439 27960 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:48.551151 28220 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.905s	user 0.014s	sys 0.007s
I20260709 13:08:48.547735 28280 raft_consensus.cc:697] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:48.585846 25315 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 8: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:48.587117 28234 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:48.562493 25938 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:48.542920 28179 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 13:08:48.595667 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:48.563398 28212 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:48.572450 27813 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:48.598673 25390 raft_consensus.cc:2410] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 13:08:48.607865 25209 leader_election.cc:400] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7.
I20260709 13:08:48.570966 28276 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:48.677912 25209 leader_election.cc:403] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 13:08:48.627599 25387 raft_consensus.cc:2393] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 7: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:48.596907 25320 rpcz_store.cc:276] 0709 13:08:47.512899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:47.514635 (+  1736us) service_pool.cc:224] Handling call
0709 13:08:48.532734 (+1018099us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:08:48.625327 25358 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:48.571074 28174 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:48.671873 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.682322 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:48.684274 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 134254us
I20260709 13:08:48.571213 28170 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:48.692261 25321 raft_consensus.cc:2368] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 6. Current term is 7.
I20260709 13:08:48.694298 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:48.698751 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:48.704644 25312 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 8: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:08:48.685343 28298 log.cc:927] Time spent T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 13:08:48.705785 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:48.679195 27960 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:48.593055 28095 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:48.686455 28032 consensus_meta.cc:287] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.107s	user 0.009s	sys 0.002s
I20260709 13:08:48.625805 28286 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:48.709594 25315 raft_consensus.cc:2376] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8. Re-sending same reply.
I20260709 13:08:48.596371 28030 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:48.748770 28030 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:48.748564 28286 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:48.596566 28181 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:48.750675 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.686024 28108 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:48.593639 28287 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:48.716514 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1400) took 1440 ms. Trace:
I20260709 13:08:48.716653 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.672283 28186 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:48.757248 25392 rpcz_store.cc:276] 0709 13:08:47.270359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:47.282809 (+ 12450us) service_pool.cc:224] Handling call
0709 13:08:47.283254 (+   445us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:47.286512 (+  3258us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:08:47.286545 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:47.313381 (+ 26836us) log.cc:844] Serialized 26 byte log entry
0709 13:08:47.550993 (+237612us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:47.551039 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:47.551099 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:48.529479 (+978380us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:48.706216 (+176737us) raft_consensus.cc:1685] finished
0709 13:08:48.706237 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:48.710673 (+  4436us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9944,"raft.queue_time_us":6848,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":17654,"spinlock_wait_cycles":862384896,"thread_start_us":193352,"threads_started":1,"wal-append.queue_time_us":539903}
I20260709 13:08:48.628436 28210 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:48.718936 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.688601 28234 raft_consensus.cc:697] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:48.768214 28234 consensus_queue.cc:237] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:48.719787 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 572) took 1677 ms. Trace:
I20260709 13:08:48.722020 25395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1403) took 1368 ms. Trace:
W20260709 13:08:48.723500 28302 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.191s	user 0.000s	sys 0.000s
I20260709 13:08:48.725263 25313 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 4.
I20260709 13:08:48.726577 28232 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:48.728479 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1777) took 1725 ms. Trace:
I20260709 13:08:48.730528 28280 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:48.716915 28267 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 13:08:48.731328 28299 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.637s	user 0.000s	sys 0.003s
I20260709 13:08:48.732933 28174 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:48.736842 28276 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:48.738390 28296 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.659s	user 0.003s	sys 0.000s
I20260709 13:08:48.738441 27813 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:48.744733 25316 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:48.746824 28151 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:48.751855 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 44979.2us
I20260709 13:08:48.648046 28265 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:48.717798 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 133410us
I20260709 13:08:48.757759 25321 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:48.718336 25212 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:48.763648 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:48.764335 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.765681 25312 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:48.766311 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:48.773276 25395 rpcz_store.cc:276] 0709 13:08:47.354130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:47.354297 (+   167us) service_pool.cc:224] Handling call
0709 13:08:47.366507 (+ 12210us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:48.592525 (+1226018us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:48.721685 (+129160us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54001e3a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:48.721944 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2817615232}
I20260709 13:08:48.792733 25938 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
W20260709 13:08:48.837721 25938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 571) took 1865 ms (client timeout 2210 ms). Trace:
I20260709 13:08:48.628880 28222 consensus_queue.cc:260] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:48.852602 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 250653us
I20260709 13:08:48.706072 27962 raft_consensus.cc:697] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:48.853971 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.854140 28181 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:48.853878 27962 consensus_queue.cc:237] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:48.855530 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.855753 28181 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:48.856168 28181 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:48.856503 28181 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:48.799417 25245 rpcz_store.cc:276] 0709 13:08:47.002635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:47.003114 (+   479us) service_pool.cc:224] Handling call
0709 13:08:47.034275 (+ 31161us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:47.036768 (+  2493us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:47.036798 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:47.053007 (+ 16209us) log.cc:844] Serialized 26 byte log entry
0709 13:08:47.604160 (+551153us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:08:47.604198 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:47.604283 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:48.728293 (+1124010us) raft_consensus.cc:1685] finished
0709 13:08:48.728311 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:48.728426 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":432887,"threads_started":1,"wal-append.queue_time_us":1559776}
I20260709 13:08:48.771569 25937 rpcz_store.cc:276] 0709 13:08:47.004526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:47.036369 (+ 31843us) service_pool.cc:224] Handling call
0709 13:08:47.045426 (+  9057us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:47.045840 (+   414us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:47.045871 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:47.061845 (+ 15974us) log.cc:844] Serialized 26 byte log entry
0709 13:08:47.327144 (+265299us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:08:48.013601 (+686457us) log.cc:844] Serialized 14 byte log entry
0709 13:08:48.681964 (+668363us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:48.682056 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:48.682102 (+    46us) raft_consensus.cc:1685] finished
0709 13:08:48.682113 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:48.682228 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":378399,"prepare.queue_time_us":167682,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":24,"raft.queue_time_us":3518,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":10596,"thread_start_us":735103,"threads_started":3,"wal-append.queue_time_us":242698}
I20260709 13:08:48.833524 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:48.849115 28286 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:48.857745 25238 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:48.791149 28030 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:48.839185 25315 raft_consensus.cc:2410] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:08:48.818526 28108 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7.
I20260709 13:08:48.849395 28312 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:48.858752 28030 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:48.859047 28312 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:48.859623 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.860348 25315 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:08:48.838284 25938 rpcz_store.cc:269] 0709 13:08:46.972726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:47.042414 (+ 69688us) service_pool.cc:224] Handling call
0709 13:08:47.640888 (+598474us) spinlock_profiling.cc:243] Waited 1.78 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:48.837648 (+1196760us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4101504}
I20260709 13:08:48.853114 28222 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:48.861577 28030 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:48.865556 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.866655 25315 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 8: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:48.867549 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:48.868288 25315 raft_consensus.cc:2393] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:48.869055 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.846190 25320 raft_consensus.cc:3055] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:08:48.872192 27813 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:08:48.872906 27813 consensus_queue.cc:1243] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer f646dfd532974dceadfe00005d2bbf59 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260709 13:08:48.873449 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 162308us
I20260709 13:08:48.874872 25212 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:48.875211 27813 consensus_queue.cc:1059] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:08:48.876226 28108 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:48.877025 28174 raft_consensus.cc:3055] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:08:48.877449 28174 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:48.878281 28174 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:48.879745 28108 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:08:48.881405 25212 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:48.870693 25320 raft_consensus.cc:740] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:48.852917 25241 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:48.857118 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:48.899884 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:48.899384 25358 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:48.901650 25313 raft_consensus.cc:2376] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8. Re-sending same reply.
I20260709 13:08:48.904167 25358 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:48.901100 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 216645us
I20260709 13:08:48.901765 25209 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:48.905609 25244 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:48.903754 25320 consensus_queue.cc:260] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:48.907336 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1405) took 1517 ms. Trace:
I20260709 13:08:48.907783 25320 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:48.908869 28210 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:48.907737 25386 rpcz_store.cc:276] 0709 13:08:47.389778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:47.390015 (+   237us) service_pool.cc:224] Handling call
0709 13:08:47.394008 (+  3993us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:08:47.394441 (+   433us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:47.394468 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:08:47.395090 (+   622us) log.cc:844] Serialized 52 byte log entry
0709 13:08:47.553310 (+158220us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:48.610194 (+1056884us) log.cc:844] Serialized 14 byte log entry
0709 13:08:48.610485 (+   291us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:48.610584 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:48.730615 (+120031us) spinlock_profiling.cc:243] Waited 361 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:48.907031 (+176416us) raft_consensus.cc:1685] finished
0709 13:08:48.907140 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:48.907277 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1142539,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":16460,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":21425,"spinlock_wait_cycles":830354304,"thread_start_us":392000,"threads_started":2,"wal-append.queue_time_us":680618}
I20260709 13:08:48.910692 28287 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:48.913144 28212 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:48.914254 28170 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:48.917320 28151 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:48.917738 28151 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:48.918115 28151 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:08:48.918533 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:48.920868 28265 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:48.904881 25242 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:48.907806 25212 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:48.919903 27960 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:48.923059 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:48.924088 25358 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:48.929147 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1401) took 1631 ms. Trace:
W20260709 13:08:48.929677 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:48.933696 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:48.935016 25315 raft_consensus.cc:2410] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 13:08:48.902638 28108 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:08:48.946458 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:48.929577 25391 rpcz_store.cc:276] 0709 13:08:47.297670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:47.297778 (+   108us) service_pool.cc:224] Handling call
0709 13:08:47.314371 (+ 16593us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:08:47.332208 (+ 17837us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:47.332272 (+    64us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:08:47.365902 (+ 33630us) log.cc:844] Serialized 78 byte log entry
0709 13:08:47.553005 (+187103us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:47.553043 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:47.553105 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:48.737650 (+1184545us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:48.923006 (+185356us) raft_consensus.cc:1685] finished
0709 13:08:48.923026 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:48.929056 (+  6030us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17361,"raft.queue_time_us":28688,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":11113,"spinlock_wait_cycles":862091136,"thread_start_us":164079,"threads_started":1,"wal-append.queue_time_us":596222}
I20260709 13:08:48.970500 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:48.973510 25245 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:48.932147 28287 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:48.979401 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 71656.8us
I20260709 13:08:48.921483 28286 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.049s
W20260709 13:08:48.984100 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 82399.1us
I20260709 13:08:48.985731 25287 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:48.947520 28311 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:48.933807 28267 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:48.969769 28276 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:48.991474 28286 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:48.989171 28179 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:48.949397 28108 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:48.948307 28030 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:48.996006 25209 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:49.004980 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:49.028304 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 1.00221e+06us
I20260709 13:08:48.950511 28232 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:48.981679 28309 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:49.031312 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:49.043870 25908 leader_election.cc:336] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.521s (SENT)
I20260709 13:08:49.044508 25908 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:49.036600 25246 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:49.047467 25284 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:49.053545 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 67935.9us
W20260709 13:08:49.053902 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:48.993408 28016 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:49.021551 28312 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:49.058321 25945 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 714) took 1982 ms. Trace:
I20260709 13:08:49.028409 28217 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:49.073474 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 83562.1us
I20260709 13:08:49.028558 28108 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:48.984862 28287 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.077562 25938 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: []
W20260709 13:08:49.081271 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:49.083604 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:49.035115 27973 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:49.094446 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:49.096666 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:49.097420 25943 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:49.075680 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:49.067147 25358 connection.cc:256] RPC call timeout handler was delayed by -1.8649025390000133s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1841s for a response
W20260709 13:08:49.099485 25358 connection.cc:256] RPC call timeout handler was delayed by -1.7605025389999582s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1957s for a response
I20260709 13:08:49.058714 25945 rpcz_store.cc:276] 0709 13:08:47.076341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:47.295092 (+218751us) service_pool.cc:224] Handling call
0709 13:08:47.295400 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:47.295648 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:08:47.368318 (+ 72670us) log.cc:844] Serialized 14 byte log entry
0709 13:08:48.366848 (+998530us) log.cc:844] Serialized 14 byte log entry
0709 13:08:48.763915 (+397067us) spinlock_profiling.cc:243] Waited 5.39 ms on lock 0x7b18003a7500. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:08:48.899760 (+135845us) spinlock_profiling.cc:243] Waited 57.9 ms on lock 0x7b18003a7500. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b3fc0a 00007f297d27f219 00007f2973bc7a40 00007f2973bc24e4 00007f2973bc2455 00007f2976caa060 00007f2976cbc716 00007f2976ccd287 00007f2976ccd208
0709 13:08:49.048220 (+148460us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:49.048279 (+    59us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:08:49.048333 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:49.057702 (+  9369us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:49.058267 (+   565us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":193211,"prepare.queue_time_us":176353,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":75,"raft.queue_time_us":1095037,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":197,"spinlock_wait_cycles":145472768,"thread_start_us":692892,"threads_started":3,"wal-append.queue_time_us":1142332}
I20260709 13:08:49.101858 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:48.986485 28087 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:49.046489 28220 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:49.021409 28311 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.077797 28222 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:49.109360 25943 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:08:49.057533 27997 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:08:49.111244 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 63893.9us
I20260709 13:08:49.052184 28317 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:08:49.108103 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 43571.2us
I20260709 13:08:49.118958 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:49.033588 28030 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.120239 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:49.120517 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:49.121038 25945 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:49.099964 28070 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:49.123700 25212 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:49.112700 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 50388.3us
I20260709 13:08:49.086683 28321 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:49.105119 28309 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:49.051318 28179 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:49.123685 25942 pending_rounds.cc:85] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 4
I20260709 13:08:49.136165 25942 pending_rounds.cc:107] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260709 13:08:49.136556 25942 raft_consensus.cc:2889] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:08:49.128401 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:49.129884 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 140344us
I20260709 13:08:49.104482 28314 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:49.133708 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:49.138798 25943 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:49.135123 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:49.134523 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:49.139885 25938 raft_consensus.cc:2410] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260709 13:08:49.144539 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:49.056962 28232 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:49.161480 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 165558us
I20260709 13:08:49.087688 28217 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.116796 27956 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:49.085651 28016 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.163790 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:49.164501 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:49.123812 28070 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:49.165860 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:49.168568 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:49.175276 25393 raft_consensus.cc:2410] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 13:08:49.169363 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:49.176528 25394 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 8: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:49.104581 27973 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.172887 25391 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:08:49.164635 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:49.131111 28317 raft_consensus.cc:697] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:49.173034 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:49.177381 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:49.186260 25945 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:49.186344 25387 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:49.187251 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:49.187731 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:49.180032 25209 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:49.163251 28310 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:49.118456 28220 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.201057 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:49.201522 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.201824 25245 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:49.202299 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.203742 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.193430 25941 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:49.126641 28222 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.207654 25242 raft_consensus.cc:2410] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260709 13:08:49.127053 27997 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.198216 25359 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:49.209556 25392 raft_consensus.cc:1275] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:08:49.211010 25943 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:49.185434 28070 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:08:49.213647 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 96053.8us
I20260709 13:08:49.216253 25314 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:49.217875 25212 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:49.216998 25246 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 3.
I20260709 13:08:49.188303 27960 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:49.206152 28317 consensus_queue.cc:237] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.214035 28309 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:49.160102 28321 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:49.223096 28293 log.cc:927] Time spent T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 1.128s	user 0.005s	sys 0.000s
I20260709 13:08:49.244526 25320 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 7. (term mismatch)
I20260709 13:08:49.244899 25320 pending_rounds.cc:85] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 6
I20260709 13:08:49.218945 28212 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:49.245105 25320 pending_rounds.cc:107] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20260709 13:08:49.248057 25320 raft_consensus.cc:2889] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:49.271018 25359 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:49.246186 28271 consensus_meta.cc:287] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.646s	user 0.011s	sys 0.000s
W20260709 13:08:49.279692 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 99716.4us
I20260709 13:08:49.281957 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:49.246878 28334 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:08:49.286794 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 125472us
I20260709 13:08:49.251107 28212 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:49.290891 28087 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:49.291306 28030 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:49.241428 28181 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:49.293596 28181 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:49.294238 28181 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:49.294526 28217 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:49.281589 28186 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:49.295511 28325 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 13:08:49.296033 25244 raft_consensus.cc:2393] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:08:49.296968 28328 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 13:08:49.279212 27945 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:49.304046 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:49.312741 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:49.314751 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 206947us
I20260709 13:08:49.298401 28174 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:49.319849 25908 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:49.276508 27960 consensus_queue.cc:1059] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260709 13:08:49.282464 28309 raft_consensus.cc:697] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:49.310539 27945 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:49.330181 27945 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:49.331377 28220 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:49.332438 28220 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:49.307566 28287 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:49.334568 28324 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
W20260709 13:08:49.336058 28322 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 13:08:49.337442 27960 raft_consensus.cc:3055] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:49.337704 25936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 709) took 2262 ms. Trace:
I20260709 13:08:49.337955 27960 raft_consensus.cc:740] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:49.339140 27960 consensus_queue.cc:260] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:49.346251 27962 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:49.346799 27962 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.348080 28016 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:49.298797 28212 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:49.347713 28151 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.349741 25390 raft_consensus.cc:1240] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9. Ops: []
I20260709 13:08:49.298683 28280 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:49.353354 28331 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:08:49.353425 28280 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:49.354529 28095 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:49.355247 25911 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:49.355659 28280 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:49.355957 28331 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:08:49.329102 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:49.344746 25358 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:49.356349 28331 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:49.281505 28271 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:49.338171 25936 rpcz_store.cc:276] 0709 13:08:47.074750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:47.109078 (+ 34328us) service_pool.cc:224] Handling call
0709 13:08:47.148428 (+ 39350us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:47.162196 (+ 13768us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:47.162242 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:47.307493 (+145251us) log.cc:844] Serialized 26 byte log entry
0709 13:08:47.470424 (+162931us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:47.470506 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:47.470593 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:49.062522 (+1591929us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:49.337280 (+274758us) raft_consensus.cc:1685] finished
0709 13:08:49.337302 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:49.337509 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":19302,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":109,"spinlock_wait_cycles":827988864,"thread_start_us":112541,"threads_started":1,"wal-append.queue_time_us":637024}
I20260709 13:08:49.366299 28271 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:49.366591 25944 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 577) took 2047 ms. Trace:
I20260709 13:08:49.367451 27997 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:49.368163 25243 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:08:49.368410 28331 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:49.332907 28220 raft_consensus.cc:697] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:49.340838 27960 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:08:49.371984 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 154153us
I20260709 13:08:49.361912 28087 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:49.377247 28087 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.379549 28087 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:49.348986 27962 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:49.382197 28151 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:49.382500 28232 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:49.328328 28336 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:49.383553 28336 raft_consensus.cc:697] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:49.346263 28321 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:49.385789 28222 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:49.385955 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.386617 25243 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
W20260709 13:08:49.388132 28221 consensus_meta.cc:287] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.913s	user 0.010s	sys 0.001s
I20260709 13:08:49.389055 25238 raft_consensus.cc:1275] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 13:08:49.389438 25238 pending_rounds.cc:85] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 3
I20260709 13:08:49.389678 25238 pending_rounds.cc:107] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 13:08:49.389950 25238 raft_consensus.cc:2889] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:08:49.390708 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 46059.1us
I20260709 13:08:49.391439 25358 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:49.384577 28336 consensus_queue.cc:237] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.388247 27945 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:49.397961 28309 consensus_queue.cc:237] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.399370 27945 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:49.366989 25944 rpcz_store.cc:276] 0709 13:08:47.319131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:47.415364 (+ 96233us) service_pool.cc:224] Handling call
0709 13:08:47.646982 (+231618us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:47.797115 (+150133us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:08:47.968285 (+171170us) log.cc:844] Serialized 14 byte log entry
0709 13:08:49.064537 (+1096252us) log.cc:844] Serialized 14 byte log entry
0709 13:08:49.087127 (+ 22590us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:49.087230 (+   103us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:08:49.087299 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:49.087423 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:49.366436 (+279013us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b5400460fd0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:49.366538 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":353811,"prepare.queue_time_us":112948,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":83,"raft.queue_time_us":501284,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":57435,"spinlock_wait_cycles":496517888,"thread_start_us":628747,"threads_started":2,"wal-append.queue_time_us":764697}
I20260709 13:08:49.401957 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:49.402832 27813 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:49.403510 27813 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:49.407814 28212 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.393949 28221 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:49.411525 28221 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:49.413595 28221 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:49.414961 28108 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:49.417517 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:49.417903 25212 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:49.418983 25392 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 13:08:49.419983 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 100354us
I20260709 13:08:49.423610 25245 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:08:49.416913 28108 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:49.425594 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 143683us
I20260709 13:08:49.429210 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:49.381018 28286 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:49.433634 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:49.434777 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:49.439845 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:49.437042 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:49.441632 25390 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:49.443872 27962 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:49.439412 25908 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:49.445833 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:49.447014 25386 raft_consensus.cc:2410] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260709 13:08:49.449595 25389 raft_consensus.cc:2393] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 7: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:49.428376 28179 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:49.488747 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:49.491298 25314 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 1991) took 2000 ms (client timeout 2185 ms). Trace:
I20260709 13:08:49.491715 25244 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:08:49.488984 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 49267.9us
I20260709 13:08:49.444305 28220 consensus_queue.cc:237] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.489643 25908 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:49.439172 28327 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 13:08:49.449035 28108 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:49.508599 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:49.509059 28108 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:49.509582 28108 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:49.510210 25936 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:49.490178 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.426353 28221 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:49.493096 25911 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:49.511271 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 372672us
W20260709 13:08:49.493115 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 365063us
I20260709 13:08:49.497126 25359 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:49.491729 25314 rpcz_store.cc:269] 0709 13:08:47.491230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:47.506496 (+ 15266us) service_pool.cc:224] Handling call
0709 13:08:49.481060 (+1974564us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b540002c6d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:49.491244 (+ 10184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3267287424}
I20260709 13:08:49.499902 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.501098 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.504160 25943 raft_consensus.cc:1240] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:49.506227 27973 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:49.515522 25241 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:08:49.516536 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:49.517076 25241 raft_consensus.cc:2376] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7. Re-sending same reply.
I20260709 13:08:49.518019 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:49.518146 28232 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:49.518508 25241 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:49.507355 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.518652 25908 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:49.518951 27956 raft_consensus.cc:697] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:49.519363 25237 raft_consensus.cc:2410] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:08:49.519862 25908 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:49.520202 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:49.527705 25908 leader_election.cc:336] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.767s (SENT)
I20260709 13:08:49.528173 25908 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:49.507983 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:49.529470 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.518332 28210 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:08:49.520581 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.520925 28212 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:49.521306 28341 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:49.522012 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:49.537554 25245 raft_consensus.cc:2376] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6. Re-sending same reply.
I20260709 13:08:49.522280 28327 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:49.522704 28070 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.512579 28276 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:49.506937 28286 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:49.507850 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 406) took 1053 ms. Trace:
I20260709 13:08:49.518586 28232 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:49.497290 28326 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:49.492832 28087 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:49.539428 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.541430 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:49.533954 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:49.539587 25359 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:49.548136 28318 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.434s	user 0.004s	sys 0.000s
I20260709 13:08:49.540771 27960 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:49.548153 28311 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:49.549455 28286 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:49.549609 27813 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:08:49.537253 28341 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:49.550839 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:49.551760 28350 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 0.119s	user 0.000s	sys 0.002s
I20260709 13:08:49.552043 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:49.557331 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 211229us
I20260709 13:08:49.583062 25284 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:49.581375 25393 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:49.584446 25284 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:49.551782 25358 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:49.561545 25941 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:49.541206 25318 rpcz_store.cc:276] 0709 13:08:48.454450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:48.574631 (+120181us) service_pool.cc:224] Handling call
0709 13:08:48.574976 (+   345us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:08:48.648996 (+ 74020us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:48.673139 (+ 24143us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:08:48.708104 (+ 34965us) log.cc:844] Serialized 52 byte log entry
0709 13:08:49.032757 (+324653us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:49.057568 (+ 24811us) log.cc:844] Serialized 14 byte log entry
0709 13:08:49.104134 (+ 46566us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:49.104212 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:49.472550 (+368338us) spinlock_profiling.cc:243] Waited 8.64 ms on lock 0x7b18003ce860. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b3fc0a 00007f297d27f219 00007f297ab8658f 00007f2973bc5571 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd
0709 13:08:49.498112 (+ 25562us) spinlock_profiling.cc:243] Waited 2.53 ms on lock 0x7b18003ce860. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:08:49.507606 (+  9494us) raft_consensus.cc:1685] finished
0709 13:08:49.507627 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:49.507772 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":73303,"prepare.queue_time_us":43915,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":116982,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":16069,"spinlock_wait_cycles":25672704,"thread_start_us":206820,"threads_started":1,"wal-append.queue_time_us":353412}
I20260709 13:08:49.563643 25241 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:49.562345 27960 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:49.566370 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 53896.3us
I20260709 13:08:49.519884 27956 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.587533 25285 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:49.591112 25320 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:49.591353 25285 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:49.592995 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:49.568120 25287 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:49.569229 28338 raft_consensus.cc:1064] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:08:49.569800 28210 consensus_queue.cc:1059] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:08:49.537171 27997 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:49.569741 28030 consensus_queue.cc:1059] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:08:49.571269 28326 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.561282 28232 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:49.599691 28179 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:49.586796 28314 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:49.573320 25244 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:08:49.586861 27960 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:49.539202 28331 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:49.573841 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 46190.2us
I20260709 13:08:49.574414 25359 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:49.540145 27973 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:08:49.589246 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:49.589826 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:49.589884 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:49.593729 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:49.593171 28267 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:49.601737 25358 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:49.602512 28016 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:49.600468 27997 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:49.603102 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:49.603189 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:49.600775 25287 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:49.604542 25319 raft_consensus.cc:1275] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:08:49.582950 28341 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:49.606846 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 403) took 1337 ms. Trace:
I20260709 13:08:49.606197 28336 consensus_queue.cc:1059] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.006s
I20260709 13:08:49.608912 28179 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:49.613662 25316 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:49.594957 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:49.618510 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.606492 28087 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:08:49.621414 28326 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:08:49.623504 25940 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:49.608737 28234 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:08:49.651005 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:49.624447 25359 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:49.632083 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 202608us
I20260709 13:08:49.633100 25358 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:49.606611 28174 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:08:49.643829 28179 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:49.637257 27997 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:49.637898 25317 rpcz_store.cc:276] 0709 13:08:48.269681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:48.552711 (+283030us) service_pool.cc:224] Handling call
0709 13:08:48.573671 (+ 20960us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:48.849356 (+275685us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:08:49.078523 (+229167us) log.cc:844] Serialized 14 byte log entry
0709 13:08:49.535395 (+456872us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:49.535864 (+   469us) log.cc:844] Serialized 26 byte log entry
0709 13:08:49.536070 (+   206us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:49.536118 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:49.597917 (+ 61799us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b54000622d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:49.597960 (+    43us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:49.598003 (+    43us) raft_consensus.cc:1685] finished
0709 13:08:49.606630 (+  8627us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:49.606781 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":328790,"prepare.queue_time_us":59936,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":327953,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":3983,"spinlock_wait_cycles":574056832,"thread_start_us":334028,"threads_started":2,"wal-append.queue_time_us":271368}
I20260709 13:08:49.644202 28186 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:08:49.612722 28314 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:49.644919 28070 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:49.645483 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:49.645778 28032 raft_consensus.cc:3055] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:49.647826 25939 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260709 13:08:49.659454 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 58750.6us
I20260709 13:08:49.660185 25238 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:49.633268 28016 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:49.649732 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:49.625988 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:49.652683 25359 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:49.661485 25244 raft_consensus.cc:2410] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:08:49.654999 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:49.631819 27962 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:49.659603 28032 raft_consensus.cc:740] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:49.667246 28286 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:49.667348 28312 consensus_queue.cc:1059] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:08:49.667505 28032 consensus_queue.cc:260] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:49.654325 28095 raft_consensus.cc:3055] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:08:49.668588 28095 raft_consensus.cc:740] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:49.669402 28228 raft_consensus.cc:3055] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:49.669816 28228 raft_consensus.cc:740] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:49.670629 28228 consensus_queue.cc:260] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:49.671622 27813 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:49.672137 27813 raft_consensus.cc:697] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:49.658332 28336 consensus_queue.cc:1243] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer 2329e428ebbf484fa3ea9452133df567 log is divergent from this leader: its last log entry 7.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:08:49.673472 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.674042 28329 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:49.674288 25240 raft_consensus.cc:2393] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 8: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:49.674640 28210 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:49.675184 28210 raft_consensus.cc:697] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:49.663218 28016 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:49.675870 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.676120 28210 consensus_queue.cc:237] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:49.677034 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.656265 28271 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:49.656086 28212 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:49.674301 25239 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:49.679174 28212 raft_consensus.cc:697] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:49.663756 25911 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:49.680459 28326 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:49.680152 28212 consensus_queue.cc:237] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.673043 27813 consensus_queue.cc:237] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:49.642932 28276 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:08:49.675474 25240 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 13:08:49.673503 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.681866 25212 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:49.680859 28326 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:49.683449 28333 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:49.684141 25244 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:49.668864 28032 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:49.681353 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:49.685875 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:49.688221 25316 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:49.689603 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:49.655014 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:49.708343 25317 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:49.660135 28314 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:49.719151 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.719962 25245 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:49.734936 28271 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:49.735726 28271 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:49.738463 28271 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:49.739231 28271 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:08:49.737291 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 82975.3us
I20260709 13:08:49.740702 27945 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:08:49.741988 27945 raft_consensus.cc:3055] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:49.742139 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.742600 27945 raft_consensus.cc:740] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:49.739686 28271 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:49.745730 28271 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:49.751077 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:49.751282 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:49.751947 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:49.669430 28095 consensus_queue.cc:260] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:49.753324 28095 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:49.707907 28333 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:49.764357 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 104327us
W20260709 13:08:49.756631 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 222603us
I20260709 13:08:49.705274 27962 raft_consensus.cc:697] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:49.708925 28326 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:49.777552 28351 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 13:08:49.779039 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:08:49.781996 25908 leader_election.cc:336] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.593s (SENT)
I20260709 13:08:49.804000 25908 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:49.804000 25936 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:49.805943 25209 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:49.718144 28070 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:49.814599 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:49.817443 25285 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:49.819839 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:49.820645 25245 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:49.780249 28333 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:49.827263 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:49.827369 25209 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:49.827898 25938 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
W20260709 13:08:49.837308 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 177148us
I20260709 13:08:49.838423 25287 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:49.842092 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:49.842999 25938 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:49.843115 25212 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:49.818490 28347 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:49.818643 28070 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:49.847404 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:49.848923 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 66970us
I20260709 13:08:49.851347 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:49.854030 25209 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:49.857472 25938 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:49.850749 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:49.871889 25945 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:49.779455 28016 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:49.859737 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:49.828732 28327 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:08:49.877036 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:49.818073 28234 consensus_queue.cc:1059] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260709 13:08:49.826550 27925 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:49.875537 25313 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: []
W20260709 13:08:49.858093 25908 leader_election.cc:336] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.228s (SENT)
I20260709 13:08:49.888914 25908 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:49.875629 25941 raft_consensus.cc:2376] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5. Re-sending same reply.
I20260709 13:08:49.813771 28317 consensus_queue.cc:1059] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
I20260709 13:08:49.878979 27962 consensus_queue.cc:237] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:49.855185 28310 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:49.855505 28347 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:49.859246 28070 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:49.895314 28345 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
I20260709 13:08:49.858624 28030 consensus_queue.cc:1059] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 13:08:49.915232 25936 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:49.909363 25909 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:49.841064 28334 consensus_queue.cc:1048] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.061s
W20260709 13:08:49.917753 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 92315.2us
I20260709 13:08:49.902631 28347 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:49.922269 25285 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:49.923615 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:49.953287 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 95192.8us
I20260709 13:08:49.908185 28286 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 13:08:49.962006 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:49.877555 28217 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.034s
I20260709 13:08:49.961941 25941 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:08:49.963292 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:49.912642 28331 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:49.912122 28327 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 13:08:49.953533 28356 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
W20260709 13:08:49.971153 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 48975.9us
I20260709 13:08:49.954836 28311 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:49.926486 28310 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:49.905390 28313 consensus_queue.cc:1059] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.056s
I20260709 13:08:49.954125 27925 raft_consensus.cc:3055] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:49.914724 27973 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20260709 13:08:49.989703 25909 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:49.979228 28095 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:49.978374 28220 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 13:08:49.998028 28362 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 13:08:49.994032 28217 raft_consensus.cc:3055] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:08:50.022872 28217 raft_consensus.cc:740] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:50.023886 28217 consensus_queue.cc:260] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:50.014422 28360 raft_consensus.cc:3055] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:50.025049 28360 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:50.008946 28341 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:49.992842 28353 log.cc:927] Time spent T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.305s	user 0.002s	sys 0.000s
I20260709 13:08:50.022431 28321 raft_consensus.cc:3055] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:50.026826 25909 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:50.035943 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:50.016201 28095 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:49.988771 28373 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:50.053316 25909 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:50.040270 28311 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:50.054980 28232 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:50.046772 25943 raft_consensus.cc:2393] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:50.058563 27997 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:50.048002 25387 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 13:08:50.048695 28361 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:50.068061 27997 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:49.990762 28310 raft_consensus.cc:697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:50.050762 25390 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:50.053182 28095 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:50.054756 28087 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:50.056773 25314 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
W20260709 13:08:50.061040 28108 consensus_meta.cc:287] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.551s	user 0.007s	sys 0.001s
I20260709 13:08:50.061398 28232 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:50.045567 28360 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:50.064308 28314 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:50.047734 28341 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:50.097018 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:50.097792 25387 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:50.071488 28271 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:50.074143 25942 raft_consensus.cc:2393] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 7: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:50.078779 28333 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:49.989668 27925 raft_consensus.cc:740] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:50.081817 28310 consensus_queue.cc:237] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:50.082975 28016 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:50.083907 28366 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 13:08:50.050418 28321 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:50.094153 25909 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:50.060583 28347 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:50.106184 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:50.131655 25392 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:50.108608 25312 raft_consensus.cc:2368] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8.
I20260709 13:08:50.132937 25284 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:50.096503 28309 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:50.093336 28087 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:50.135607 25284 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:50.148257 25284 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:50.149816 25284 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:50.150997 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 297034us
I20260709 13:08:50.113972 28360 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:50.179850 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:50.095126 28108 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:50.151717 28210 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:50.147341 28330 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:50.164346 28361 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:50.169291 28016 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:50.182776 28369 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:50.169548 28032 consensus_queue.cc:1059] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20260709 13:08:50.184194 28087 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:50.185050 28179 consensus_meta.cc:287] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.521s	user 0.010s	sys 0.000s
I20260709 13:08:50.207451 25387 raft_consensus.cc:2410] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
W20260709 13:08:50.198966 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 352479us
I20260709 13:08:50.185632 28333 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:50.206514 25239 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:50.182811 28334 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:50.210167 25392 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:50.192973 28210 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:50.208951 25287 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:50.223312 28371 log.cc:927] Time spent T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 13:08:50.224857 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:50.225695 25943 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 8: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:50.197809 28361 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:50.232885 25321 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.002s	sys 0.016s
W20260709 13:08:50.234023 25321 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.003s	sys 0.016s
I20260709 13:08:50.235579 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:50.219434 28334 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:50.232511 27960 consensus_meta.cc:287] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.591s	user 0.011s	sys 0.000s
I20260709 13:08:50.222282 28369 raft_consensus.cc:697] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 6 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:50.181252 27925 consensus_queue.cc:260] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:50.247234 25908 leader_election.cc:336] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.920s (SENT)
I20260709 13:08:50.229148 28210 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:50.250610 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 407548us
W20260709 13:08:50.251205 28151 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.019s	sys 0.007s
I20260709 13:08:50.265159 25246 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 13:08:50.265576 25392 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 13:08:50.209607 28326 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:50.232414 28361 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:50.235910 28186 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:50.268013 28016 raft_consensus.cc:697] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:50.268383 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:50.263164 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 492721us
I20260709 13:08:50.240876 28070 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:50.233258 28181 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 13:08:50.249626 28334 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:50.221071 28179 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:50.264242 28151 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.019s	sys 0.008s
I20260709 13:08:50.196336 28321 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:50.296502 25358 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.185s (SENT) [suppressed 37 similar messages]
W20260709 13:08:50.318702 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:50.253101 27960 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:50.321501 25242 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:08:50.327525 25315 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 9: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:50.300843 28314 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:50.330574 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:50.334942 25243 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:50.314603 28271 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:50.313611 27956 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:50.337719 28321 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:50.320770 28347 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:50.348507 25390 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:50.385759 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:50.393522 28382 log.cc:927] Time spent T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
W20260709 13:08:50.395102 28390 log.cc:927] Time spent T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:08:50.395422 28179 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:50.396735 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 145863us
I20260709 13:08:50.398454 25287 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:50.321161 28333 consensus_queue.cc:1175] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" responder_term: 6 status { last_received { term: 6 index: 4 } last_committed_idx: 2 last_received_current_leader { term: 6 index: 4 } }
I20260709 13:08:50.374168 28267 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:50.425411 25212 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:50.397166 28108 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:50.433089 25938 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 13:08:50.438053 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:08:50.446079 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 124637us
I20260709 13:08:50.448401 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:50.449462 25244 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 7: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:50.431579 28267 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:50.452580 25939 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 13:08:50.453336 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 54996.4us
I20260709 13:08:50.457336 28221 consensus_queue.cc:1059] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:08:50.458531 28221 consensus_queue.cc:1243] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer 2329e428ebbf484fa3ea9452133df567 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:08:50.459174 28221 raft_consensus.cc:3055] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:08:50.459568 28221 raft_consensus.cc:740] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:50.461066 25212 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:50.387946 28016 consensus_queue.cc:237] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:50.471915 28385 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:08:50.473578 28070 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:50.474210 28070 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:50.475346 27960 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:50.487596 28377 log.cc:927] Time spent T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.175s	user 0.000s	sys 0.003s
I20260709 13:08:50.472419 28095 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:50.493893 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:50.494956 28095 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:50.443359 28222 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:50.503201 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:50.504457 25943 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:08:50.506116 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:50.527565 25936 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 7: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:50.431869 28333 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 13:08:50.534546 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:50.535458 25390 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:50.532709 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 206268us
I20260709 13:08:50.539189 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:50.371132 28369 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:50.541828 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:50.531399 28326 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:50.540887 28314 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260709 13:08:50.531607 28222 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:50.748680 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:50.748160 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 208263us
I20260709 13:08:50.758455 25287 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:50.498730 28391 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:08:50.530850 28399 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:08:50.751689 25390 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:50.564962 28372 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
W20260709 13:08:50.752631 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:50.755252 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:50.765452 25246 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:50.749176 28087 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:50.767925 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:50.748241 28271 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:50.760035 28398 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:50.760879 28222 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:50.793717 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:50.796901 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:50.796195 28016 consensus_queue.cc:1059] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 13:08:50.767530 28404 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:50.799499 25245 raft_consensus.cc:2393] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 7: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:08:50.781890 28309 consensus_meta.cc:287] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.626s	user 0.011s	sys 0.000s
I20260709 13:08:50.812377 28326 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:50.819255 28333 raft_consensus.cc:3055] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:50.819773 28333 raft_consensus.cc:740] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:50.820202 28347 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:50.821449 25243 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:50.820806 28347 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:50.822731 25243 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:50.823508 28347 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:50.823768 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:50.776575 28108 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:50.838774 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:50.779031 28361 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:50.820672 28333 consensus_queue.cc:260] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:51.161527 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 395560us
I20260709 13:08:50.977612 25246 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:51.183847 25316 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260709 13:08:51.102240 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 348888us
I20260709 13:08:50.812471 28398 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:50.930471 28321 consensus_meta.cc:287] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.561s	user 0.008s	sys 0.002s
W20260709 13:08:50.929351 28360 consensus_meta.cc:287] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.740s	user 0.012s	sys 0.000s
I20260709 13:08:50.815263 28326 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:50.878391 28309 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:51.209578 25941 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:51.209815 28070 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:51.210325 28070 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:51.212368 28070 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:51.204939 28398 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:50.883773 28108 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:50.847967 28404 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:50.798761 28401 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:50.786317 28385 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:50.798095 28271 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:51.154238 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:51.228798 28334 consensus_meta.cc:287] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.913s	user 0.009s	sys 0.000s
I20260709 13:08:51.229471 28334 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:51.231724 28271 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:51.206702 28361 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:51.215616 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:50.914047 28151 raft_consensus.cc:2764] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 13:08:51.227348 28385 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:51.235126 25240 raft_consensus.cc:2393] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:51.235234 28222 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:51.236270 28385 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:51.232501 28267 consensus_meta.cc:287] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.709s	user 0.008s	sys 0.001s
I20260709 13:08:51.236939 28267 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:51.238271 28221 consensus_queue.cc:260] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:51.238972 28267 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:51.239285 28221 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:51.241986 28334 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:51.245424 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 517) took 1010 ms. Trace:
I20260709 13:08:51.245630 25314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1941) took 1649 ms. Trace:
I20260709 13:08:51.245957 25240 rpcz_store.cc:276] 0709 13:08:50.235127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:50.235259 (+   132us) service_pool.cc:224] Handling call
0709 13:08:51.245164 (+1009905us) spinlock_profiling.cc:243] Waited 996 ms on lock 0x7b54002b2d54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:51.245348 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2291945856}
I20260709 13:08:51.247651 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:51.250468 28333 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:08:51.245028 25315 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1935) took 1650 ms (client timeout 1918 ms). Trace:
W20260709 13:08:51.255961 25315 rpcz_store.cc:269] 0709 13:08:49.594324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:49.594449 (+   125us) service_pool.cc:224] Handling call
0709 13:08:51.244789 (+1650340us) spinlock_profiling.cc:243] Waited 692 ms on lock 0x7b5400043dd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:51.244954 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1592674560}
I20260709 13:08:51.257877 25320 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 13:08:51.246124 25314 rpcz_store.cc:276] 0709 13:08:49.595887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:49.685541 (+ 89654us) service_pool.cc:224] Handling call
0709 13:08:51.245438 (+1559897us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b54002f3fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:51.245566 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":765580672}
W20260709 13:08:51.222808 28210 consensus_meta.cc:287] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.950s	user 0.013s	sys 0.013s
I20260709 13:08:51.267674 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:51.273598 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 465113us
I20260709 13:08:51.274443 25314 raft_consensus.cc:2393] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:51.268672 28385 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:50.783128 28087 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:51.279395 25284 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:51.350777 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:51.347168 25358 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.001s	sys 0.019s
W20260709 13:08:51.352274 25358 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.001s	sys 0.019s
W20260709 13:08:51.352691 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 528948us
I20260709 13:08:51.262669 28228 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:08:51.293385 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:51.226890 28321 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:51.303028 28217 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:08:51.376703 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 97428.8us
I20260709 13:08:51.353575 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:51.379029 25358 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:51.380498 28321 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:51.380618 28411 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:08:51.295078 28222 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:51.422410 28321 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:51.424440 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 594) took 1542 ms. Trace:
I20260709 13:08:51.452912 25938 rpcz_store.cc:276] 0709 13:08:49.882093 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:49.882205 (+   112us) service_pool.cc:224] Handling call
0709 13:08:50.432741 (+550536us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:50.433556 (+   815us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:51.424171 (+990615us) spinlock_profiling.cc:243] Waited 550 ms on lock 0x7b54001b2bd0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:51.424359 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1265301376}
I20260709 13:08:51.282783 28314 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:08:51.469075 25209 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.064s	sys 0.123s
W20260709 13:08:51.479591 25209 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.064s	sys 0.123s
I20260709 13:08:51.482447 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.483256 25314 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:51.268811 28210 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:51.469934 25287 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.029s	sys 0.054s
W20260709 13:08:51.487596 25287 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.727s	user 0.029s	sys 0.054s
I20260709 13:08:51.411460 28032 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
W20260709 13:08:51.573184 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 814765us
W20260709 13:08:51.573225 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 194274us
I20260709 13:08:51.575547 25358 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:51.576247 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:51.578440 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:51.579196 25320 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 6
I20260709 13:08:51.416287 28385 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:51.583549 28411 consensus_queue.cc:1059] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:08:51.586189 25315 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:08:51.589457 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:08:51.594870 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:51.592952 25314 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:08:51.607018 28321 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:08:51.614291 25938 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:51.616853 25284 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:51.615935 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.617165 25321 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:51.622869 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:51.623430 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.624117 25314 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:51.624974 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:51.607519 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:51.626147 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:51.627498 25209 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:51.613277 28329 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.017s
I20260709 13:08:51.630028 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.630321 28326 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:51.613634 28361 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260709 13:08:51.631233 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.632330 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.632869 25315 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:08:51.633162 28108 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:51.633549 28347 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:51.634567 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.635465 25315 raft_consensus.cc:2376] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6. Re-sending same reply.
I20260709 13:08:51.636883 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.639174 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.642227 25940 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 13:08:51.589401 28404 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:51.574360 28314 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:51.642499 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.648427 25944 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:51.649786 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:51.650341 25945 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:51.652222 25315 raft_consensus.cc:2410] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:08:51.653834 25942 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:08:51.649417 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:51.651263 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:51.644855 27925 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:51.654775 25944 raft_consensus.cc:2393] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:51.639250 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 522) took 1107 ms. Trace:
I20260709 13:08:51.655390 25943 raft_consensus.cc:1240] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:51.646742 28087 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:51.670292 25320 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:51.671423 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.672149 25320 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:51.657704 28406 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:51.673942 28406 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:51.676481 28406 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:51.677460 25312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1929) took 2088 ms (client timeout 2048 ms). Trace:
I20260709 13:08:51.654793 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:51.680213 25318 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:51.655778 25936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 730) took 1126 ms. Trace:
I20260709 13:08:51.665270 28210 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:51.662745 28408 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:51.663657 28321 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:51.657982 25244 rpcz_store.cc:276] 0709 13:08:50.531900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:50.532044 (+   144us) service_pool.cc:224] Handling call
0709 13:08:50.542042 (+  9998us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:50.752255 (+210213us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:50.752285 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:50.752732 (+   447us) log.cc:844] Serialized 26 byte log entry
0709 13:08:51.275204 (+522472us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:51.275251 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:51.276166 (+   915us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:51.639023 (+362857us) raft_consensus.cc:1685] finished
0709 13:08:51.639048 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:51.639189 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2824,"raft.queue_time_us":229,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":127,"thread_start_us":478446,"threads_started":1,"wal-append.queue_time_us":857304}
I20260709 13:08:51.660810 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.668337 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:51.677557 25317 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: []
W20260709 13:08:51.677938 25312 rpcz_store.cc:269] 0709 13:08:49.589125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:49.589695 (+   570us) service_pool.cc:224] Handling call
0709 13:08:51.677247 (+2087552us) spinlock_profiling.cc:243] Waited 494 ms on lock 0x7b5400045954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:51.677397 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1135924480}
I20260709 13:08:51.657805 28412 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:51.681387 25938 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:51.655898 25314 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 8: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:51.684396 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:51.681917 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 478780us
I20260709 13:08:51.687371 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:51.690269 25320 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:08:51.693763 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:51.693907 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 77182.9us
I20260709 13:08:51.646749 28314 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:51.657600 28309 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:51.695582 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.696580 25242 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260709 13:08:51.698144 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:51.698508 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.699080 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.682602 28408 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:51.712248 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:51.685307 28087 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:51.682235 28321 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:51.694805 28372 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:51.715217 25284 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:51.682296 28108 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:51.710740 25314 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:51.685115 28334 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:51.710433 25242 pending_rounds.cc:85] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 4
I20260709 13:08:51.730187 25242 pending_rounds.cc:107] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 13:08:51.730522 25242 raft_consensus.cc:2889] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:51.719187 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:51.714337 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:51.710830 25315 raft_consensus.cc:2410] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
W20260709 13:08:51.723959 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:51.725050 25938 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 13:08:51.732331 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:51.732613 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:51.726029 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:51.726292 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:51.707166 28217 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.010s
I20260709 13:08:51.696904 28181 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:51.694490 28070 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:51.781863 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:51.749799 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 156459us
I20260709 13:08:51.784281 28404 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:08:51.785988 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1614) took 2062 ms. Trace:
I20260709 13:08:51.786374 25239 rpcz_store.cc:276] 0709 13:08:49.723845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:49.724284 (+   439us) service_pool.cc:224] Handling call
0709 13:08:49.724551 (+   267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:50.216484 (+491933us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:51.785723 (+1569239us) spinlock_profiling.cc:243] Waited 482 ms on lock 0x7b5400193054. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:51.785911 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1108017920}
I20260709 13:08:51.782351 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.708700 28221 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:51.707177 28411 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:08:51.737695 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.713557 28419 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:51.714607 28408 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:51.693554 28412 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:51.721645 28321 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:51.834379 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:51.834558 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.835004 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.734300 28032 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:51.838696 25315 raft_consensus.cc:2376] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6. Re-sending same reply.
I20260709 13:08:51.716303 28309 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:51.835855 25908 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:51.693411 27945 consensus_queue.cc:260] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:51.839879 25320 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:51.854594 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.736527 28108 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:51.840186 25945 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:08:51.791322 28221 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:51.714515 28406 consensus_queue.cc:1059] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.112s
I20260709 13:08:51.830052 25314 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:08:51.833673 28181 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:51.851152 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.782989 28334 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:51.734220 28372 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:51.793552 28421 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:51.794652 28422 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.074s
I20260709 13:08:51.855294 25320 raft_consensus.cc:2393] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 8: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:51.858703 25284 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:51.862799 28408 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:51.864284 28032 raft_consensus.cc:697] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:51.840265 25312 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:51.841159 28314 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:51.791824 28070 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:51.869671 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:51.834291 28087 consensus_queue.cc:1059] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20260709 13:08:51.872300 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.874890 28344 raft_consensus.cc:3055] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:51.868911 28221 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:51.835700 28419 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:51.853601 27960 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260709 13:08:51.888263 25314 raft_consensus.cc:2410] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:08:51.883426 28179 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:08:51.891716 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:51.892340 25908 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:51.904106 25320 raft_consensus.cc:2393] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 6: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:51.888248 28414 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:51.908736 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.910352 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.911913 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.913374 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:51.914384 25314 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 9: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:51.917481 25314 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: []
I20260709 13:08:51.919147 25314 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 13:08:51.889304 28344 raft_consensus.cc:740] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:51.898583 28385 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:51.981474 28397 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.004s	sys 0.005s
W20260709 13:08:51.981891 28397 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.004s	sys 0.005s
I20260709 13:08:51.981825 28412 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:51.983044 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 407524us
I20260709 13:08:51.900353 28070 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:51.884933 28314 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:51.992935 28108 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:51.983902 25358 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:51.993561 28108 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:51.993980 27962 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:51.908565 28179 raft_consensus.cc:697] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:51.978586 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:51.991425 28228 consensus_meta.cc:287] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.580s	user 0.010s	sys 0.000s
I20260709 13:08:52.015046 25358 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:52.029222 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 329538us
W20260709 13:08:52.025161 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 327555us
W20260709 13:08:52.035651 25209 leader_election.cc:336] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.048s (SENT)
I20260709 13:08:52.039373 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:51.994663 28070 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:51.997304 27962 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:52.076385 28443 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:52.083441 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:52.083359 28443 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:52.079098 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:52.084676 25938 raft_consensus.cc:2368] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9.
I20260709 13:08:52.086144 25938 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:08:52.079623 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:52.080379 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:52.080782 28345 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.852s	user 0.006s	sys 0.008s
W20260709 13:08:52.088441 28345 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.859s	user 0.006s	sys 0.008s
I20260709 13:08:52.079555 28439 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:52.082367 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:52.090912 25314 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:52.097590 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:51.994134 28108 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:52.113657 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:52.121942 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 71497us
I20260709 13:08:52.114567 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:52.124306 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:52.084312 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:52.125057 25238 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:52.025435 28438 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:52.105719 25940 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 8: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:52.043576 28181 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:52.083874 25358 connection.cc:256] RPC call timeout handler was delayed by -1.6782837150000205s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1559s for a response
I20260709 13:08:52.082594 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:52.077026 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:52.024250 28442 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:52.046250 28359 raft_consensus.cc:3055] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:08:52.016474 28385 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:08:52.133939 25358 connection.cc:256] RPC call timeout handler was delayed by -1.6719837150000103s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15660000000000002s for a response
I20260709 13:08:52.139016 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:52.047478 28326 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:52.141456 28326 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:52.137187 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 442549us
I20260709 13:08:52.105262 27962 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:52.151254 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:52.145447 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:52.146456 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:51.991132 28032 consensus_queue.cc:237] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.148387 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:52.126314 28424 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.290s	user 0.001s	sys 0.000s
W20260709 13:08:52.155009 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:52.164340 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:52.165794 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:52.166232 25241 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:52.173800 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:52.175449 25911 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:52.178676 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:52.126770 28429 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:52.150415 28181 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:52.138216 28359 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:52.103657 28070 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.198863 25390 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:52.199572 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:52.200842 25390 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:08:52.203447 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:52.204902 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:52.207816 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:52.210270 25390 raft_consensus.cc:2410] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 13:08:52.181679 28309 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:52.223479 25911 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:52.225766 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:52.235805 25390 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:08:52.242674 28458 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 13:08:52.205782 28181 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:52.242584 25316 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:52.242584 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:52.245232 25394 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 13:08:52.243551 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:52.247577 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 133081us
I20260709 13:08:52.250571 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:52.251295 25394 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:52.190623 28334 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:52.201555 28440 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:52.190227 28421 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:52.275995 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:52.277092 25911 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:52.282116 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:52.206033 28372 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:52.138674 28438 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:52.430105 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 153101us
I20260709 13:08:52.430269 25389 raft_consensus.cc:2410] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 13:08:52.430848 25911 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:52.431564 28438 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:52.432104 25389 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:08:52.434367 25394 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:08:52.424190 25319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1949) took 1247 ms. Trace:
I20260709 13:08:52.436492 25394 raft_consensus.cc:1240] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 13:08:52.373760 28447 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:08:52.104195 28344 consensus_queue.cc:260] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.374223 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:52.189117 28429 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.435315 25319 rpcz_store.cc:276] 0709 13:08:51.176882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:51.177033 (+   151us) service_pool.cc:224] Handling call
0709 13:08:51.177459 (+   426us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:08:51.221768 (+ 44309us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:51.221815 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:08:51.225814 (+  3999us) log.cc:844] Serialized 52 byte log entry
0709 13:08:51.608165 (+382351us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:51.608219 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:51.608305 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:52.365972 (+757667us) raft_consensus.cc:1685] finished
0709 13:08:52.423922 (+ 57950us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:52.424117 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21349,"raft.queue_time_us":48704,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":131,"thread_start_us":353730,"threads_started":1,"wal-append.queue_time_us":462575}
I20260709 13:08:52.464687 25243 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:08:52.246954 28267 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:52.281710 28408 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:52.176774 28179 consensus_queue.cc:237] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.149156 28442 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.190083 28449 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.181s
I20260709 13:08:52.279836 28436 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:52.247110 28437 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:08:52.218101 28435 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:52.450580 28421 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.470351 25287 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:52.518414 28461 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
W20260709 13:08:52.519065 28347 consensus_meta.cc:287] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.885s	user 0.007s	sys 0.003s
I20260709 13:08:52.519503 28347 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:52.480700 25319 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:08:52.254006 28345 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:08:52.523644 25319 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 13:08:52.457064 28432 log.cc:927] Time spent T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.341s	user 0.001s	sys 0.000s
I20260709 13:08:52.500954 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:52.461382 25391 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:52.455525 28314 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:52.487146 25359 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:52.511852 25358 connection.cc:256] RPC call timeout handler was delayed by -1.5927837149999959s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1654s for a response
I20260709 13:08:52.473956 28419 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:52.518535 27925 consensus_meta.cc:287] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.848s	user 0.009s	sys 0.001s
W20260709 13:08:52.484812 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 233418us
I20260709 13:08:52.440847 28372 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:52.488528 28361 consensus_queue.cc:1059] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
W20260709 13:08:52.536042 28321 consensus_meta.cc:287] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.685s	user 0.005s	sys 0.005s
W20260709 13:08:52.533491 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 46431.8us
I20260709 13:08:52.513813 25390 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
W20260709 13:08:52.530814 25358 connection.cc:256] RPC call timeout handler was delayed by -1.58288371499998s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1665s for a response
W20260709 13:08:52.558029 25358 connection.cc:256] RPC call timeout handler was delayed by -1.5801837149999756s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1668s for a response
W20260709 13:08:52.558485 25358 connection.cc:256] RPC call timeout handler was delayed by -1.5666837150000674s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1683s for a response
W20260709 13:08:52.558869 25358 connection.cc:256] RPC call timeout handler was delayed by -1.447883714999989s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1815s for a response
W20260709 13:08:52.559176 25358 connection.cc:256] RPC call timeout handler was delayed by -1.437983714999973s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1826s for a response
W20260709 13:08:52.559540 25358 connection.cc:256] RPC call timeout handler was delayed by -1.401083715000027s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1867s for a response
W20260709 13:08:52.559913 25358 connection.cc:256] RPC call timeout handler was delayed by -1.3542837150000651s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19190000000000002s for a response
W20260709 13:08:52.560278 25358 connection.cc:256] RPC call timeout handler was delayed by -1.3227837150000141s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19540000000000002s for a response
W20260709 13:08:52.560660 25358 connection.cc:256] RPC call timeout handler was delayed by -1.2858837150000682s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1995s for a response
W20260709 13:08:52.522884 28397 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.004s	sys 0.024s
I20260709 13:08:52.544339 25313 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:52.531643 28456 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:08:52.521768 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 51538.3us
W20260709 13:08:52.566968 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 318462us
I20260709 13:08:52.570194 25287 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:52.551237 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 120458us
I20260709 13:08:52.510848 28468 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:52.555486 25209 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:52.456584 28070 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:52.473323 28267 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:52.502427 28359 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:52.568315 25212 leader_election.cc:336] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.554s (SENT)
I20260709 13:08:52.550923 28309 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:08:52.564924 28397 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.004s	sys 0.025s
I20260709 13:08:52.438751 28440 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:52.586956 25287 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:52.579497 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:52.600589 25390 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: [5.5-8.7]
I20260709 13:08:52.608230 25212 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:52.568864 28108 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:52.576975 28447 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:52.551324 28372 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:52.577732 28443 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:52.601255 28314 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:08:52.697201 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:52.604883 28419 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:08:52.636406 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 49530.7us
I20260709 13:08:52.699169 25287 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:52.585783 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:52.700980 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:52.523751 28435 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.566569 28321 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:52.674209 28444 consensus_queue.cc:1059] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.110s
I20260709 13:08:52.674974 28462 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.110s
I20260709 13:08:52.677865 28457 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:52.679018 28368 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:08:52.679806 28455 log.cc:927] Time spent T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.218s	user 0.001s	sys 0.000s
W20260709 13:08:52.680518 28460 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.239s	user 0.001s	sys 0.000s
I20260709 13:08:52.683643 28421 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:52.736497 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 168183us
I20260709 13:08:52.687875 28372 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:52.696108 28436 raft_consensus.cc:697] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:52.581926 28456 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.632808 28468 raft_consensus.cc:697] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:52.733861 27925 consensus_queue.cc:1059] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.014s
I20260709 13:08:52.748239 28464 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:52.748586 28468 consensus_queue.cc:237] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.599885 28406 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
I20260709 13:08:52.703440 25287 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:52.701716 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 116019us
I20260709 13:08:52.602784 28391 consensus_queue.cc:1059] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.097s
I20260709 13:08:52.689813 28448 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:52.753551 28440 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:52.754889 28456 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:52.698994 28309 raft_consensus.cc:697] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:52.729087 28419 raft_consensus.cc:697] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:08:52.756196 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 52827.3us
I20260709 13:08:52.725844 28457 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:52.756454 28419 consensus_queue.cc:237] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:52.757414 28457 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:52.743571 25243 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:52.756464 28309 consensus_queue.cc:237] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.759115 28032 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:52.759663 28032 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.725070 28359 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:52.721521 28447 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:52.785837 28385 consensus_meta.cc:287] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.634s	user 0.009s	sys 0.000s
I20260709 13:08:52.786271 28410 raft_consensus.cc:3055] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:08:52.786553 28447 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:52.767726 28032 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:52.787060 28470 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:52.786777 28410 raft_consensus.cc:740] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:52.787571 28470 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:52.788611 28391 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:52.788272 28410 consensus_queue.cc:260] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.789143 28391 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:52.791241 28391 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:52.779228 28433 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.675s	user 0.001s	sys 0.000s
I20260709 13:08:52.794632 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1938) took 3199 ms. Trace:
I20260709 13:08:52.795017 25317 rpcz_store.cc:276] 0709 13:08:49.595130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:49.716778 (+121648us) service_pool.cc:224] Handling call
0709 13:08:51.677381 (+1960603us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:51.683425 (+  6044us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:52.794416 (+1110991us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b5400045950. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:52.794551 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4421996416}
I20260709 13:08:52.795848 28470 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:52.796846 25316 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:08:52.799358 28228 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:52.799856 28228 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:52.801945 28228 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:52.758510 28397 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:08:52.762172 28466 log.cc:927] Time spent T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 13:08:52.803129 28397 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:52.725409 28411 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:52.746507 25392 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 3.
I20260709 13:08:52.803706 28397 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:52.748832 28464 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:52.726452 28443 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:52.804579 28443 raft_consensus.cc:697] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:52.798919 28457 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.805794 28464 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:52.806444 25394 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:52.767869 28361 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:52.807135 28457 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:52.807050 28361 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.768795 28429 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:52.770439 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:52.733824 28314 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 13:08:52.771950 25209 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:52.772480 27945 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:52.810254 28344 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:52.811136 28470 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:52.811546 28470 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:52.811921 28470 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:52.777685 28442 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:52.779225 28321 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:52.846957 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:52.780405 28413 log.cc:927] Time spent T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 1.225s	user 0.002s	sys 0.000s
I20260709 13:08:51.680778 25936 rpcz_store.cc:276] 0709 13:08:50.529097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:50.529273 (+   176us) service_pool.cc:224] Handling call
0709 13:08:50.538940 (+  9667us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:50.539158 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:08:51.183021 (+643863us) log.cc:844] Serialized 14 byte log entry
0709 13:08:51.655154 (+472133us) log.cc:844] Serialized 14 byte log entry
0709 13:08:51.655447 (+   293us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:51.655496 (+    49us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:08:51.655544 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:51.655610 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:51.655721 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3321,"prepare.queue_time_us":340463,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":62,"raft.queue_time_us":16097,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":22311,"thread_start_us":794323,"threads_started":2,"wal-append.queue_time_us":479867}
I20260709 13:08:52.782841 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:52.783768 27962 consensus_meta.cc:287] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.602s	user 0.009s	sys 0.001s
I20260709 13:08:52.723527 28108 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:08:52.694352 28267 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:52.805430 28443 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.810160 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:52.773197 28435 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:52.812564 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 726) took 2308 ms. Trace:
I20260709 13:08:52.814500 28186 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:52.773928 28221 consensus_meta.cc:287] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.873s	user 0.009s	sys 0.002s
I20260709 13:08:52.815301 28406 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:52.767681 28436 consensus_queue.cc:237] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.818756 28368 consensus_queue.cc:1059] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:08:52.819738 28181 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:52.822453 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 77407.8us
I20260709 13:08:52.824157 27960 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:52.828744 25237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1626) took 1140 ms. Trace:
I20260709 13:08:52.835055 28389 raft_consensus.cc:3055] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:52.851430 25244 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:52.856154 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 445) took 1028 ms. Trace:
I20260709 13:08:52.868942 25320 rpcz_store.cc:276] 0709 13:08:51.827316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:51.908809 (+ 81493us) service_pool.cc:224] Handling call
0709 13:08:52.855971 (+947162us) spinlock_profiling.cc:243] Waited 856 ms on lock 0x7b540035cad4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:52.856096 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1969334528}
I20260709 13:08:52.856221 25246 raft_consensus.cc:2410] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
W20260709 13:08:52.866375 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:52.849104 28397 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.866156 28435 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:52.873594 28435 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.874634 28397 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:52.836840 25358 connection.cc:256] RPC call timeout handler was delayed by -1.1562837149999723s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21389999999999998s for a response
I20260709 13:08:52.836977 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:52.876475 25358 connection.cc:256] RPC call timeout handler was delayed by -1.1274837150000394s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2171s for a response
I20260709 13:08:52.877094 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:52.860332 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 50247.6us
I20260709 13:08:52.786477 28385 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:52.838475 28428 raft_consensus.cc:3055] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Stepping down as leader of term 7
W20260709 13:08:52.841486 25318 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1952) took 1626 ms (client timeout 1859 ms). Trace:
I20260709 13:08:52.889379 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:52.889984 25318 rpcz_store.cc:269] 0709 13:08:51.215137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:51.215295 (+   158us) service_pool.cc:224] Handling call
0709 13:08:52.841271 (+1625976us) spinlock_profiling.cc:243] Waited 438 ms on lock 0x7b54000f9954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:52.841421 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1006667136}
I20260709 13:08:52.842708 28087 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:08:52.844683 25319 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:52.846367 28361 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:52.878513 25909 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:52.864288 28430 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.939s	user 0.005s	sys 0.000s
I20260709 13:08:52.860663 25941 rpcz_store.cc:276] 0709 13:08:50.504176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:50.504312 (+   136us) service_pool.cc:224] Handling call
0709 13:08:50.536960 (+ 32648us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:51.209955 (+672995us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:52.812341 (+1602386us) spinlock_profiling.cc:243] Waited 672 ms on lock 0x7b54006608d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:52.812506 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1546402560}
I20260709 13:08:52.906306 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:52.906939 25936 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:52.865911 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:52.879447 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:52.864187 25237 rpcz_store.cc:276] 0709 13:08:51.688753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:51.688876 (+   123us) service_pool.cc:224] Handling call
0709 13:08:51.711673 (+ 22797us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:52.131522 (+419849us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:52.131562 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:52.161486 (+ 29924us) log.cc:844] Serialized 26 byte log entry
0709 13:08:52.460290 (+298804us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:52.460326 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:52.828253 (+367927us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b54003a98d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:52.828318 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:52.828381 (+    63us) raft_consensus.cc:1685] finished
0709 13:08:52.828401 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:52.828676 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":200946,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":114,"spinlock_wait_cycles":375299840,"thread_start_us":510584,"threads_started":2,"wal-append.queue_time_us":376711}
I20260709 13:08:52.907936 25246 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:08:52.861032 28108 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:52.862284 28368 raft_consensus.cc:3055] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 13:08:52.910269 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:52.911036 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 728) took 2404 ms. Trace:
I20260709 13:08:52.854480 28465 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:52.911036 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:52.913738 25238 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:08:52.912441 25937 rpcz_store.cc:276] 0709 13:08:50.506521 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:50.507517 (+   996us) service_pool.cc:224] Handling call
0709 13:08:50.535996 (+ 28479us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:50.536324 (+   328us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:50.536361 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:50.576565 (+ 40204us) log.cc:844] Serialized 26 byte log entry
0709 13:08:51.011445 (+434880us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:08:51.011488 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:51.011585 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:52.779748 (+1768163us) spinlock_profiling.cc:243] Waited 7.29 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:52.906812 (+127064us) raft_consensus.cc:1685] finished
0709 13:08:52.906831 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:52.910975 (+  4144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":222342,"raft.queue_time_us":2742,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":22656,"spinlock_wait_cycles":16757504,"thread_start_us":228614,"threads_started":1,"wal-append.queue_time_us":644039}
I20260709 13:08:52.919210 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:52.922736 25285 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:52.864341 28389 raft_consensus.cc:740] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:52.923795 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:52.889838 28428 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:52.860563 28470 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:52.861359 28221 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.863988 27960 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:52.927436 28267 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:52.936242 28267 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:52.936650 28267 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:52.927587 28397 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:52.856014 27962 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.938339 25386 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 7: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:52.924144 25285 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:52.941509 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:52.926014 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:52.913024 27945 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:52.861580 28406 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.933511 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:52.912091 28368 raft_consensus.cc:740] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:52.914324 28385 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:52.928619 28465 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:52.943522 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:52.860896 28186 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:52.940022 27962 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:52.951809 28309 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:52.954706 28309 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:52.955569 28309 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:52.955875 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:52.957118 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:52.958850 25285 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:52.963490 25240 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:52.967239 25358 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:52.967628 28406 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:52.968472 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:52.968575 28428 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.971127 28435 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:52.971891 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:52.977558 25238 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:08:52.961582 28389 consensus_queue.cc:260] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.978499 27945 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:52.979288 28389 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:52.980628 28470 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:52.947413 28179 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:08:52.980988 28392 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:08:52.981325 28179 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:52.981745 28179 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:08:52.986308 28487 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 13:08:52.987465 25943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 732) took 1340 ms. Trace:
I20260709 13:08:52.987576 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:52.987884 25943 rpcz_store.cc:276] 0709 13:08:51.647462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:51.647590 (+   128us) service_pool.cc:224] Handling call
0709 13:08:51.647851 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:51.659961 (+ 12110us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:52.987195 (+1327234us) spinlock_profiling.cc:243] Waited 7.36 ms on lock 0x7b54004ad8d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:52.987405 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16922880}
I20260709 13:08:52.983595 28309 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:52.989319 25939 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 13:08:52.988227 25241 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:52.991307 25359 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:52.993561 28418 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:52.994851 28428 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:52.995464 28438 raft_consensus.cc:3055] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:52.995868 28438 raft_consensus.cc:740] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:53.001101 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:53.003906 25393 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:52.997632 28368 consensus_queue.cc:260] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.005257 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.008625 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:53.009085 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.009421 25238 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:53.009795 25285 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:53.011104 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:52.991823 28470 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:53.010717 28392 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:53.018460 25358 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:53.005964 28435 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:53.007633 28418 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:08:53.027578 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:53.033005 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1947) took 1879 ms. Trace:
I20260709 13:08:53.033502 25321 rpcz_store.cc:276] 0709 13:08:51.154077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:51.154189 (+   112us) service_pool.cc:224] Handling call
0709 13:08:51.176414 (+ 22225us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:51.655193 (+478779us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:53.032810 (+1377617us) spinlock_profiling.cc:243] Waited 440 ms on lock 0x7b5400044554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:53.032956 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1013075840}
I20260709 13:08:53.037086 25358 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:53.037914 28439 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:08:53.038630 28309 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:53.039983 25287 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:53.040540 25358 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:53.040799 28016 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:08:53.041281 28016 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:53.041339 28439 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:53.041774 28016 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:53.044543 28418 raft_consensus.cc:697] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:53.044677 25389 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 7. Ops: [5.7-5.7]
I20260709 13:08:53.013955 27945 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:53.045469 28418 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.049788 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.050179 25389 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 13:08:53.030278 28368 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:53.033293 28429 consensus_queue.cc:1059] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.018s
I20260709 13:08:53.024626 27962 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:08:53.077086 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.077483 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:53.025482 28392 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:53.062393 28221 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:53.063607 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.046142 28447 consensus_queue.cc:1059] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20260709 13:08:53.069553 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:53.073052 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.077625 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:53.057790 28016 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.078167 25319 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:53.080806 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:53.154122 25937 raft_consensus.cc:2410] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:08:53.081699 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:53.082526 25238 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
W20260709 13:08:53.082872 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:53.085520 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:53.086053 25241 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 13:08:53.086565 25909 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:53.087587 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:53.088795 25944 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:53.148877 25943 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:53.156313 25940 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:53.148850 25389 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
W20260709 13:08:53.157209 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 70749us
I20260709 13:08:53.027014 28470 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:53.158710 28468 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:53.159138 28468 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:53.159456 28468 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:53.160355 28470 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:53.157056 28442 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:53.161672 28442 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.163702 28442 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:53.084960 28438 consensus_queue.cc:260] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.166231 28438 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:53.144109 28392 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:53.144151 25908 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:53.151765 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 526) took 1085 ms. Trace:
I20260709 13:08:53.171802 25240 rpcz_store.cc:276] 0709 13:08:52.066173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:52.066285 (+   112us) service_pool.cc:224] Handling call
0709 13:08:52.066589 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:52.966237 (+899648us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:53.151516 (+185279us) spinlock_profiling.cc:243] Waited 897 ms on lock 0x7b5400364a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:53.151691 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2062292224}
I20260709 13:08:53.172866 25943 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:08:53.152441 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 525) took 1086 ms. Trace:
I20260709 13:08:53.173557 25239 rpcz_store.cc:276] 0709 13:08:52.065807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:52.065978 (+   171us) service_pool.cc:224] Handling call
0709 13:08:52.114464 (+ 48486us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:52.146111 (+ 31647us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:52.146161 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:52.146215 (+    54us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:08:52.146264 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:53.152215 (+1005951us) spinlock_profiling.cc:243] Waited 31.4 ms on lock 0x7b54000a0354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:53.152283 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:53.152392 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":72181504}
I20260709 13:08:53.154424 28309 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:53.177587 28309 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.180176 25240 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:53.185623 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.092548 28267 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.172885 28108 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:53.193889 25388 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: []
I20260709 13:08:53.193639 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.194772 25393 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 7: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:53.175235 28334 consensus_queue.cc:1059] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.020s
I20260709 13:08:53.189587 28473 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:53.195811 28334 raft_consensus.cc:3055] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:53.195885 28473 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.189625 28447 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:53.196642 28334 raft_consensus.cc:740] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:53.197220 28267 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:53.198002 28334 consensus_queue.cc:260] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.198518 28473 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:53.198688 28108 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.200837 28108 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:53.201093 28334 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:53.201810 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:53.202267 28436 raft_consensus.cc:3055] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:08:53.202701 28436 raft_consensus.cc:740] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:53.203617 28436 consensus_queue.cc:260] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.204201 25240 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:08:53.206856 28414 raft_consensus.cc:3055] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:08:53.207247 28414 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:53.208024 28414 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.209446 28338 consensus_queue.cc:237] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 6, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.211485 28414 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:53.204978 28436 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:53.222532 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:53.223160 25943 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:51.449688 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:08:53.225391 25940 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:08:53.226945 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:53.231513 25908 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:53.238865 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:53.231418 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:53.239737 25315 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:53.239920 25240 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
W20260709 13:08:53.256127 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:53.257220 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:53.259848 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:53.261808 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:53.188453 28032 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 13:08:53.263592 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:53.263762 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.264034 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:53.221545 28267 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:53.270001 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:53.271715 25316 raft_consensus.cc:2410] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 9 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:08:53.287997 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:53.288892 25320 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:53.187985 28442 consensus_queue.cc:1059] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20260709 13:08:53.269608 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.292291 25395 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 7: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:53.295020 25938 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 13:08:53.251191 28447 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:08:53.303156 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 152321us
I20260709 13:08:53.244011 28392 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:53.314589 25358 leader_election.cc:400] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9.
W20260709 13:08:53.314837 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:53.315053 25358 leader_election.cc:403] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
W20260709 13:08:53.321157 25359 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60946 dropped due to backpressure: service queue is full with 50 items
I20260709 13:08:53.321645 25358 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:53.321736 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.323417 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.323680 25358 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:53.324371 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:53.324929 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1432) took 1126 ms. Trace:
I20260709 13:08:53.325372 25386 rpcz_store.cc:276] 0709 13:08:52.198895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:52.199020 (+   125us) service_pool.cc:224] Handling call
0709 13:08:53.324742 (+1125722us) spinlock_profiling.cc:243] Waited 690 ms on lock 0x7b54003e4254. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:53.324869 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1587187584}
I20260709 13:08:53.329941 28439 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:08:53.330862 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.331429 25394 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:53.333747 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.334985 28309 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:53.336242 25911 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:53.336405 25358 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:53.337455 28087 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:53.338114 28087 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:53.330709 27945 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:08:53.337997 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.341154 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.341516 25394 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:53.341619 28016 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:53.342293 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.343567 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.342231 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.345396 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.290434 28329 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260709 13:08:53.346416 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:53.349444 28449 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:53.346942 28439 consensus_queue.cc:1059] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:08:53.348959 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.350591 28439 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:08:53.351048 28439 raft_consensus.cc:697] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:53.351323 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.352018 25392 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 7: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:53.355115 25394 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:53.352936 25389 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 8: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:08:53.356316 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:53.353439 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:53.358201 25391 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 10: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:08:53.378214 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:53.378027 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:53.383791 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.384697 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1624) took 2599 ms. Trace:
I20260709 13:08:53.386232 25212 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:53.388151 25212 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:53.389503 25319 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:08:53.401747 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.385272 25242 rpcz_store.cc:276] 0709 13:08:50.756407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:50.756563 (+   156us) service_pool.cc:224] Handling call
0709 13:08:50.756929 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:51.730920 (+973991us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:53.353456 (+1622536us) spinlock_profiling.cc:243] Waited 939 ms on lock 0x7b54000a00d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:53.353651 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2159920000}
I20260709 13:08:53.383486 28440 raft_consensus.cc:3055] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:53.404826 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.266752 28179 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:53.405718 25389 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 7: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:53.405507 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.408672 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.411211 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.409792 25285 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:53.313334 28186 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:53.321503 25359 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60946 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2061)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1451)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2017)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2026)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2059)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2060)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2065)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41830 (request call id 505)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2016)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2044)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2062)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2025)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2043)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2063)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2037)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2041)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2015)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2053)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2064)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2023)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2054)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2047)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1454)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2018)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2020)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2029)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2030)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2031)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2035)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2036)
I20260709 13:08:53.391398 28438 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:53.341630 27960 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:53.352897 28313 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:53.429925 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:53.457252 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 47680us
I20260709 13:08:53.420253 25386 raft_consensus.cc:2376] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6. Re-sending same reply.
I20260709 13:08:53.458529 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.459331 25386 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 9: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:53.355052 28435 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9.
I20260709 13:08:53.303236 28267 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:53.456462 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.391861 28443 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:53.405278 28221 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:08:53.447986 28412 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.034s	sys 0.058s
W20260709 13:08:53.477855 28412 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.035s	sys 0.058s
I20260709 13:08:53.478602 25939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 617) took 1383 ms. Trace:
I20260709 13:08:53.479059 25939 rpcz_store.cc:276] 0709 13:08:52.095006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:52.095132 (+   126us) service_pool.cc:224] Handling call
0709 13:08:52.988982 (+893850us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:52.992160 (+  3178us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:53.478377 (+486217us) spinlock_profiling.cc:243] Waited 871 ms on lock 0x7b54004ad8d0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:53.478529 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2004429568}
I20260709 13:08:53.412066 28440 raft_consensus.cc:740] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:53.412617 28087 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:53.447374 28463 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:53.475921 25941 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 13:08:53.351846 28439 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.456547 28438 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:53.463907 28389 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260709 13:08:53.480787 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.503695 25392 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:53.474512 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.483055 25315 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:53.463388 27956 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:53.483958 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.508057 25389 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 9: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:53.508373 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:53.509020 25244 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:08:53.430085 28334 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:53.495245 25359 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:53.499838 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:53.500913 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:53.517565 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:53.517829 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:53.497264 28428 consensus_meta.cc:287] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.502s	user 0.009s	sys 0.002s
I20260709 13:08:53.506842 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:53.505765 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:53.558025 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:53.558101 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:53.501603 28438 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:53.473821 28368 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20260709 13:08:53.563915 28334 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:53.559870 25390 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:08:53.556442 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:53.570883 28334 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:53.493203 28032 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:53.572017 28032 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:53.574303 28179 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:53.575016 28032 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:53.575299 28440 consensus_queue.cc:260] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.576134 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.576771 25391 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:53.562935 25313 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:08:53.577699 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.579058 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:53.564083 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.586267 25389 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:08:53.559702 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 64782.3us
I20260709 13:08:53.495579 28186 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20260709 13:08:53.502525 28087 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.507591 27956 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:53.473474 28313 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.581501 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:53.594621 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:53.603824 25939 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:53.596884 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:53.584201 25911 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:53.502089 28463 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.603742 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.597116 25313 raft_consensus.cc:1240] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 4. Current term is 5. Ops: []
I20260709 13:08:53.595922 25392 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:53.609376 28267 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:53.610356 28313 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:53.612202 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.612708 28087 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:53.603170 27956 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:53.613406 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.614645 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.583230 28212 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:53.610461 28410 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:53.616014 25389 raft_consensus.cc:2393] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:53.618227 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:53.624569 25319 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:08:53.627274 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.580536 25320 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:53.618044 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:53.631309 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 233576us
I20260709 13:08:53.619582 28212 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:53.633067 28212 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:53.582768 28468 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.637287 28222 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:53.637871 28222 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:53.638375 28500 log.cc:927] Time spent T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.307s	user 0.003s	sys 0.000s
I20260709 13:08:53.638883 28468 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:53.640012 28222 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:53.643237 28429 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260709 13:08:53.644388 28326 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:53.641520 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.645479 25395 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:08:53.646349 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.645856 27956 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:53.641935 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.647666 28392 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:53.647890 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.648360 27956 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:53.650019 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.650496 25318 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:08:53.632674 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:53.650854 25389 raft_consensus.cc:2393] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 8: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:53.648960 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:53.661458 25391 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:08:53.662739 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:53.664485 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 276385us
I20260709 13:08:53.620540 28477 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:53.667024 25945 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:08:53.668538 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.668538 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:53.671181 25941 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:08:53.670460 25389 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:53.671729 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.669219 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.670460 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:53.673513 25392 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:53.672638 25212 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:53.672479 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:53.670531 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:53.675880 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.674606 25388 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 5.
I20260709 13:08:53.670019 28477 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:53.687372 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 392514us
I20260709 13:08:53.669226 28221 consensus_queue.cc:1059] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 13:08:53.666873 28326 consensus_queue.cc:1059] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 13:08:53.734193 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:53.698160 25908 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:53.744570 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 46544.2us
I20260709 13:08:53.744073 25315 raft_consensus.cc:2376] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6. Re-sending same reply.
I20260709 13:08:53.698210 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.747924 25392 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 4.
I20260709 13:08:53.746379 25908 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:53.709235 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:53.708344 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:53.759516 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 284485us
I20260709 13:08:53.698938 28477 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:08:53.785604 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:53.787676 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:53.787705 25209 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:53.735338 28465 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:53.793931 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:53.801365 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:53.802152 25246 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 11
I20260709 13:08:53.804605 25319 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:53.806022 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:53.806602 25240 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:53.753206 28472 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:53.809911 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:53.816465 25387 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:53.749748 28505 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:08:53.834915 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:53.816519 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:53.807484 28392 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:53.841595 28392 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:53.843413 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:53.843636 28392 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:53.838157 25237 raft_consensus.cc:2410] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 13:08:53.845578 25246 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:08:53.832913 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:53.701289 28222 consensus_queue.cc:1059] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.127s
I20260709 13:08:53.846748 25285 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:53.848875 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:53.840457 25387 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:53.801554 28385 raft_consensus.cc:3055] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:08:53.854192 25246 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:53.794416 28506 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:53.860430 25240 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:53.865187 25387 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:53.818022 28472 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:53.866449 25319 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 13:08:53.866943 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 194389us
I20260709 13:08:53.872613 25212 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:53.878657 25319 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:08:53.882202 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:53.884604 25944 raft_consensus.cc:2410] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 13:08:53.859499 27956 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:08:53.857983 28385 raft_consensus.cc:740] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:53.895522 25387 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:53.898250 25941 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:08:53.898591 25387 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:08:53.874069 28472 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:53.916146 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:53.904336 27956 raft_consensus.cc:697] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:53.942096 25284 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:53.943235 25320 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:53.944216 28087 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:53.947927 28087 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:53.950201 28087 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:53.951505 25320 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:53.952250 28392 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:53.944377 28507 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:53.953531 28507 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:53.953881 28507 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:53.954278 25284 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:53.956444 28397 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:53.956830 28397 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:53.957191 28397 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:53.957953 28514 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 13:08:53.961869 28517 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:08:53.964898 28397 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:53.967097 28397 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:53.972683 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:53.975396 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:53.977540 25936 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 13:08:53.995893 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 315702us
I20260709 13:08:54.014091 25209 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:53.945570 28410 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
I20260709 13:08:54.024353 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:54.025118 28515 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 13:08:54.038376 28507 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.023458 28517 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:54.024173 28397 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:54.016992 28438 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:54.017781 27956 consensus_queue.cc:237] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:54.175273 25242 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:08:54.135847 28217 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:54.176007 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 146623us
I20260709 13:08:54.178663 25209 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:54.179248 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:54.180219 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:54.131690 27962 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:08:54.174700 28443 consensus_meta.cc:287] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.692s	user 0.009s	sys 0.002s
I20260709 13:08:54.187275 25943 raft_consensus.cc:1240] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: [5.8-5.8]
I20260709 13:08:54.187850 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:54.189025 25243 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: []
I20260709 13:08:54.174530 28221 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:54.169430 28397 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:54.210048 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:54.193245 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:54.214548 25238 raft_consensus.cc:2393] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:54.190774 28443 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:54.192682 28221 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:54.182703 28217 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.182541 28477 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.230175 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:54.192294 28521 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 13:08:54.189183 27962 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.231469 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:54.233971 28528 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
W20260709 13:08:54.234748 28524 log.cc:927] Time spent T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:08:54.235320 28507 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:54.235926 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:54.253850 28369 raft_consensus.cc:3055] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:54.236619 25387 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:54.236848 28438 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:54.229492 28530 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:54.241147 28397 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:54.249935 25238 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:54.242372 28221 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:08:54.253773 28212 consensus_meta.cc:287] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.620s	user 0.009s	sys 0.002s
I20260709 13:08:54.263013 28212 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:54.247833 25243 raft_consensus.cc:2393] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 7: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:54.239794 28472 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.265252 25944 raft_consensus.cc:2393] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 7: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:54.256248 25245 raft_consensus.cc:2393] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 7: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:54.266126 25238 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260709 13:08:54.266664 25238 pending_rounds.cc:85] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 7
I20260709 13:08:54.266956 25238 pending_rounds.cc:107] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260709 13:08:54.267268 25238 raft_consensus.cc:2889] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:54.257217 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:54.261343 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:54.251984 27962 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:54.274586 28477 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:54.261763 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:54.257268 28369 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:54.279474 25395 raft_consensus.cc:2368] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7.
W20260709 13:08:54.280350 28526 log.cc:927] Time spent T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.150s	user 0.004s	sys 0.000s
I20260709 13:08:54.281498 28212 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:54.260694 28530 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:54.281692 28369 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.283823 28429 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:54.291589 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:54.304392 25936 raft_consensus.cc:2393] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 7: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:54.306234 25391 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: []
I20260709 13:08:54.294175 28472 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:54.307696 28369 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:54.310611 25390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41830 (request call id 502) took 1307 ms. Trace:
I20260709 13:08:54.311005 25390 rpcz_store.cc:276] 0709 13:08:53.003579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:53.008853 (+  5274us) service_pool.cc:224] Handling call
0709 13:08:54.310462 (+1301609us) spinlock_profiling.cc:243] Waited 550 ms on lock 0x7b54003e4754. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:54.310563 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1265213312}
W20260709 13:08:54.312978 28532 log.cc:927] Time spent T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 13:08:54.296083 28217 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:54.298169 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2018) took 1246 ms. Trace:
I20260709 13:08:54.299363 25387 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:08:54.299980 25389 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:08:54.300273 28530 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:54.302474 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:54.332345 25943 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:54.309033 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:54.329730 28517 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:54.334036 28517 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:54.336012 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 447310us
W20260709 13:08:54.336560 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:54.336994 28517 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:54.329947 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:54.324276 25392 rpcz_store.cc:276] 0709 13:08:53.051429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:53.754785 (+703356us) service_pool.cc:224] Handling call
0709 13:08:53.756945 (+  2160us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:08:53.802725 (+ 45780us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:53.802757 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:08:53.846962 (+ 44205us) log.cc:844] Serialized 52 byte log entry
0709 13:08:53.985634 (+138672us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:53.985687 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:53.985775 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:54.297948 (+312173us) raft_consensus.cc:1685] finished
0709 13:08:54.297971 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:54.298110 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19619,"raft.queue_time_us":112,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":42577,"thread_start_us":110121,"threads_started":1,"wal-append.queue_time_us":170798}
W20260709 13:08:54.331750 25287 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.062s	user 0.000s	sys 0.043s
I20260709 13:08:54.333277 25393 raft_consensus.cc:2376] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7. Re-sending same reply.
I20260709 13:08:54.342244 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2026) took 1288 ms. Trace:
I20260709 13:08:54.342697 25393 rpcz_store.cc:276] 0709 13:08:53.053533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:54.308747 (+1255214us) service_pool.cc:224] Handling call
0709 13:08:54.342165 (+ 33418us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:08:54.345795 25287 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.076s	user 0.000s	sys 0.043s
W20260709 13:08:54.346300 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 1.07651e+06us
I20260709 13:08:54.351274 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:54.353302 25209 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:54.353555 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:54.354208 25390 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:08:54.356168 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:54.356426 25209 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:54.349581 28517 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:54.357892 28310 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:54.358309 28310 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:54.359102 25209 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:54.357268 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:54.361476 25358 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:54.362496 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:54.358754 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:54.363380 25387 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:08:54.364102 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:54.364490 25387 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:54.366295 25209 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:54.369236 25390 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:08:54.370430 25209 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:54.372948 25358 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:54.374459 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:54.375186 25392 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 8: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:54.358667 28310 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:54.359157 28470 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:54.379741 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:54.375296 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:54.378327 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:54.377504 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:54.365751 28438 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:54.380460 25391 raft_consensus.cc:2393] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 8: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:54.381850 25393 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:54.366046 28517 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:08:54.383769 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:54.383973 28418 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:54.379002 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:54.385406 25392 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 8: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:54.385473 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:54.385653 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:54.383679 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:54.386369 25391 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 8: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:54.386265 25358 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:54.386572 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:54.387748 28418 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:54.356042 28221 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.390623 28221 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:54.378413 28533 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:54.391469 28221 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:08:54.392161 28533 raft_consensus.cc:697] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:54.393178 28533 consensus_queue.cc:237] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:54.394397 25393 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 7: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:54.395363 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:54.395889 25393 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:54.376719 28464 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:08:54.397594 28530 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.398583 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:54.399217 25392 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:08:54.400157 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:54.414363 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:54.415273 25391 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:54.400907 28385 consensus_queue.cc:260] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.374010 28212 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:08:54.376101 28535 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:54.419019 25392 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:08:54.430629 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:54.431380 25392 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:54.432410 25393 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:54.419950 25391 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:08:54.427560 25387 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:54.382875 28438 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:54.406164 28517 raft_consensus.cc:697] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:54.441780 25389 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:08:54.424681 28385 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:54.447361 25392 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
W20260709 13:08:54.448860 28338 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.105s	user 0.006s	sys 0.005s
W20260709 13:08:54.451871 28519 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.318s	user 0.002s	sys 0.000s
I20260709 13:08:54.445453 28535 raft_consensus.cc:697] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:54.406044 28507 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260709 13:08:54.455399 28517 consensus_queue.cc:237] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:54.456461 28447 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:54.445343 28329 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:54.447149 28438 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:54.473826 28535 consensus_queue.cc:237] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:54.477486 28447 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:54.465062 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 90249.4us
W20260709 13:08:54.464180 28338 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.120s	user 0.006s	sys 0.006s
I20260709 13:08:54.468438 28329 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:54.489005 28525 log.cc:927] Time spent T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.308s	user 0.000s	sys 0.001s
I20260709 13:08:54.514843 25394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2029) took 1460 ms. Trace:
I20260709 13:08:54.515326 25394 rpcz_store.cc:276] 0709 13:08:53.054282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:53.818231 (+763949us) service_pool.cc:224] Handling call
0709 13:08:53.850899 (+ 32668us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:53.851363 (+   464us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:53.851403 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:53.863226 (+ 11823us) log.cc:844] Serialized 26 byte log entry
0709 13:08:53.943574 (+ 80348us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:08:54.023284 (+ 79710us) log.cc:844] Serialized 14 byte log entry
0709 13:08:54.027160 (+  3876us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:54.027224 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:54.488253 (+461029us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:54.514531 (+ 26278us) raft_consensus.cc:1685] finished
0709 13:08:54.514639 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:54.514781 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1247,"prepare.queue_time_us":26339,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":2562,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":10076,"spinlock_wait_cycles":27109504,"thread_start_us":67171,"threads_started":1,"wal-append.queue_time_us":144972}
I20260709 13:08:54.517460 28095 raft_consensus.cc:1027] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: attempt to promote peer f646dfd532974dceadfe00005d2bbf59: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260709 13:08:54.517916 28359 raft_consensus.cc:3055] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:08:54.518332 28359 raft_consensus.cc:740] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:54.518429 25940 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:54.518632 28447 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:54.518996 25395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2053) took 1454 ms. Trace:
I20260709 13:08:54.519302 28359 consensus_queue.cc:260] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.519331 25395 rpcz_store.cc:276] 0709 13:08:53.064585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:53.646092 (+581507us) service_pool.cc:224] Handling call
0709 13:08:54.518814 (+872722us) spinlock_profiling.cc:243] Waited 632 ms on lock 0x7b5400283cd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:54.518941 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1454400896}
I20260709 13:08:54.506196 28334 raft_consensus.cc:3055] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:08:54.521557 28334 raft_consensus.cc:740] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:54.522202 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:54.522349 28334 consensus_queue.cc:260] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.523021 25394 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 7: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:54.523588 28334 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:54.495431 28463 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:54.528803 25359 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:08:54.528825 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:54.553607 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:54.563405 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:54.565883 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:54.569882 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 183666us
I20260709 13:08:54.570431 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:54.530032 28212 raft_consensus.cc:2764] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:08:54.572746 25358 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:54.504527 28530 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:54.587250 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 1970) took 1086 ms. Trace:
I20260709 13:08:54.581431 25244 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:54.587669 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:54.587630 25318 rpcz_store.cc:276] 0709 13:08:53.506329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:53.506493 (+   164us) service_pool.cc:224] Handling call
0709 13:08:54.587084 (+1080591us) spinlock_profiling.cc:243] Waited 92.7 ms on lock 0x7b5400275454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:54.587198 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":213185280}
W20260709 13:08:54.583997 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:54.588835 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:54.561622 28447 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:54.595860 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:54.597797 25945 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:54.598351 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:54.495422 28438 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:54.590580 28418 raft_consensus.cc:2764] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 13:08:54.591598 28472 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:08:54.609241 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:54.562711 28310 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:54.593175 28222 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.728s	user 0.013s	sys 0.027s
I20260709 13:08:54.616063 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:54.627244 25242 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9. Ops: []
I20260709 13:08:54.627283 25238 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:54.598383 28212 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:54.610211 25941 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:54.620710 25945 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:54.628315 25239 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 13:08:54.593667 28464 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260709 13:08:54.629323 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:54.586185 28435 consensus_queue.cc:1059] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 13:08:54.591259 27962 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.031s
W20260709 13:08:54.627434 28222 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.762s	user 0.013s	sys 0.027s
W20260709 13:08:54.645143 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 56388.1us
I20260709 13:08:54.642333 28418 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:54.632472 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:54.636656 25244 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 13:08:54.646601 25358 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:54.640009 25240 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9. Ops: []
I20260709 13:08:54.649645 28222 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:54.650249 28222 raft_consensus.cc:697] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:54.647369 25244 pending_rounds.cc:85] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 6
I20260709 13:08:54.651306 25244 pending_rounds.cc:107] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 13:08:54.651643 25244 raft_consensus.cc:2889] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:54.631707 28542 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:54.605317 28369 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260709 13:08:54.643704 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:54.641546 25911 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:54.672365 25271 heartbeater.cc:646] Failed to heartbeat to 127.23.140.126:33305 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.23.140.126:33305 timed out after 15.000s (SENT)
I20260709 13:08:54.635953 25238 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:08:54.606487 28447 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.662843 25358 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:54.675071 28439 raft_consensus.cc:3055] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:54.675531 25909 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:54.675679 28439 raft_consensus.cc:740] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:54.675997 25358 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:54.667972 25938 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:08:54.676869 25944 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:08:54.672780 25911 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:54.681488 28551 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:54.676407 28505 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260709 13:08:54.683882 28472 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:54.686543 28326 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:54.688680 25388 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:08:54.681819 28463 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:08:54.687026 28326 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:54.701493 28326 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:54.700256 28429 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20260709 13:08:54.661962 28418 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:08:54.708357 28418 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:54.680527 28435 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 13:08:54.710095 28546 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.132s	user 0.003s	sys 0.000s
I20260709 13:08:54.690717 28442 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:08:54.690972 28329 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:54.693835 28472 raft_consensus.cc:697] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:54.711643 28472 consensus_queue.cc:237] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.694646 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:54.696085 28437 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:08:54.697857 28310 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:54.678596 28385 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:54.676946 28533 consensus_queue.cc:1059] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
W20260709 13:08:54.709957 25212 leader_election.cc:336] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.834s (SENT)
I20260709 13:08:54.672698 28222 consensus_queue.cc:237] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.718607 25212 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:54.716069 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 83950.8us
I20260709 13:08:54.682476 28551 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:54.723140 25212 leader_election.cc:336] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.844s (SENT)
I20260709 13:08:54.723810 25212 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:54.716688 28523 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.005s	sys 0.002s
I20260709 13:08:54.688024 28439 consensus_queue.cc:260] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:54.717222 28385 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:54.729341 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:54.725378 28523 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.006s	sys 0.002s
I20260709 13:08:54.734742 28212 consensus_queue.cc:1059] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:08:54.735795 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:54.737370 25212 leader_election.cc:336] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.859s (SENT)
I20260709 13:08:54.739533 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:54.726119 28517 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:54.725874 28334 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 13:08:54.745447 25140 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39572 (request call id 289) took 15515 ms (client timeout 15000 ms). Trace:
I20260709 13:08:54.751487 25239 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:08:54.740267 28418 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:54.736405 28385 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:54.754709 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:54.761799 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:54.741994 28440 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:54.755676 28418 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.744807 28326 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.817401 25239 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
W20260709 13:08:54.818461 25212 leader_election.cc:336] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.918s (SENT)
W20260709 13:08:54.821568 25212 leader_election.cc:336] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.918s (SENT)
I20260709 13:08:54.822515 28222 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:54.824404 28222 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.825519 28418 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:54.826725 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:54.827554 28222 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:54.828080 28326 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:54.724817 28551 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:54.833438 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:54.835067 28438 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:54.841333 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 380819us
I20260709 13:08:54.842341 28506 raft_consensus.cc:3055] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:54.842720 28506 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:54.843087 28234 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:54.843704 28361 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:54.843465 28506 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.843578 28234 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:54.844264 28361 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.845561 28338 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:54.845822 28234 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:54.846061 28338 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.846376 28234 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:08:54.848217 28338 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:54.850669 28359 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:54.851336 25909 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:54.851914 28361 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:54.746050 25140 rpcz_store.cc:269] 0709 13:08:39.229778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:39.229950 (+   172us) service_pool.cc:224] Handling call
0709 13:08:40.270092 (+1040142us) spinlock_profiling.cc:243] Waited 74.4 ms on lock 0x7b5400470ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1c090 00007f2976cbaf3a 00007f297ca711cc 00007f297cb6f37e 00007f297b0302a4 00007f297b030232 00007f297b030161 00007f297b0300dc 00007f297b02f392 00007f2974fed48c 00007f2974fec8c6
0709 13:08:54.745344 (+14475252us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:08:48.596421 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:08:48.599209 (+  2788us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:08:48.599294 (+    85us) write_op.cc:435] Acquired schema lock
0709 13:08:48.599318 (+    24us) tablet.cc:662] Decoding operations
0709 13:08:48.718212 (+118894us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:08:48.759556 (+ 41344us) write_op.cc:641] Partition lock acquired for write op
0709 13:08:48.765677 (+  6121us) tablet.cc:685] Acquiring locks for 16 operations
0709 13:08:49.216267 (+450590us) tablet.cc:701] Row locks acquired
0709 13:08:49.227438 (+ 11171us) write_op.cc:260] PREPARE: finished
0709 13:08:49.279635 (+ 52197us) write_op.cc:270] Start()
0709 13:08:49.298128 (+ 18493us) write_op.cc:276] Timestamp: P: 1783602529267985 usec, L: 0
0709 13:08:49.298166 (+    38us) op_driver.cc:348] REPLICATION: starting
0709 13:08:49.364652 (+ 66486us) log.cc:844] Serialized 6395 byte log entry
0709 13:08:50.048350 (+683698us) spinlock_profiling.cc:243] Waited 39.8 ms on lock 0x7b18002c25a0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:08:50.048687 (+   337us) op_driver.cc:464] REPLICATION: finished
0709 13:08:50.238033 (+189346us) write_op.cc:301] APPLY: starting
0709 13:08:50.238141 (+   108us) tablet.cc:1367] starting BulkCheckPresence
0709 13:08:50.238477 (+   336us) tablet.cc:1370] finished BulkCheckPresence
0709 13:08:50.238519 (+    42us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:08:50.382818 (+144299us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:08:50.383344 (+   526us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
0709 13:08:50.383415 (+    71us) write_op.cc:312] APPLY: finished
0709 13:08:50.383979 (+   564us) log.cc:844] Serialized 113 byte log entry
0709 13:08:50.384221 (+   242us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:50.384436 (+   215us) write_op.cc:454] Released schema lock
0709 13:08:50.384526 (+    90us) write_op.cc:341] FINISH: Updating metrics
0709 13:08:50.385074 (+   548us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:50.385112 (+    38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2452782,"reported_tablets":42,"spinlock_wait_cycles":171054336,"child_traces":[["op",{"apply.queue_time_us":189074,"apply.run_cpu_time_us":9123,"apply.run_wall_time_us":146997,"num_ops":16,"prepare.queue_time_us":1063260,"prepare.run_cpu_time_us":54353,"prepare.run_wall_time_us":860941,"raft.queue_time_us":43719,"raft.run_cpu_time_us":17731,"raft.run_wall_time_us":179574,"replication_time_us":750338,"spinlock_wait_cycles":91467136,"thread_start_us":560217,"threads_started":4,"wal-append.queue_time_us":67734,"wal-append.run_cpu_time_us":22927,"wal-append.run_wall_time_us":1960730}]]}
I20260709 13:08:54.848748 28095 consensus_queue.cc:1059] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:08:54.864012 25212 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:54.848137 28234 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:08:54.854900 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:54.859082 28437 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:54.861879 28511 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:54.870922 28437 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.871153 28511 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.873348 28437 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:54.856241 25209 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:54.876227 25285 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:54.877230 25285 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:54.878270 25285 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:54.876250 28406 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:54.876587 28551 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:54.879253 28406 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:54.880227 28410 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:54.880650 28410 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:54.856256 25390 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
W20260709 13:08:54.866127 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 190181us
I20260709 13:08:54.888650 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:54.889302 25940 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
W20260709 13:08:54.893529 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 71987.8us
W20260709 13:08:54.894800 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:54.898329 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:54.899554 25245 raft_consensus.cc:2393] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:54.881104 28410 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:54.832844 25139 catalog_manager.cc:5697] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 3 to 4, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 4 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:54.918887 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:54.919806 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:54.920718 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:54.922091 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:54.922338 25212 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:54.923642 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:54.924283 25318 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:08:54.922851 25243 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:08:54.927059 25939 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:54.929725 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:54.930446 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:54.921492 25241 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:08:54.877228 28217 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:54.934048 25318 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:54.934460 25212 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:54.935729 25139 catalog_manager.cc:5697] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 5 to 9, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 9 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:54.937731 25317 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 13:08:54.936241 25316 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:54.939922 25312 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 13:08:54.940577 25312 pending_rounds.cc:85] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 6
I20260709 13:08:54.940872 25312 pending_rounds.cc:107] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 13:08:54.941627 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:54.942862 25312 raft_consensus.cc:2889] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:54.944872 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:54.945856 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:54.899991 28338 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:08:54.947188 25212 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:54.901481 28521 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:08:54.918421 28443 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:54.953332 25139 catalog_manager.cc:5697] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 5 to 6, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 6 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:08:54.956877 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:54.959818 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:54.878494 28087 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 13:08:54.956952 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:54.959925 25312 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
W20260709 13:08:54.955829 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 76012.1us
I20260709 13:08:54.931062 28234 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:54.969831 25317 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 7: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:54.968231 28406 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:54.924713 28437 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:54.968070 28338 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:54.971145 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:54.970322 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 127609us
I20260709 13:08:54.944985 28470 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:08:54.974956 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:54.975272 25909 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:54.978116 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:54.952315 28212 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:54.975068 25284 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:54.934248 28548 consensus_queue.cc:1059] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 13:08:54.954767 28217 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.986106 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:54.985877 25139 catalog_manager.cc:5697] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 5 to 6, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 6 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:54.938510 28549 consensus_queue.cc:1059] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260709 13:08:54.969092 28533 raft_consensus.cc:2764] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:08:54.993685 25358 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:54.991032 25139 catalog_manager.cc:5697] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:08:54.993815 25212 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:54.930790 25315 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:08:54.997622 25284 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:54.977677 28463 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:54.976069 28550 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:54.974826 28551 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:55.006348 25315 raft_consensus.cc:606] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:55.009532 25212 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:54.939714 28385 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:54.979796 28443 raft_consensus.cc:697] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:54.985846 28530 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:54.982357 28472 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:55.014976 25320 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:55.006621 28521 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:08:54.973645 28517 consensus_queue.cc:1059] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260709 13:08:55.017982 28520 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:54.976296 28535 consensus_queue.cc:1059] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260709 13:08:55.012190 25139 catalog_manager.cc:5697] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:08:55.024379 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:54.991642 28212 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:55.026058 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:55.026975 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:55.015650 25212 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:54.979214 28437 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:55.032730 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 84303.3us
I20260709 13:08:55.065174 25287 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:55.010715 28470 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:55.042897 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:54.989691 28108 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20260709 13:08:54.979566 28338 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:55.048696 25391 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:55.049448 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:55.011632 28511 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:55.052739 25939 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:08:55.053357 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:55.025229 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:55.011631 28557 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:55.062721 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:55.072093 25392 raft_consensus.cc:2393] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 4: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:55.033910 28418 raft_consensus.cc:3055] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:08:55.033768 28464 raft_consensus.cc:3055] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:08:55.043450 28334 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:55.035421 28472 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:55.071563 25394 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 8: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:55.069659 25139 catalog_manager.cc:5697] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:08:55.014292 28463 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:55.079816 25393 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 5: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:55.034192 27962 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:08:55.096782 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 81423.7us
W20260709 13:08:55.081921 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 84433.2us
I20260709 13:08:55.051162 28517 consensus_queue.cc:1243] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer a2a77ecf2c0941618b5bc79d4c04db61 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 13:08:55.055269 28549 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:55.115191 28549 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:55.117336 25390 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:08:55.096563 28472 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:55.135219 28437 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:08:55.136571 28385 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:55.038928 28530 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:55.141912 28338 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:55.142875 28530 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:55.080780 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:55.073278 28557 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:55.143836 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:55.144040 28557 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:55.144707 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:55.145452 25941 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 13:08:55.072790 28418 raft_consensus.cc:740] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:55.106066 28310 raft_consensus.cc:3055] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:55.079663 28464 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:08:55.111867 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 96317.3us
I20260709 13:08:55.068233 28535 consensus_queue.cc:1243] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer a2a77ecf2c0941618b5bc79d4c04db61 log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:08:55.100978 28443 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:55.070348 28570 consensus_queue.cc:1059] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 13:08:55.095501 28568 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:08:55.133952 28217 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:55.134542 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:55.143146 25139 catalog_manager.cc:5697] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:55.168776 25139 catalog_manager.cc:5697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:08:55.074275 28520 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:08:55.106967 28564 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 13:08:55.170867 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 57735.8us
W20260709 13:08:55.157764 28560 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
I20260709 13:08:55.164377 28463 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:55.205106 25212 leader_election.cc:400] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7.
I20260709 13:08:55.206091 25212 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:55.207268 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:55.157119 28555 log.cc:927] Time spent T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.172s	user 0.000s	sys 0.000s
I20260709 13:08:55.196367 28574 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:55.208400 28549 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:55.208616 28234 raft_consensus.cc:697] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:08:55.199788 28558 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.120s	user 0.004s	sys 0.000s
I20260709 13:08:55.209528 28535 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:55.167318 28568 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:55.153169 28095 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:55.227658 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:55.218320 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:55.221591 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:55.176728 28511 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:55.171224 28520 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:55.229296 25312 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:08:55.215917 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 168932us
I20260709 13:08:55.151866 28310 raft_consensus.cc:740] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:55.221705 25139 catalog_manager.cc:5697] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 7, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 7 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:55.225278 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:55.226218 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:55.222576 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:55.230964 25945 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:55.231912 25212 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:55.228803 25316 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:08:55.235415 25314 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:55.236027 25212 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:55.236596 25314 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:55.217377 28439 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:55.240217 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:55.196007 28334 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.029s
I20260709 13:08:55.221366 28442 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:55.248894 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:55.239481 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:55.227341 28568 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:08:55.222951 28572 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:55.244784 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:55.273180 25321 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:55.250560 25238 raft_consensus.cc:2410] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:08:55.166447 28418 consensus_queue.cc:260] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:55.232900 28535 raft_consensus.cc:697] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:55.265329 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:55.270362 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:55.275215 28310 consensus_queue.cc:260] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:55.276245 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:55.241550 28582 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:55.233994 28464 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:55.279541 25241 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
W20260709 13:08:55.282753 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 53674.2us
I20260709 13:08:55.283324 25237 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:55.237210 28095 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:55.234520 28520 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:08:55.239413 28511 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:55.321679 28310 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:55.323544 28511 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:55.325610 28535 consensus_queue.cc:237] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:55.328014 28404 raft_consensus.cc:1027] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: attempt to promote peer f646dfd532974dceadfe00005d2bbf59: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260709 13:08:55.281551 28212 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:55.329633 28212 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:55.330073 28212 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:55.332082 28212 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:55.333004 25313 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 13:08:55.333313 28369 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:55.333853 28369 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:55.336301 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:55.338044 28418 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:55.324993 25139 catalog_manager.cc:5697] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 1 to 6. New cstate: current_term: 6 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:08:55.342681 28234 consensus_queue.cc:237] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:55.345377 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 49332us
I20260709 13:08:55.345825 28470 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:55.346696 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:55.276536 28581 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:08:55.299337 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 46481us
I20260709 13:08:55.353469 25359 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:55.293964 28584 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 13:08:55.310230 28557 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:55.298064 28574 consensus_queue.cc:1059] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20260709 13:08:55.314720 28447 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:55.366825 25389 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:08:55.320493 28410 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:55.320493 28549 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:55.349658 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:55.349828 25388 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 13:08:55.340657 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:55.353833 25128 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f75b60146e7d4b9b9954b4fb2c414679 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 2329e428ebbf484fa3ea9452133df567 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:55.299501 25312 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:08:55.367645 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:55.387451 25392 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:55.314795 28095 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:55.369644 28464 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:55.371011 28472 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:55.369206 25321 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:08:55.373365 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:55.373890 25285 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:55.375231 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:55.408793 25243 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 13:08:55.377854 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:55.381522 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:55.388757 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 323680us
I20260709 13:08:55.280052 28572 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:55.414649 25285 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:55.416543 25287 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:55.417014 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:55.420442 25392 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:55.422441 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:55.374722 28440 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:55.403774 28369 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:55.454419 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:55.448330 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 108876us
I20260709 13:08:55.451185 25940 raft_consensus.cc:2393] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 9: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:55.452071 25285 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:55.372900 28568 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:55.413513 28442 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:55.459678 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:55.465631 25392 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
W20260709 13:08:55.465713 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 179473us
I20260709 13:08:55.467067 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:55.456658 25287 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:55.435386 28506 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:55.473084 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:55.438140 28440 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:55.456375 28087 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:55.482090 25388 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:55.475584 27962 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.009s
I20260709 13:08:55.485869 28562 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:55.487198 28511 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:08:55.487821 28579 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:55.461054 28404 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:55.405275 28549 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:55.460841 28410 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:55.463954 28186 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:08:55.463660 28406 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:55.468663 28583 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:55.482422 28440 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:55.539942 28521 raft_consensus.cc:3055] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:55.484998 28568 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:55.551321 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:55.498008 28562 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:55.515957 28574 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:55.532858 28404 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:55.582707 25243 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:55.551190 28521 raft_consensus.cc:740] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:55.581794 28404 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:55.557451 28406 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:55.586273 28585 log.cc:927] Time spent T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 13:08:55.500146 28087 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:55.582053 28574 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:55.581949 28592 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:55.574774 28326 raft_consensus.cc:670] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } [suppressed 3 similar messages]
W20260709 13:08:55.593410 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:55.616361 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:55.616984 25944 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 10
W20260709 13:08:55.618439 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 117818us
I20260709 13:08:55.594462 28572 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:55.596575 25358 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.196s (SENT) [suppressed 33 similar messages]
I20260709 13:08:55.622690 25936 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 13:08:55.599743 28549 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:55.600518 28472 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:55.600638 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:55.600551 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:55.571451 28562 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:55.610657 28557 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:55.625000 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:55.629184 25937 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:55.628145 25938 raft_consensus.cc:2410] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 13:08:55.607726 28442 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:55.607869 28095 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:55.629663 25942 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:08:55.640097 25911 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:55.642019 25389 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:55.643059 25911 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:55.644434 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 158946us
I20260709 13:08:55.646552 25911 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:55.631633 25243 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:08:55.647691 25911 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:55.651997 25392 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:55.653981 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:55.664484 25936 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:55.672940 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:55.629241 28359 consensus_queue.cc:1059] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 13:08:55.673728 25287 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:55.673754 25942 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:08:55.652046 28472 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:08:55.644759 28442 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:55.683171 25358 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:55.685086 25392 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:08:55.625576 28592 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:55.644814 28505 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:08:55.696396 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:55.687852 25287 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:55.699990 25936 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:08:55.684562 28369 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:08:55.627399 28521 consensus_queue.cc:260] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:55.703702 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:55.705591 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:55.716482 25287 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:55.661691 28464 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.028s
I20260709 13:08:55.669934 28511 consensus_queue.cc:1059] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260709 13:08:55.675210 28568 consensus_queue.cc:1059] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260709 13:08:55.723037 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:55.724695 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:55.691301 28359 consensus_queue.cc:1243] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:08:55.683724 28579 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 13:08:55.689244 28234 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:08:55.720178 28505 raft_consensus.cc:697] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:08:55.767947 28597 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
W20260709 13:08:55.769397 28599 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 13:08:55.769701 25942 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
W20260709 13:08:55.761541 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 113934us
I20260709 13:08:55.756558 27962 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:08:55.771026 27962 raft_consensus.cc:697] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:55.771899 27962 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:55.715533 28572 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:08:55.725221 28410 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:55.777571 28574 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:55.780423 25245 raft_consensus.cc:2410] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 13:08:55.691005 28594 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:55.781334 28594 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:55.707649 28472 raft_consensus.cc:697] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 6 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:55.781796 28087 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:55.782389 28472 consensus_queue.cc:237] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:55.783254 28406 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:55.725488 28338 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:55.714450 28369 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:55.788398 28369 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:08:55.781719 25944 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:08:55.781751 28594 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:08:55.767220 28590 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 13:08:55.787992 28592 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:55.718062 28557 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:55.799793 25386 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260709 13:08:55.801608 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 78676us
I20260709 13:08:55.803824 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:55.804513 25321 raft_consensus.cc:2368] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7.
I20260709 13:08:55.805328 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:55.805667 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:55.793634 28521 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:55.806839 28326 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:55.814005 28326 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:55.816305 28326 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:55.808885 28579 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:55.818442 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:55.820986 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:55.762980 28588 consensus_queue.cc:1059] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 13:08:55.821730 25321 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:08:55.823915 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:55.826843 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:55.828434 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:55.828998 25315 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:55.808063 28557 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:55.834386 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:55.835175 25321 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:08:55.835819 28579 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:55.839301 28557 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:08:55.856945 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:55.768729 28505 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:55.857308 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 48854.7us
I20260709 13:08:55.836702 28579 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:55.840806 28611 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.021s
I20260709 13:08:55.853188 27962 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:55.868657 25312 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 13:08:55.870793 28338 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:55.893728 25312 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9. Ops: [6.4-8.5]
W20260709 13:08:55.899230 28600 log.cc:927] Time spent T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 13:08:55.899093 28563 raft_consensus.cc:3055] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:08:55.912052 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:55.913417 25312 raft_consensus.cc:2410] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 13:08:55.889369 28338 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:08:55.924618 28520 consensus_meta.cc:287] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.567s	user 0.008s	sys 0.000s
I20260709 13:08:55.908063 28563 raft_consensus.cc:740] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:55.964339 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:55.932091 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:55.968314 25313 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:55.972416 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:55.981765 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:55.868508 27962 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:55.985004 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:55.999934 25318 raft_consensus.cc:2410] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:08:56.006412 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:56.019088 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:55.894554 28404 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.028904 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:56.030122 25318 raft_consensus.cc:2410] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:08:56.045782 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:55.984167 28087 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:56.049217 28087 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:56.049355 28594 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.051512 28087 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:56.052042 28594 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:55.929286 28520 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:56.054461 28606 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.293s	user 0.002s	sys 0.000s
I20260709 13:08:56.055022 28520 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:55.926885 28338 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:56.056239 25313 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:56.062033 25318 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:08:56.063180 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:56.061558 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:56.063939 25317 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:56.064172 25394 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:56.065894 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:55.984284 28410 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:56.068238 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:55.996995 28549 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:56.073395 25315 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:56.082715 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:56.085513 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:56.089481 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:56.090030 25358 leader_election.cc:400] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7.
I20260709 13:08:56.090541 25358 leader_election.cc:403] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:08:56.090726 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:56.091506 25390 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:56.092546 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:56.093286 25358 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:56.099725 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:56.100255 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:56.100517 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:56.100957 25318 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:08:56.102212 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:56.102483 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:56.103006 25315 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:56.105679 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:56.111696 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:56.112732 25390 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:08:56.112954 25315 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:08:56.114408 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:56.116232 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:56.116914 25318 raft_consensus.cc:2368] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10.
I20260709 13:08:56.115213 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:56.117633 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:56.120417 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:56.121476 25386 raft_consensus.cc:2393] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 7: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:56.122485 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:56.123215 25386 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:56.111836 25317 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:08:56.131359 25942 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 6 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: []
I20260709 13:08:56.147051 25394 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:08:56.148453 25395 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:08:56.111330 28439 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7.
W20260709 13:08:56.177361 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 76193.5us
I20260709 13:08:56.110394 28464 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:56.110597 28087 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:56.172195 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:56.192607 25318 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:56.168385 28361 raft_consensus.cc:3055] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:56.097738 28549 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:56.185317 25390 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:08:56.181617 25287 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:56.233338 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:56.255725 25313 raft_consensus.cc:2410] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:08:56.256031 25239 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:56.226991 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 1996) took 1006 ms. Trace:
I20260709 13:08:56.207772 28578 raft_consensus.cc:993] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567: : Instructing follower 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:08:56.231864 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:56.257248 25316 rpcz_store.cc:276] 0709 13:08:55.220211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:55.234253 (+ 14042us) service_pool.cc:224] Handling call
0709 13:08:55.234490 (+   237us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:55.238358 (+  3868us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:08:55.252795 (+ 14437us) log.cc:844] Serialized 14 byte log entry
0709 13:08:55.372909 (+120114us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b180042c100. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f297685223a 00007f2976cbca2a
0709 13:08:55.670834 (+297925us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:55.670888 (+    54us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:08:55.670940 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:56.226792 (+555852us) spinlock_profiling.cc:243] Waited 3.67 ms on lock 0x7b5400274f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:56.226836 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:56.226941 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21031,"prepare.queue_time_us":21445,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":19197,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":3947,"spinlock_wait_cycles":39029376,"thread_start_us":280361,"threads_started":1,"wal-append.queue_time_us":499140}
I20260709 13:08:56.209978 28361 raft_consensus.cc:740] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:56.210718 28087 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:56.301702 25284 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:56.212325 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:56.237555 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2080) took 1197 ms. Trace:
W20260709 13:08:56.195047 28615 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.289s	user 0.001s	sys 0.000s
I20260709 13:08:56.181960 25358 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:56.258287 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 76798.7us
I20260709 13:08:56.306370 25287 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:56.239097 28562 consensus_meta.cc:287] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.600s	user 0.008s	sys 0.000s
I20260709 13:08:56.263276 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:56.311103 25394 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:08:56.257246 25386 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:56.151850 28369 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.303051 25393 rpcz_store.cc:276] 0709 13:08:55.023526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:55.034089 (+ 10563us) service_pool.cc:224] Handling call
0709 13:08:56.220024 (+1185935us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b54001f59d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:56.221051 (+  1027us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28177280}
I20260709 13:08:56.224730 25388 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:08:56.235172 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:56.307176 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:08:56.305080 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 123197us
I20260709 13:08:56.228200 28522 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:08:56.223204 28440 consensus_meta.cc:287] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.654s	user 0.007s	sys 0.002s
I20260709 13:08:56.313429 25911 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:56.300757 28561 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:56.315254 28562 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:56.313349 28628 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:56.322268 28562 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:56.317721 28440 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.230794 27962 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:56.315188 25937 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 8
I20260709 13:08:56.229961 28404 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:56.323688 28361 consensus_queue.cc:260] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.324584 28440 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:56.324903 28361 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:56.316071 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:56.307600 25387 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:08:56.319370 28437 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:56.307756 28095 raft_consensus.cc:1081] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Signalling peer 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:08:56.327281 25391 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:56.329047 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:56.330497 25358 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:56.331466 28463 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:56.332260 28463 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:56.332356 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:56.333510 28463 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:08:56.333865 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:56.221036 28557 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.334858 28463 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:08:56.337191 27962 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:56.341194 28437 raft_consensus.cc:697] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:56.326179 28522 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.344314 28533 raft_consensus.cc:3055] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:08:56.363654 28437 consensus_queue.cc:237] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.358738 28570 raft_consensus.cc:3055] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:56.385040 28570 raft_consensus.cc:740] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:56.389796 28570 consensus_queue.cc:260] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.391083 28570 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:56.238603 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:56.435362 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 196536us
I20260709 13:08:56.444361 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:56.234272 25237 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260709 13:08:56.226317 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:56.226154 25212 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:56.450631 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:56.227046 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:56.452471 25239 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:08:56.453487 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:56.452785 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 226567us
I20260709 13:08:56.455148 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:56.456342 25237 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:08:56.457937 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:56.459707 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:56.460742 25212 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:56.468804 25243 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:08:56.461607 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:56.472254 25237 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
W20260709 13:08:56.474007 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 172253us
I20260709 13:08:56.475260 25284 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:56.478935 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 145019us
I20260709 13:08:56.486065 28310 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:08:56.486897 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 180509us
I20260709 13:08:56.488366 25287 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:56.488516 28470 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:56.352108 27962 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:56.489548 28217 raft_consensus.cc:2764] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 13:08:56.489815 28470 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:08:56.491815 28517 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:56.492327 28095 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:08:56.493040 28578 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:56.494735 28572 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:08:56.495446 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:56.495375 28234 raft_consensus.cc:3055] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:56.496378 28438 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:56.528172 28438 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:56.548380 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 77966.3us
W20260709 13:08:56.548833 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 99807.4us
I20260709 13:08:56.531643 28234 raft_consensus.cc:740] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:56.551214 28234 consensus_queue.cc:260] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:56.553054 28222 raft_consensus.cc:3055] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:56.535821 28095 raft_consensus.cc:697] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:56.362371 28533 raft_consensus.cc:740] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:56.558809 28521 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:56.559092 28470 consensus_queue.cc:1243] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer 2329e428ebbf484fa3ea9452133df567 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 13:08:56.559818 28470 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:08:56.561965 28535 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:56.562455 28535 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:56.562902 28535 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:56.563210 28554 consensus_queue.cc:1059] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:08:56.546446 28440 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:56.565821 28440 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:56.366731 28549 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:56.567842 28463 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.021s
I20260709 13:08:56.570241 28095 consensus_queue.cc:237] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.571158 28541 raft_consensus.cc:3055] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:56.571748 28541 raft_consensus.cc:740] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:56.559710 28521 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.573242 25209 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:08:56.583020 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:56.585485 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 97363.8us
I20260709 13:08:56.586637 25287 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:56.590190 28217 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:56.590566 28561 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:56.590206 28579 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:56.591938 28579 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.593247 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 1682) took 1218 ms. Trace:
I20260709 13:08:56.593807 25245 rpcz_store.cc:276] 0709 13:08:55.374830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:55.421916 (+ 47086us) service_pool.cc:224] Handling call
0709 13:08:56.592766 (+1170850us) spinlock_profiling.cc:243] Waited 354 ms on lock 0x7b54002d0fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:56.592990 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":813099264}
I20260709 13:08:56.595031 28579 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:56.597304 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:56.601464 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 47328.3us
I20260709 13:08:56.602698 28438 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:56.582804 25209 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:56.604044 25239 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:08:56.603003 25285 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:56.556563 28234 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:56.554838 28222 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:56.608819 28310 raft_consensus.cc:697] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:56.568928 28440 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:56.617080 28438 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:08:56.558384 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:56.593307 28578 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.619659 27962 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:56.634016 28557 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:56.635407 28578 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:56.637867 28222 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:56.629288 28438 raft_consensus.cc:697] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:08:56.641412 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 99077.6us
I20260709 13:08:56.647801 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1855) took 1296 ms. Trace:
I20260709 13:08:56.648062 28438 consensus_queue.cc:237] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:56.648406 25238 rpcz_store.cc:276] 0709 13:08:55.351402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:55.351566 (+   164us) service_pool.cc:224] Handling call
0709 13:08:55.351888 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:55.352114 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:08:55.373645 (+ 21531us) log.cc:844] Serialized 14 byte log entry
0709 13:08:56.218999 (+845354us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b18002caa00. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b3fc0a 00007f297d27f219 00007f2973bc7a40 00007f2973bc24e4 00007f2973bc2455 00007f2976caa060 00007f2976cbc716 00007f2976ccd287 00007f2976ccd208
0709 13:08:56.618729 (+399730us) log.cc:844] Serialized 14 byte log entry
0709 13:08:56.647045 (+ 28316us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:56.647126 (+    81us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:08:56.647258 (+   132us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:56.647371 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:56.647606 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13148,"prepare.queue_time_us":317297,"prepare.run_cpu_time_us":119,"prepare.run_wall_time_us":101,"raft.queue_time_us":847440,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"spinlock_wait_cycles":36850432,"thread_start_us":422305,"threads_started":2}
I20260709 13:08:56.650616 28338 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:56.624666 28520 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:56.657881 28520 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.658803 28522 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:56.660697 28520 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:56.614470 28544 consensus_queue.cc:1059] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.090s
I20260709 13:08:56.660392 28533 consensus_queue.cc:260] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:56.630379 28554 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:08:56.677471 25285 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:56.623953 28579 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20260709 13:08:56.673808 28557 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:08:56.687206 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:56.684031 28554 raft_consensus.cc:697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:56.691259 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:56.692957 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 110358us
I20260709 13:08:56.642572 28541 consensus_queue.cc:260] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.678573 28369 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:56.696244 25389 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:08:56.700183 28618 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
W20260709 13:08:56.700234 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:56.698143 28613 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:56.722187 28613 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:56.722410 28418 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:56.706130 25395 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260709 13:08:56.723304 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:56.723457 28418 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.727388 25238 raft_consensus.cc:2393] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 9: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:56.727274 28616 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:08:56.728193 28418 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:56.709092 28310 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.710306 28410 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:56.718195 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:56.719174 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:56.746001 25394 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:08:56.746587 28521 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:56.746645 25244 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
W20260709 13:08:56.747159 28609 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.990s	user 0.004s	sys 0.018s
I20260709 13:08:56.720091 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:56.747970 25241 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
W20260709 13:08:56.721732 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 79344.7us
I20260709 13:08:56.744136 27962 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.722666 28613 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:56.703802 28338 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:56.708093 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:56.732640 25908 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:56.728390 25358 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:56.732916 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:56.733773 28642 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:56.734012 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:56.734377 28533 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:56.735246 28541 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:56.735870 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 550) took 1845 ms. Trace:
I20260709 13:08:56.738391 25937 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:56.755482 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1851) took 1410 ms. Trace:
I20260709 13:08:56.759707 25209 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:56.761776 25389 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:08:56.762247 25945 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:08:56.754721 28517 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:08:56.766144 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1688) took 1059 ms. Trace:
I20260709 13:08:56.768234 25395 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:56.773074 28520 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:56.758539 25240 rpcz_store.cc:276] 0709 13:08:54.889950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:54.890445 (+   495us) service_pool.cc:224] Handling call
0709 13:08:55.316573 (+426128us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:55.316923 (+   350us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:55.316951 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:55.317389 (+   438us) log.cc:844] Serialized 26 byte log entry
0709 13:08:55.474277 (+156888us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:55.474316 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:55.474392 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:55.844383 (+369991us) raft_consensus.cc:1685] finished
0709 13:08:55.844404 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:56.735413 (+891009us) spinlock_profiling.cc:243] Waited 426 ms on lock 0x7b54003c2dd0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:56.735679 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2366,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":74,"spinlock_wait_cycles":979139456,"thread_start_us":156522,"threads_started":1,"wal-append.queue_time_us":325873}
I20260709 13:08:56.767251 25246 rpcz_store.cc:276] 0709 13:08:55.618652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:55.635084 (+ 16432us) service_pool.cc:224] Handling call
0709 13:08:55.647371 (+ 12287us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:56.226424 (+579053us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:56.226457 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:56.226903 (+   446us) log.cc:844] Serialized 26 byte log entry
0709 13:08:56.643116 (+416213us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:08:56.662059 (+ 18943us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:56.669431 (+  7372us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:56.669496 (+    65us) raft_consensus.cc:1685] finished
0709 13:08:56.669512 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:56.678232 (+  8720us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":578681,"raft.queue_time_us":580808,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"thread_start_us":391160,"threads_started":1,"wal-append.queue_time_us":323128}
W20260709 13:08:56.793402 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 111048us
I20260709 13:08:56.793642 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:56.775918 28520 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:56.793546 25321 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:08:56.796308 25284 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:56.796918 28604 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 13:08:56.758668 25242 rpcz_store.cc:276] 0709 13:08:55.345341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:55.348002 (+  2661us) service_pool.cc:224] Handling call
0709 13:08:55.348280 (+   278us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:55.348580 (+   300us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:55.348606 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:55.349002 (+   396us) log.cc:844] Serialized 26 byte log entry
0709 13:08:55.565968 (+216966us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:08:55.566019 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:55.566107 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:56.732679 (+1166572us) spinlock_profiling.cc:243] Waited 2.23 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:56.749107 (+ 16428us) raft_consensus.cc:1685] finished
0709 13:08:56.749141 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:56.755255 (+  6114us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":852593,"raft.queue_time_us":1104,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":9662,"spinlock_wait_cycles":5134080,"thread_start_us":216567,"threads_started":1,"wal-append.queue_time_us":331462}
I20260709 13:08:56.798259 28418 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:56.801851 28369 consensus_queue.cc:1059] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:08:56.803162 28616 raft_consensus.cc:3055] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:08:56.803661 28616 raft_consensus.cc:740] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:56.804322 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:56.805279 25312 raft_consensus.cc:2368] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 7. Current term is 8.
I20260709 13:08:56.800318 28520 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:56.802498 28418 raft_consensus.cc:697] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:56.775645 28338 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.778888 28642 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.811390 25321 raft_consensus.cc:606] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:56.817806 28634 consensus_queue.cc:1059] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:08:56.819013 28634 raft_consensus.cc:3055] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:08:56.819427 28634 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:56.822746 28338 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:56.824479 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:56.827556 28440 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:56.815567 28611 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:08:56.794601 28554 consensus_queue.cc:237] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:56.829298 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:56.835574 25312 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:08:56.838581 25940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 637) took 1703 ms. Trace:
I20260709 13:08:56.838706 28517 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:56.839346 28517 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.839109 25940 rpcz_store.cc:276] 0709 13:08:55.134108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:55.134255 (+   147us) service_pool.cc:224] Handling call
0709 13:08:56.833044 (+1698789us) spinlock_profiling.cc:243] Waited 280 ms on lock 0x7b5400671e54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:56.836955 (+  3911us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":644476160}
I20260709 13:08:56.840894 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2090) took 1200 ms. Trace:
I20260709 13:08:56.841331 25388 rpcz_store.cc:276] 0709 13:08:55.640623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:55.641217 (+   594us) service_pool.cc:224] Handling call
0709 13:08:56.224402 (+583185us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:56.314635 (+ 90233us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:56.840439 (+525804us) spinlock_profiling.cc:243] Waited 583 ms on lock 0x7b54001f59d0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:56.840681 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1340517760}
I20260709 13:08:56.841641 28535 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:56.842569 28517 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:56.843274 28447 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20260709 13:08:56.844810 28535 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:56.847082 28442 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:56.848507 28222 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:56.852989 28535 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:56.853484 28440 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.853787 28535 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.854486 25908 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:56.855319 28442 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:56.856629 28440 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:56.856729 28535 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:56.857573 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:56.858417 25394 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:08:56.860731 25911 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:56.861372 25937 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 7 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:56.861377 28535 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:56.862267 28535 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.862766 28604 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:08:56.863196 28604 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:08:56.845325 28647 log.cc:927] Time spent T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.085s	user 0.000s	sys 0.002s
I20260709 13:08:56.804725 28616 consensus_queue.cc:260] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.863538 28604 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:08:56.878099 25359 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:56.884240 25312 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:08:56.820678 28634 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:56.886122 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 117629us
I20260709 13:08:56.866076 27962 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:56.890172 28440 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:08:56.892772 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:56.893217 28517 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:56.893774 28517 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.891781 28440 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:56.897703 28440 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:56.898402 28517 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:56.900044 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 533) took 1105 ms. Trace:
I20260709 13:08:56.900458 25391 rpcz_store.cc:276] 0709 13:08:55.794645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:55.797297 (+  2652us) service_pool.cc:224] Handling call
0709 13:08:55.797594 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:56.327766 (+530172us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:56.899582 (+571816us) spinlock_profiling.cc:243] Waited 528 ms on lock 0x7b5400323cd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:56.899868 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1213837184}
I20260709 13:08:56.906991 25321 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:08:56.910645 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:56.911584 25316 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 10: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:56.911458 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 763) took 1120 ms. Trace:
I20260709 13:08:56.912681 25942 rpcz_store.cc:276] 0709 13:08:55.791257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:55.795512 (+  4255us) service_pool.cc:224] Handling call
0709 13:08:55.797726 (+  2214us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:56.149807 (+352081us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:56.910952 (+761145us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b5400381b54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:56.911259 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":766846848}
I20260709 13:08:56.926859 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:56.945159 28634 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:56.945977 25316 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
W20260709 13:08:56.948143 28652 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 13:08:56.945199 25212 leader_election.cc:336] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.191s (SENT)
W20260709 13:08:56.950328 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 95980.9us
I20260709 13:08:56.959100 25312 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:56.923834 28570 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:56.922629 28442 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:08:56.933697 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 67741us
I20260709 13:08:56.922945 28535 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:56.970059 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 44246.2us
I20260709 13:08:56.925859 28418 consensus_queue.cc:237] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:56.977260 28447 consensus_queue.cc:1059] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:08:56.978173 28442 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:56.980055 28653 log.cc:927] Time spent T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 13:08:56.978334 28447 raft_consensus.cc:3055] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:08:56.981077 28447 raft_consensus.cc:740] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:56.982271 28447 consensus_queue.cc:260] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.983551 25908 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:56.983800 28447 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:56.984284 25937 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 10: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:56.984658 28592 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:08:56.978964 28442 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:56.985626 25316 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:08:56.985491 25287 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:56.989029 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:56.989740 25319 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:56.991714 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:56.992578 25317 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:08:56.994637 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:56.995396 25359 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:56.996493 28435 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:56.996660 25359 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:56.957791 28613 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:56.996981 28435 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:57.002699 28563 consensus_queue.cc:260] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:56.994840 28369 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:56.965195 28616 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:08:57.015596 25908 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:57.012392 28435 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:57.029407 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:57.012959 28570 raft_consensus.cc:697] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:57.072239 28570 consensus_queue.cc:237] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:57.047353 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 62278us
W20260709 13:08:57.074200 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 88900us
I20260709 13:08:57.013600 28557 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:57.070003 28369 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.019748 28611 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:57.079228 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:57.083897 25238 raft_consensus.cc:1217] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Deduplicated request from leader. Original: 4.6->[6.7-6.7]   Dedup: 6.7->[]
I20260709 13:08:57.086452 25238 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260709 13:08:57.087394 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:57.087162 28617 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.009s
I20260709 13:08:57.089891 25245 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:08:57.093536 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 77253.2us
I20260709 13:08:57.089944 28564 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:57.099642 28440 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:57.101729 25241 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:08:57.109261 28634 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:08:57.109915 28634 raft_consensus.cc:697] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:57.111380 28369 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:57.109941 25287 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:57.078413 28557 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:57.019520 28592 consensus_queue.cc:1059] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
I20260709 13:08:57.111021 28634 consensus_queue.cc:237] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:57.127969 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 53094.8us
I20260709 13:08:57.080731 28563 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:57.060120 28520 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:57.134141 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:57.118916 28557 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:57.135334 28645 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:57.136984 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:57.145413 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:57.135977 28645 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.152455 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:57.152794 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:57.158867 25242 raft_consensus.cc:2410] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 13:08:57.129894 28642 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:57.160338 25237 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:57.165241 28359 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:57.160527 27962 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:57.166326 27962 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:57.167338 28581 raft_consensus.cc:3055] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:57.167768 28581 raft_consensus.cc:740] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:57.168524 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:57.162357 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 75622.1us
W20260709 13:08:57.175069 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 111785us
I20260709 13:08:57.169518 25240 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:57.172857 25237 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:57.174955 25908 leader_election.cc:400] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10.
I20260709 13:08:57.182281 25908 leader_election.cc:403] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:08:57.131268 28611 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.187676 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:57.189651 25245 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:57.190440 25237 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:32992
I20260709 13:08:57.191143 25237 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:08:57.191184 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:57.191767 25237 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:57.192502 28361 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
W20260709 13:08:57.193732 28655 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 13:08:57.201840 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:57.200292 25237 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:57.203380 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:57.204054 25240 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:57.171298 28359 raft_consensus.cc:697] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:57.205992 28359 consensus_queue.cc:237] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:57.207161 28611 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:57.207715 28435 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:57.208325 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 55680.1us
I20260709 13:08:57.209239 25908 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:57.209431 28440 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:57.209990 25237 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:57.210932 25908 leader_election.cc:400] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 7. Current term is 8.
I20260709 13:08:57.211153 28594 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10.
I20260709 13:08:57.211486 28642 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:57.212004 28594 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:57.212496 28594 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:57.212617 28520 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:57.211526 25237 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:08:57.213284 28222 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:08:57.172940 28564 consensus_queue.cc:1059] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
W20260709 13:08:57.214500 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 420062us
I20260709 13:08:57.215153 28564 raft_consensus.cc:3055] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:08:57.215521 28564 raft_consensus.cc:740] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:08:57.218562 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:57.219650 28338 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:57.220211 28338 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:57.221190 25237 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9. Ops: []
I20260709 13:08:57.221310 28613 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:57.222543 28338 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:57.211287 25908 leader_election.cc:403] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:08:57.212073 28642 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:57.212890 28594 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:57.204928 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:57.236682 25240 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:57.224706 25241 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:08:57.216454 28564 consensus_queue.cc:260] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:57.241786 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:57.244207 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 73777.1us
I20260709 13:08:57.168685 28581 consensus_queue.cc:260] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:57.251610 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:57.251715 25908 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:57.252367 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:57.253060 28447 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 7. Current term is 8.
I20260709 13:08:57.251932 28338 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:57.266359 25238 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 10: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:57.269783 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:57.270592 25389 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:57.273203 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:57.253376 28592 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:08:57.275807 28645 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:57.283228 28642 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:57.283934 28564 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:57.289230 25911 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:57.290536 28663 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:57.265120 28438 consensus_queue.cc:1059] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20260709 13:08:57.291625 28438 raft_consensus.cc:3055] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:57.291980 28438 raft_consensus.cc:740] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:57.290944 28663 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:57.294152 28663 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:57.300776 28581 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:57.289709 28338 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.312012 25240 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
W20260709 13:08:57.314704 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:57.311300 28592 raft_consensus.cc:697] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:57.316821 28442 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:57.333088 28435 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:57.363173 28359 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:57.375837 25317 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:08:57.369524 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:57.371399 28359 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:57.395641 28310 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:08:57.335006 28438 consensus_queue.cc:260] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:57.397927 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:57.402256 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:57.410971 25909 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:08:57.349756 28604 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.422199 28310 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:57.425495 28570 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:57.425573 28588 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:57.397737 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:57.439842 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:57.437670 28570 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:57.437579 28310 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.440001 28634 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:57.440721 25315 raft_consensus.cc:2410] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:08:57.441628 25937 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:08:57.399654 28359 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:57.440721 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:57.445243 25318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
W20260709 13:08:57.452396 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:57.455338 25940 raft_consensus.cc:2410] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 13:08:57.457239 25945 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:57.452356 25318 raft_consensus.cc:606] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:57.459228 25318 raft_consensus.cc:614] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:08:57.463876 28588 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.464576 28338 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:57.466058 28588 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:57.466838 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:57.469290 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:57.469913 25238 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:57.470148 28645 consensus_queue.cc:1059] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:08:57.470451 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:57.470942 25285 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:57.471560 28522 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:57.471921 28522 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:57.472302 28522 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:57.473939 28361 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:57.474491 28361 raft_consensus.cc:697] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:57.474444 28571 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:08:57.475684 28571 consensus_queue.cc:1243] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:08:57.475503 28361 consensus_queue.cc:237] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:57.470846 28470 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:08:57.474619 28613 raft_consensus.cc:993] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567: : Instructing follower 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:08:57.477568 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:57.478099 28613 raft_consensus.cc:1081] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Signalling peer 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:08:57.471125 28645 raft_consensus.cc:3055] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:57.479028 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:57.477603 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:57.440551 28611 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:57.472322 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:57.483256 25945 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:08:57.481746 25316 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:57.479413 28645 raft_consensus.cc:740] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:57.486286 28310 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:57.488499 28570 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:57.460595 28563 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:57.489409 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:57.489809 28563 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.491624 28611 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.451601 28634 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:57.493407 28634 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:57.495404 25941 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:08:57.454824 28592 consensus_queue.cc:237] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.497692 28438 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:57.478299 25318 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
W20260709 13:08:57.463263 28541 consensus_meta.cc:287] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.696s	user 0.007s	sys 0.005s
I20260709 13:08:57.512410 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:57.513361 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 121534us
I20260709 13:08:57.474459 28505 raft_consensus.cc:3055] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:08:57.508072 28645 consensus_queue.cc:260] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:57.509485 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:57.510123 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:57.523397 25316 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:57.462247 28418 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:57.547546 28418 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:57.548035 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:57.549664 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:57.550393 25312 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:57.515581 28435 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:57.554152 28435 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:57.555229 25389 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 3. Current term is 4. Ops: []
I20260709 13:08:57.556238 28435 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:57.518517 28505 raft_consensus.cc:740] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:57.557053 25319 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 13:08:57.536273 25313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2061) took 1566 ms (client timeout 1761 ms). Trace:
I20260709 13:08:57.536775 25239 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:08:57.539415 25321 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:57.540570 27962 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:57.570909 25312 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:08:57.572001 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:57.546640 28557 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:57.513360 28570 consensus_queue.cc:1059] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 13:08:57.558177 28418 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:57.521092 28440 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:57.558748 28505 consensus_queue.cc:260] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.559827 28594 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.560252 28611 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:57.561094 28563 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:57.561959 28664 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:08:57.564306 25320 raft_consensus.cc:2410] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20260709 13:08:57.563938 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 44299.4us
W20260709 13:08:57.564584 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:57.507108 28522 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:57.565483 25313 rpcz_store.cc:269] 0709 13:08:55.969905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:55.972059 (+  2154us) service_pool.cc:224] Handling call
0709 13:08:57.535880 (+1563821us) spinlock_profiling.cc:243] Waited 83.1 ms on lock 0x7b5400352354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:08:57.536080 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":191153024}
I20260709 13:08:57.595933 25312 raft_consensus.cc:2410] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 13:08:57.596681 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:08:57.601431 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 100129us
I20260709 13:08:57.611081 25285 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:57.612203 25285 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:57.606082 25395 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:57.622659 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:57.613714 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:57.624603 25394 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9. Ops: []
W20260709 13:08:57.622084 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 150276us
W20260709 13:08:57.631875 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 60718.1us
I20260709 13:08:57.632440 25320 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260709 13:08:57.633795 28674 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 13:08:57.633982 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:57.638252 25938 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:08:57.628666 28418 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:08:57.612792 28645 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:57.641873 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:57.646265 28594 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:57.646600 28522 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:57.647248 28604 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:57.624428 28613 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:57.648031 28613 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:57.648396 28613 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:57.650290 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:57.636963 27962 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.014s
I20260709 13:08:57.650914 25244 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:57.651345 25240 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:32992
I20260709 13:08:57.651865 25240 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:08:57.652242 25240 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:57.655186 25237 raft_consensus.cc:1217] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[3.3-7.5]   Dedup: 3.3->[5.4-7.5]
W20260709 13:08:57.655452 28666 log.cc:927] Time spent T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.217s	user 0.002s	sys 0.000s
I20260709 13:08:57.655709 25237 pending_rounds.cc:85] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 3
I20260709 13:08:57.655978 25237 pending_rounds.cc:107] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260709 13:08:57.656277 25237 raft_consensus.cc:2889] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:57.617784 28359 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:57.658210 28359 raft_consensus.cc:697] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:08:57.659386 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:57.659170 28359 consensus_queue.cc:237] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:57.661314 25285 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:57.669378 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:57.670575 25209 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:57.671401 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:57.678774 25317 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9. Ops: []
I20260709 13:08:57.626356 28505 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:57.640645 28634 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.688122 25388 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:08:57.688758 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:57.695417 25941 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:57.686760 25240 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:57.688474 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:57.697821 25394 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:57.693948 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:08:57.699028 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 65569us
I20260709 13:08:57.701576 25358 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:57.678422 28522 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:57.677522 28581 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:08:57.623838 28440 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:57.705169 25240 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:57.707257 25317 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 13:08:57.755396 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 53880.4us
I20260709 13:08:57.758015 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:57.761416 25938 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
W20260709 13:08:57.762687 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:57.766098 25358 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:57.767832 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:57.768435 25941 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 7 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:57.706573 28522 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:57.772140 25315 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:08:57.782842 25387 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:08:57.685541 28663 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.763999 28581 raft_consensus.cc:697] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:57.785887 28581 consensus_queue.cc:237] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:57.790366 28440 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:57.790882 28522 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:57.793536 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:57.794157 25390 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:08:57.802726 28634 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:57.804462 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:57.806972 28680 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:57.807152 28464 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:57.807791 28680 raft_consensus.cc:697] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:57.810801 25240 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:08:57.759814 28617 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.053s
I20260709 13:08:57.807829 28464 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.814396 28464 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:57.816906 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:57.771029 28418 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:08:57.817866 28418 raft_consensus.cc:697] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:57.819104 28440 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:57.818871 28418 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.819252 28522 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.819615 28440 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:57.821273 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:57.821909 28440 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:57.822005 25392 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 7: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:57.822132 28522 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:57.845700 28677 log.cc:927] Time spent T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.217s	user 0.001s	sys 0.000s
I20260709 13:08:57.846876 28562 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:57.847327 28562 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:57.847707 28562 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:57.850199 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:57.850703 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:57.851352 25940 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:57.851473 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:57.851691 25937 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: []
I20260709 13:08:57.852133 25943 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:57.852594 25940 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:57.853215 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:57.853247 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:57.854563 25939 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:57.864096 25942 raft_consensus.cc:2393] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 8: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:57.865398 28663 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:57.866015 28675 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.332s	user 0.012s	sys 0.013s
I20260709 13:08:57.868615 25943 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:08:57.868789 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:57.782449 28554 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:57.869453 25940 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 4.
I20260709 13:08:57.869622 28554 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:57.871083 28669 log.cc:927] Time spent T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.438s	user 0.001s	sys 0.000s
I20260709 13:08:57.872212 28554 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:57.874339 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:57.874920 25319 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 10
I20260709 13:08:57.876533 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:57.878185 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:08:57.878571 28668 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.325s	user 0.005s	sys 0.000s
I20260709 13:08:57.878762 25317 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:57.880367 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:57.880981 25388 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:57.881737 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:57.881928 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:57.882535 25393 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:57.882604 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:57.884368 25391 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:08:57.706954 28222 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.076s
I20260709 13:08:57.808974 28680 consensus_queue.cc:237] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.885103 25395 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 13:08:57.886626 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:57.899835 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:57.900818 25941 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 8: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:08:57.913239 25939 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:08:57.913457 25394 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:57.882560 25388 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:08:57.916280 25937 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:57.937248 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:57.937891 25395 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:57.938796 25938 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:57.942709 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:57.945659 25386 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:08:57.885391 28557 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:57.958961 28613 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:57.962455 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 196429us
I20260709 13:08:57.967550 25393 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:08:57.952886 25319 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:08:57.976861 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:57.977850 25941 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:57.978906 25317 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:08:57.979954 25316 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
W20260709 13:08:57.982200 28679 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 13:08:57.984517 28222 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:57.985028 28222 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:57.986995 28222 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:57.987335 25390 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:08:57.988586 25911 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:57.989367 28664 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:08:57.989857 28664 raft_consensus.cc:697] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:57.990769 28664 consensus_queue.cc:237] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:58.016240 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:58.017274 25390 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 10: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:58.014739 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:08:58.019827 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:58.020187 25936 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:08:58.015287 28438 consensus_meta.cc:287] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.517s	user 0.009s	sys 0.000s
W20260709 13:08:58.036373 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:58.055656 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:58.055797 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:58.021090 28234 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:58.067185 25386 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:58.019553 28613 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:58.076776 25394 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 13:08:58.070981 25945 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:58.078747 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:58.083089 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:58.083765 25395 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:08:58.084787 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:58.085076 25393 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: []
I20260709 13:08:58.089435 25938 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 6: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:08:58.089952 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:58.096325 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:58.097476 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:58.071044 28663 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:58.101935 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:58.106810 25391 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 13:08:58.104584 25390 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: []
I20260709 13:08:58.105589 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:58.105639 25241 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9. Ops: []
I20260709 13:08:58.103101 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:58.108551 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:58.108974 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:58.103623 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:58.110574 25243 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
W20260709 13:08:58.112645 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 351806us
I20260709 13:08:58.089866 28438 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:58.076536 28234 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:58.083817 28447 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260709 13:08:58.151427 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:58.165097 28588 raft_consensus.cc:3055] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:08:58.165573 28588 raft_consensus.cc:740] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:58.166368 28588 consensus_queue.cc:260] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:58.167564 28588 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:58.170428 25391 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: [2.2-6.4]
W20260709 13:08:58.172557 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:08:58.173058 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:58.111768 28418 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:58.179728 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:58.178833 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:58.180915 25908 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:58.182649 28682 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:58.183177 28682 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:58.185361 28442 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:08:58.186502 25936 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:08:58.178723 28438 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:58.185627 28681 log.cc:927] Time spent T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 13:08:58.188846 28438 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:58.194618 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:58.195649 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:58.251555 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 70687.7us
I20260709 13:08:58.325223 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:58.345579 28685 log.cc:927] Time spent T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
I20260709 13:08:58.259891 28682 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:58.420068 25287 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:58.422619 28682 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:58.423110 25287 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:58.423986 28682 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:08:58.399268 25244 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:58.116614 28663 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:58.411947 28361 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 13:08:58.413118 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:58.414000 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:58.418862 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:58.417950 28683 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.228s	user 0.005s	sys 0.006s
I20260709 13:08:58.419407 28338 consensus_queue.cc:1059] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:08:58.403283 28464 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:58.419454 28571 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:08:58.405696 28447 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:58.417598 28438 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:58.424091 25287 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:58.424619 28570 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:58.213586 28418 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:58.428056 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:58.429750 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:58.496062 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:58.496171 28338 raft_consensus.cc:3055] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:08:58.429849 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:58.496789 28338 raft_consensus.cc:740] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:58.497196 25239 raft_consensus.cc:2393] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:58.497782 28338 consensus_queue.cc:260] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:58.499121 28338 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:58.429801 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:58.430723 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:08:58.437656 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:58.437500 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:58.503333 25244 raft_consensus.cc:2410] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20260709 13:08:58.439656 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:58.496657 25393 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:58.414454 28359 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:58.483939 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:58.496243 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:08:58.476286 28562 consensus_meta.cc:287] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.628s	user 0.015s	sys 0.005s
I20260709 13:08:58.514353 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:58.478820 25359 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:58.510190 25392 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:58.515035 25391 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:08:58.508697 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:58.516206 25246 raft_consensus.cc:2393] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:58.485778 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:58.517879 25391 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
I20260709 13:08:58.515167 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:58.519183 25391 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:08:58.487283 28570 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:58.525369 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:58.526084 25395 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260709 13:08:58.529982 25386 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:58.529940 25395 pending_rounds.cc:85] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 5
I20260709 13:08:58.535689 25395 pending_rounds.cc:107] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 13:08:58.491827 28447 raft_consensus.cc:697] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:08:58.536003 25395 raft_consensus.cc:2889] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:58.547596 25392 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:52318
I20260709 13:08:58.548636 25392 raft_consensus.cc:606] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
W20260709 13:08:58.540933 28563 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.901s	user 0.026s	sys 0.048s
I20260709 13:08:58.522719 28562 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:58.551013 28563 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.911s	user 0.026s	sys 0.049s
I20260709 13:08:58.550989 28440 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:58.552117 28563 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:58.552615 28562 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:58.553025 28588 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:58.552680 28563 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:58.556704 28563 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:58.558465 28418 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:58.559173 28557 consensus_queue.cc:1059] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.007s
I20260709 13:08:58.560194 28557 consensus_queue.cc:1243] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:08:58.560804 28557 raft_consensus.cc:3055] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:08:58.561240 28557 raft_consensus.cc:740] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:58.537734 28570 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:58.492166 28438 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:58.489022 28464 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:58.571620 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:58.546283 28613 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:08:58.581876 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:58.594684 25244 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:08:58.536983 28359 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:58.600263 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:58.596560 25387 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
W20260709 13:08:58.605937 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 188946us
W20260709 13:08:58.609613 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 201018us
I20260709 13:08:58.562414 28557 consensus_queue.cc:260] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:58.607579 28505 consensus_meta.cc:287] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.906s	user 0.006s	sys 0.026s
I20260709 13:08:58.603399 28222 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:58.615885 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:58.665256 25244 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 4. Current term is 5. Ops: []
W20260709 13:08:58.690869 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:58.676290 25242 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260709 13:08:58.744174 25242 pending_rounds.cc:85] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 3
I20260709 13:08:58.728066 28359 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:08:58.672051 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 150260us
I20260709 13:08:58.750826 25242 pending_rounds.cc:107] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260709 13:08:58.751415 25242 raft_consensus.cc:2889] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:58.587100 25393 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
W20260709 13:08:58.733937 28695 log.cc:927] Time spent T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
W20260709 13:08:58.724704 28688 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20260709 13:08:58.722285 28692 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.217s	user 0.001s	sys 0.000s
W20260709 13:08:58.834295 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 410235us
W20260709 13:08:58.725469 28693 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260709 13:08:58.734727 28557 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:58.844834 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:58.726325 28464 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:58.671474 28663 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:58.607028 28613 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:58.748065 28554 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:58.849301 25287 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:58.851740 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:58.842121 28698 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:58.859766 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:58.863631 25242 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:58.616847 28447 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:58.769405 28222 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:58.882570 25393 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
mode: GRACEFUL
new_leader_uuid: "f646dfd532974dceadfe00005d2bbf59"
 from {username='slave'} at 127.0.0.1:52318
I20260709 13:08:58.883285 25393 raft_consensus.cc:606] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Received request to transfer leadership to TS f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:58.883644 25393 raft_consensus.cc:614] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:08:58.886400 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 289155us
I20260709 13:08:58.851049 28562 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.024s
I20260709 13:08:58.870045 28438 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:58.909978 25358 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:58.914860 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2016) took 1397 ms. Trace:
I20260709 13:08:58.890388 28563 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:58.872243 28359 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260709 13:08:58.867602 28698 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:58.874671 28554 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:58.924220 25386 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:08:58.911731 27962 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:58.890729 28702 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:08:58.916196 25313 rpcz_store.cc:276] 0709 13:08:57.514419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:57.669057 (+154638us) service_pool.cc:224] Handling call
0709 13:08:57.692784 (+ 23727us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:57.918534 (+225750us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:57.918561 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:57.938156 (+ 19595us) log.cc:844] Serialized 26 byte log entry
0709 13:08:58.079987 (+141831us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:58.080023 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:58.080091 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:58.911492 (+831401us) raft_consensus.cc:1685] finished
0709 13:08:58.911511 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:58.911641 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23153,"raft.queue_time_us":62904,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":2929,"thread_start_us":336350,"threads_started":2,"wal-append.queue_time_us":632142}
I20260709 13:08:58.932566 28359 raft_consensus.cc:3055] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:08:58.913306 28464 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:58.940621 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:08:58.920531 28438 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:58.950469 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:58.961025 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:58.962438 25395 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:58.962047 25386 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:08:58.965114 25318 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:08:58.968165 25358 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:58.970191 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:58.970966 25386 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
W20260709 13:08:58.972994 28709 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 13:08:58.975109 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:58.951551 28418 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:58.938759 28563 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:58.977545 28418 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:58.978238 25395 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:08:58.979723 25390 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:08:58.979849 28418 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:58.962556 28438 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:58.977893 28704 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:08:58.926059 28588 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.030s
I20260709 13:08:58.986441 25317 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:08:58.941197 28359 raft_consensus.cc:740] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:58.940975 28702 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:58.945264 28464 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:58.991875 28438 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:58.941664 27962 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:58.995033 27962 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:58.995224 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:58.996987 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:58.962445 28711 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:08:59.000172 28464 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:59.022428 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:59.023517 28222 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:59.023855 28613 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:58.982744 28680 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.038s
I20260709 13:08:58.998673 28702 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:59.027275 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:59.028280 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:59.027886 25314 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:08:59.025617 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:58.987106 28570 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.029228 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:59.031771 25321 raft_consensus.cc:2393] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 5: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:08:59.034260 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 182572us
I20260709 13:08:59.035482 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:59.038393 25209 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:59.044700 25242 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:08:59.052670 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:59.055415 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:59.051218 25287 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:59.030566 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:59.064318 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:59.064311 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:59.064690 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:59.066524 25209 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:59.054113 28663 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:59.067391 28437 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.007s
I20260709 13:08:59.067350 28663 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.067348 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:59.069495 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:59.065917 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:59.072999 25388 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:08:59.054502 28604 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:59.074644 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 44155.2us
I20260709 13:08:59.078804 25242 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:08:59.060460 28505 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.019s
I20260709 13:08:59.130860 25318 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:59.091288 28570 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:59.074218 28222 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:59.038558 28359 consensus_queue.cc:260] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.125432 25241 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:08:59.086182 28554 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:59.068189 28338 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:08:59.089586 28698 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:59.094460 28701 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:08:59.080173 28715 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:08:59.091666 28663 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:59.060395 28704 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:08:59.152714 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 75923.4us
I20260709 13:08:59.081561 25243 raft_consensus.cc:3055] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:08:59.128109 28705 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:08:59.076503 28711 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:59.075420 28604 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.160482 28604 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:59.101436 28712 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:08:59.137866 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 169777us
W20260709 13:08:59.147652 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 81177us
I20260709 13:08:59.079404 28438 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.002s
I20260709 13:08:59.153939 25243 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:59.156114 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:59.157203 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:59.156626 28704 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.157452 25395 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:08:59.145794 28338 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:59.060597 28613 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260709 13:08:59.163055 25241 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:59.165369 25390 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:59.166038 25209 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:59.162997 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:59.164891 28712 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:59.177417 28438 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:59.177724 28712 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:08:59.178050 28438 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:59.178552 28438 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:08:59.215245 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 52321us
I20260709 13:08:59.215318 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:59.216348 25318 raft_consensus.cc:2410] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260709 13:08:59.176170 25209 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:08:59.217898 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:59.154274 28554 consensus_queue.cc:1059] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20260709 13:08:59.219753 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:59.220631 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:59.173059 28338 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:08:59.165482 28705 raft_consensus.cc:697] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:59.171542 28359 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:59.147720 28222 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.225409 28570 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:59.223855 25358 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:59.223335 25313 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:59.218186 28711 consensus_queue.cc:1059] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20260709 13:08:59.220113 28715 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:59.252730 25390 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:08:59.223351 28680 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:59.274506 25242 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260709 13:08:59.274955 25242 pending_rounds.cc:85] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 3
I20260709 13:08:59.275194 25242 pending_rounds.cc:107] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260709 13:08:59.275429 25242 raft_consensus.cc:2889] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:59.275358 25908 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:59.223121 28718 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:08:59.232012 28716 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:59.230860 28721 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:59.241791 28570 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:59.278532 28505 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:59.241735 28698 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260709 13:08:59.285988 28705 consensus_queue.cc:237] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.291780 28234 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:59.248803 28562 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:59.294879 28721 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:59.302382 28721 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:59.301570 28702 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.306161 28712 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.308725 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:08:59.279270 28722 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:59.320149 25241 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260709 13:08:59.321095 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 97315.7us
W20260709 13:08:59.324308 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 106378us
I20260709 13:08:59.296502 28570 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:59.334345 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:59.295424 28505 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:59.322238 25358 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:59.277791 28680 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.318311 28702 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:59.308283 28712 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:59.275260 28715 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.359370 25241 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: []
I20260709 13:08:59.371156 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:59.347034 28505 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:08:59.363045 28723 log.cc:927] Time spent T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20260709 13:08:59.367677 28604 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:08:59.318061 28562 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.376421 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:59.296236 28716 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.381702 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:59.382858 25240 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:08:59.323532 28464 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.377306 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32994 (request call id 1877) took 1200 ms. Trace:
I20260709 13:08:59.390619 25245 rpcz_store.cc:276] 0709 13:08:58.177276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:58.180576 (+  3300us) service_pool.cc:224] Handling call
0709 13:08:59.376255 (+1195679us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:59.377242 (+   987us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30116224}
I20260709 13:08:59.392427 28222 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:59.373071 28338 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.394729 28338 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:59.395975 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:59.396570 25245 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:08:59.394753 28562 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:59.401742 28704 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:08:59.393201 28686 log.cc:927] Time spent T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.963s	user 0.006s	sys 0.008s
I20260709 13:08:59.404064 25237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1718) took 1777 ms. Trace:
I20260709 13:08:59.404475 25237 rpcz_store.cc:276] 0709 13:08:57.626199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:57.626331 (+   132us) service_pool.cc:224] Handling call
0709 13:08:57.654957 (+ 28626us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:08:57.656870 (+  1913us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:08:57.849086 (+192216us) log.cc:844] Serialized 14 byte log entry
0709 13:08:58.092474 (+243388us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:08:58.103253 (+ 10779us) log.cc:844] Serialized 52 byte log entry
0709 13:08:58.103397 (+   144us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:08:58.103428 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:58.103486 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:59.362932 (+1259446us) spinlock_profiling.cc:243] Waited 39 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:59.403760 (+ 40828us) raft_consensus.cc:1685] finished
0709 13:08:59.403859 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:59.404009 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":221706,"prepare.queue_time_us":192857,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":27,"raft.queue_time_us":28445,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":13354,"spinlock_wait_cycles":89734912,"thread_start_us":226356,"threads_started":1,"wal-append.queue_time_us":329845}
I20260709 13:08:59.406427 25143 catalog_manager.cc:5697] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 3 to 6, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 6 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:08:59.407985 28464 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:59.376998 28730 consensus_queue.cc:1059] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260709 13:08:59.409794 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:08:59.410506 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:59.411474 28712 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:59.411775 28716 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:59.412074 28712 raft_consensus.cc:697] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:59.412469 28680 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:59.413064 28712 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.414470 28715 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:59.345342 28722 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:59.403610 25139 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39582 (request call id 307) took 15298 ms (client timeout 15000 ms). Trace:
I20260709 13:08:59.409505 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:59.411612 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:59.417218 25909 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:59.419423 28234 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:59.419896 28721 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.420480 28735 consensus_queue.cc:1059] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:08:59.486753 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:08:59.488829 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 166657us
I20260709 13:08:59.491102 25358 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:59.491302 25245 raft_consensus.cc:2393] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 9: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:59.486516 25143 catalog_manager.cc:5697] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 5 to 7, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 7 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:08:59.493103 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:59.495201 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 78117us
I20260709 13:08:59.493101 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:59.440145 25240 raft_consensus.cc:2410] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:08:59.497618 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32994 (request call id 1879) took 1214 ms. Trace:
I20260709 13:08:59.497951 25240 rpcz_store.cc:276] 0709 13:08:58.283388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:58.403832 (+120444us) service_pool.cc:224] Handling call
0709 13:08:59.381541 (+977709us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:08:59.497559 (+116018us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":53943552}
I20260709 13:08:59.486984 25243 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.415057 28715 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:59.506105 25395 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:08:59.506778 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:59.506927 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:59.509909 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:59.511516 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:59.512830 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:08:59.390753 28554 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:59.490088 28664 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:59.422739 28722 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:59.486351 27962 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:08:59.525368 25358 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:59.433393 28734 raft_consensus.cc:3055] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:08:59.516559 25390 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:08:59.480062 28594 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:59.517055 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:59.519699 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:59.516335 28727 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:08:59.423293 28730 consensus_queue.cc:1243] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer f646dfd532974dceadfe00005d2bbf59 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:08:59.496308 28447 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:59.520897 25243 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:59.503804 28563 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:59.523650 28721 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:08:59.578418 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 48390.4us
I20260709 13:08:59.504119 28234 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.381304 28604 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.530457 27962 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:59.597344 27962 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:08:59.583094 25394 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:08:59.493811 28562 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:59.601800 28562 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.602285 28682 raft_consensus.cc:3055] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:08:59.602691 28682 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:59.603423 25243 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260709 13:08:59.603948 28562 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:59.583133 25237 raft_consensus.cc:2393] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 9: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:59.530213 28734 raft_consensus.cc:740] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:59.584607 25143 catalog_manager.cc:5697] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:08:59.590498 25358 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:59.577158 28722 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:59.593837 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:08:59.579418 28557 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:59.520645 28554 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.604020 25243 pending_rounds.cc:85] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 7
I20260709 13:08:59.661209 25243 pending_rounds.cc:107] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260709 13:08:59.661552 25243 raft_consensus.cc:2889] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:08:59.516654 28715 raft_consensus.cc:697] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:59.582374 28418 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:59.658960 28722 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:08:59.421890 25139 rpcz_store.cc:269] 0709 13:08:44.105474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:44.105761 (+   287us) service_pool.cc:224] Handling call
0709 13:08:59.403356 (+15297595us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:08:55.796073 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:08:55.837473 (+ 41400us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:08:55.837509 (+    36us) write_op.cc:435] Acquired schema lock
0709 13:08:55.837526 (+    17us) tablet.cc:662] Decoding operations
0709 13:08:55.856419 (+ 18893us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:08:55.860104 (+  3685us) write_op.cc:641] Partition lock acquired for write op
0709 13:08:55.863186 (+  3082us) tablet.cc:685] Acquiring locks for 11 operations
0709 13:08:55.871271 (+  8085us) tablet.cc:701] Row locks acquired
0709 13:08:55.871289 (+    18us) write_op.cc:260] PREPARE: finished
0709 13:08:55.871440 (+   151us) write_op.cc:270] Start()
0709 13:08:55.871564 (+   124us) write_op.cc:276] Timestamp: P: 1783602535871393 usec, L: 0
0709 13:08:55.871585 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 13:08:55.872367 (+   782us) log.cc:844] Serialized 4397 byte log entry
0709 13:08:56.368700 (+496333us) op_driver.cc:464] REPLICATION: finished
0709 13:08:56.400704 (+ 32004us) write_op.cc:301] APPLY: starting
0709 13:08:56.400823 (+   119us) tablet.cc:1367] starting BulkCheckPresence
0709 13:08:56.401197 (+   374us) tablet.cc:1370] finished BulkCheckPresence
0709 13:08:56.401346 (+   149us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:08:56.402792 (+  1446us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:08:56.403169 (+   377us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0709 13:08:56.403241 (+    72us) write_op.cc:312] APPLY: finished
0709 13:08:56.403708 (+   467us) log.cc:844] Serialized 83 byte log entry
0709 13:08:56.403916 (+   208us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:56.404129 (+   213us) write_op.cc:454] Released schema lock
0709 13:08:56.404200 (+    71us) write_op.cc:341] FINISH: Updating metrics
0709 13:08:56.621538 (+217338us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:08:56.629504 (+  7966us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1774526,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":31672,"apply.run_cpu_time_us":3981,"apply.run_wall_time_us":3985,"num_ops":11,"prepare.queue_time_us":293741,"prepare.run_cpu_time_us":14895,"prepare.run_wall_time_us":200075,"raft.queue_time_us":310177,"raft.run_cpu_time_us":8998,"raft.run_wall_time_us":280300,"replication_time_us":496917,"thread_start_us":679894,"threads_started":4,"wal-append.queue_time_us":162735,"wal-append.run_cpu_time_us":16197,"wal-append.run_wall_time_us":1581096}]]}
I20260709 13:08:59.667409 28716 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:59.668056 28736 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:59.668062 28716 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.670651 28554 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:59.670876 28716 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:59.527498 28664 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.605906 28730 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:59.673758 25315 raft_consensus.cc:2410] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:08:59.673578 28730 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.673976 28664 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:59.605958 28704 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:08:59.675815 28730 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:59.676151 28438 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.676581 28734 consensus_queue.cc:260] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:59.677820 28739 log.cc:927] Time spent T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 13:08:59.677739 28418 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.679057 28234 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:59.680104 28418 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:59.660640 28717 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:08:59.681229 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:59.671020 28604 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:59.668582 28736 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.683096 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:08:59.662523 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 72115.2us
I20260709 13:08:59.680867 28717 raft_consensus.cc:697] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:08:59.686535 28717 consensus_queue.cc:237] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.686733 25143 catalog_manager.cc:5697] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 4 to 7, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 7 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:08:59.689611 25284 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:59.690871 25284 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:59.683717 25237 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:08:59.684983 25209 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:08:59.589483 28594 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:59.681026 28711 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260709 13:08:59.700196 25284 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:59.700803 25241 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 9: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:08:59.705153 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:59.706039 25237 raft_consensus.cc:2410] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260709 13:08:59.689571 25358 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:59.666129 28735 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:08:59.709704 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 213615us
I20260709 13:08:59.709666 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:59.702353 28562 raft_consensus.cc:2764] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 13:08:59.702262 28730 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:59.693367 28234 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:59.699241 28359 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:08:59.736433 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:59.737051 25244 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:08:59.740384 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:59.707794 25284 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:59.707332 28338 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:08:59.718984 27962 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.721275 28604 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:59.722525 28234 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.724303 28440 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:59.712354 28740 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:08:59.726102 28594 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:08:59.714999 28718 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:08:59.716068 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:59.728150 28722 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:59.729447 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:59.778077 25317 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:08:59.730576 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:59.692252 28682 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.781656 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:59.715329 28588 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:59.733472 28701 consensus_queue.cc:1059] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:08:59.786116 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:59.703630 28698 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260709 13:08:59.793255 28588 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.795360 28701 raft_consensus.cc:3055] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:08:59.795517 28588 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:59.795804 28701 raft_consensus.cc:740] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:08:59.796064 28234 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:59.777443 28440 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.753806 28730 raft_consensus.cc:697] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:59.797917 28730 consensus_queue.cc:237] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.770438 28361 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260709 13:08:59.743124 28604 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.668637 28570 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.743968 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1729) took 1125 ms. Trace:
I20260709 13:08:59.744766 25938 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:08:59.745889 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:59.812280 25936 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:59.692369 28712 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260709 13:08:59.759730 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:59.724542 28736 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:59.768230 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:59.770514 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:59.818921 25314 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:08:59.776312 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:08:59.777019 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:59.820712 25939 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:08:59.788658 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:59.821905 25317 raft_consensus.cc:2393] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 10: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:08:59.743888 28338 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:59.743556 28663 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:59.802930 25312 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:08:59.804435 28604 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:59.829296 28338 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:59.713948 28735 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.805851 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:59.806365 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:59.848227 25321 raft_consensus.cc:2410] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:08:59.806527 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:59.807976 25908 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:59.809774 28740 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:59.767336 28562 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
I20260709 13:08:59.814375 25316 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: []
I20260709 13:08:59.796890 28733 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
W20260709 13:08:59.815882 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 56245.6us
W20260709 13:08:59.823933 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 113060us
I20260709 13:08:59.793278 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:08:59.859959 25944 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:08:59.860311 25909 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:08:59.858420 28740 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:59.863379 28706 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.950s	user 0.005s	sys 0.009s
I20260709 13:08:59.864851 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2115) took 1696 ms. Trace:
W20260709 13:08:59.866966 28738 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.281s	user 0.001s	sys 0.000s
I20260709 13:08:59.865324 25391 rpcz_store.cc:276] 0709 13:08:58.168579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:58.521040 (+352461us) service_pool.cc:224] Handling call
0709 13:08:58.521432 (+   392us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:08:58.555457 (+ 34025us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:58.555501 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:08:58.556183 (+   682us) log.cc:844] Serialized 52 byte log entry
0709 13:08:58.679892 (+123709us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:08:58.869304 (+189412us) log.cc:844] Serialized 14 byte log entry
0709 13:08:58.870000 (+   696us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:58.870071 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:08:59.862818 (+992747us) spinlock_profiling.cc:243] Waited 2.45 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:08:59.864453 (+  1635us) raft_consensus.cc:1685] finished
0709 13:08:59.864584 (+   131us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:59.864776 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1784,"prepare.queue_time_us":232676,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":31,"raft.queue_time_us":27296,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":17295,"spinlock_wait_cycles":5633280,"thread_start_us":251040,"threads_started":2,"wal-append.queue_time_us":306167}
I20260709 13:08:59.803159 28722 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:08:59.787469 28594 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:08:59.850051 25908 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:59.850805 25939 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:08:59.810773 25239 rpcz_store.cc:276] 0709 13:08:58.618995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:58.744500 (+125505us) service_pool.cc:224] Handling call
0709 13:08:58.744849 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:08:58.952854 (+208005us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:58.952900 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:08:58.952960 (+    60us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:08:58.953006 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:08:59.743752 (+790746us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b54003ec454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:08:59.743797 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:08:59.743915 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":477805952}
I20260709 13:08:59.816637 28743 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:08:59.856704 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:08:59.856845 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:08:59.894724 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 44737.4us
I20260709 13:08:59.796682 28701 consensus_queue.cc:260] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.895777 25944 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:08:59.781209 28222 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 13:08:59.857242 28604 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:08:59.805042 28505 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:08:59.898839 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 163662us
I20260709 13:08:59.900009 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:08:59.903198 25241 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:08:59.835443 28663 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.782441 28715 consensus_queue.cc:237] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.895830 28722 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:59.901412 28437 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:08:59.915993 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 55721.4us
I20260709 13:08:59.917342 25285 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:08:59.894528 28743 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:08:59.904568 28664 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:08:59.919121 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 148011us
I20260709 13:08:59.899524 28604 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:08:59.923151 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 215447us
W20260709 13:08:59.909385 28705 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.007s	sys 0.005s
I20260709 13:08:59.897505 28557 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.022s
I20260709 13:08:59.928843 25212 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:59.926501 25911 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:08:59.930434 25359 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:08:59.931260 25284 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:08:59.932165 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:59.934981 25320 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:08:59.936017 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:59.936558 25320 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:08:59.932039 28437 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:08:59.938844 28438 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:59.939577 28438 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:08:59.939069 28437 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.940059 28438 raft_consensus.cc:697] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:08:59.941502 28570 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:59.943257 28338 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.921983 28701 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:08:59.901178 28594 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.918447 28722 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:59.925716 28664 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:08:59.925395 28743 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:08:59.932830 28705 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.008s	sys 0.005s
I20260709 13:08:59.963212 28338 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:08:59.963907 28761 consensus_queue.cc:1059] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:08:59.978386 28437 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:59.955544 28754 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:08:59.966822 28557 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:08:59.979096 28594 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:08:59.932487 28604 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:08:59.978257 28761 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:08:59.979802 25314 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:08:59.958127 28664 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:59.979038 28754 raft_consensus.cc:697] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:08:59.995096 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 57425.9us
I20260709 13:08:59.997182 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:08:59.982412 28717 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:09:00.001152 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.001809 25395 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:08:59.940968 28438 consensus_queue.cc:237] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:08:59.986055 27962 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:00.005059 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:00.006345 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.006587 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:00.006337 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 54780us
I20260709 13:09:00.006772 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:08:59.979442 28557 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:08:59.995079 28740 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:00.009716 25392 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:09:00.013361 25389 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:09:00.008097 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:00.014324 25387 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:08:59.988015 28594 consensus_queue.cc:1059] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.020s
I20260709 13:08:59.994032 28716 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:00.003732 28717 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:00.004812 28755 raft_consensus.cc:3055] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:00.023718 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:00.024492 25316 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:00.022032 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:00.023114 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.025425 25314 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:09:00.027745 28735 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:00.029320 28767 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:00.029891 28767 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:00.020560 28762 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:00.031113 28557 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:00.031466 28762 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:00.032143 28767 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:00.032757 28767 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:09:00.033849 28762 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:00.048831 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:00.037714 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:00.044275 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:00.118176 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 200910us
W20260709 13:09:00.207571 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 276407us
W20260709 13:09:00.139288 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:00.164507 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 215921us
I20260709 13:09:00.183044 25387 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:00.213604 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:00.195711 25314 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:00.214360 25312 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:00.145469 28338 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:09:00.215144 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.049187 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.170516 28705 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:00.216679 25316 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:09:00.217247 25386 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:09:00.216782 28705 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:00.242061 25945 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:09:00.242660 25285 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:00.221872 25212 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:00.244212 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:00.244292 25212 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:00.223373 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:00.244915 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:00.244943 25945 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 4.
I20260709 13:09:00.245333 25944 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:09:00.245697 25212 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:00.238675 25312 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:00.221947 25359 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:00.249591 28764 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:09:00.250281 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 256501us
I20260709 13:09:00.266301 25285 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:00.269520 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:00.270648 25314 raft_consensus.cc:2368] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5.
I20260709 13:09:00.271106 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:00.271720 25908 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:00.272151 25312 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:00.023564 28717 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:00.276124 25285 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:00.277000 25285 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:00.277333 25314 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:00.273945 25212 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:00.023823 28755 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:00.067916 28766 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:00.221371 28682 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:00.281319 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:00.264844 28712 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:09:00.288534 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:00.264777 28730 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:00.118477 28731 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:00.286981 25911 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:00.075119 28505 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:00.290439 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:00.282616 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.278026 28222 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:00.021202 28716 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:00.291659 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 581) took 1681 ms. Trace:
I20260709 13:09:00.311115 25238 rpcz_store.cc:276] 0709 13:08:58.610391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:58.610999 (+   608us) service_pool.cc:224] Handling call
0709 13:08:58.736680 (+125681us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:08:58.886812 (+150132us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:08:58.886845 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:08:58.890931 (+  4086us) log.cc:844] Serialized 26 byte log entry
0709 13:08:59.894396 (+1003465us) spinlock_profiling.cc:243] Waited 2.34 ms on lock 0x7b18000c08e0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f29768520d1 00007f2976c6da0f
0709 13:09:00.006710 (+112314us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:00.290819 (+284109us) log.cc:844] Serialized 14 byte log entry
0709 13:09:00.291034 (+   215us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:00.291158 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:00.291197 (+    39us) raft_consensus.cc:1685] finished
0709 13:09:00.291264 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:00.291597 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":141336,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":2587,"spinlock_wait_cycles":5376768,"thread_start_us":497932,"threads_started":3,"wal-append.queue_time_us":1142409}
I20260709 13:09:00.042342 28663 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:00.116945 28604 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:00.298275 25389 raft_consensus.cc:2393] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:00.291436 28564 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:00.286383 28613 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:09:00.242990 28762 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
W20260709 13:09:00.352511 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 78657.1us
W20260709 13:09:00.389392 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 102481us
I20260709 13:09:00.278611 28752 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:00.365993 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:00.367791 25312 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:00.295370 28594 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:00.295650 28733 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:00.251912 28722 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:00.292197 28712 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:00.376664 28564 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:00.281473 28771 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:00.294725 28562 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:00.419852 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:00.428884 25389 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:00.407080 28663 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:00.436437 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:09:00.448099 28722 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:00.430614 28564 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:00.449337 25319 raft_consensus.cc:2368] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9.
I20260709 13:09:00.450229 28562 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:00.436355 28751 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:00.471727 25315 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: [6.4-9.6]
W20260709 13:09:00.465319 28763 log.cc:927] Time spent T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 13:09:00.428121 28712 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:00.428367 28594 raft_consensus.cc:697] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:00.427451 28771 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:00.469921 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:00.488663 25312 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:09:00.042692 28754 consensus_queue.cc:237] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:00.463966 28730 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:09:00.486436 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:00.486788 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:00.281402 28764 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:00.497618 28604 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:00.498674 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.417225 28770 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.131s
I20260709 13:09:00.499823 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.500396 25390 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:00.471287 28705 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:00.417389 28733 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:00.501418 28733 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:00.292852 28766 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:00.502575 28594 consensus_queue.cc:237] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:00.471632 28751 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:00.476284 28562 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:00.504822 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:00.508795 25390 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 3 FOLLOWER]: Advancing to term 5
W20260709 13:09:00.504776 28701 consensus_meta.cc:287] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.539s	user 0.007s	sys 0.003s
I20260709 13:09:00.517977 25389 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:00.519071 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.519738 25389 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:00.492885 28771 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:00.522854 25313 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260709 13:09:00.511513 28743 consensus_meta.cc:287] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.528s	user 0.009s	sys 0.000s
I20260709 13:09:00.509174 28751 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:00.494561 28664 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:00.581949 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:00.552197 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:00.553963 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:00.572536 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:00.580237 25358 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:00.554385 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:00.531035 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.584542 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:00.584640 25389 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:00.585556 25244 raft_consensus.cc:2410] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:09:00.587456 25314 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 5.
I20260709 13:09:00.587553 25390 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:09:00.534715 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:00.592535 25358 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:00.593962 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:00.594676 25315 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
W20260709 13:09:00.590193 25909 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.011s	sys 0.059s
I20260709 13:09:00.417755 28755 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:00.596407 25909 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.011s	sys 0.059s
I20260709 13:09:00.595438 28727 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:09:00.597930 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 584401us
I20260709 13:09:00.599028 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:00.599350 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:00.604276 25909 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:00.571426 28728 raft_consensus.cc:3055] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:00.608172 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:00.608783 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:00.609012 25389 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:00.594702 28764 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:00.571126 28564 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:00.602234 28768 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:00.544581 28743 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:00.633116 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:00.633812 25386 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:00.607432 28728 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:00.615856 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:00.596017 28418 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260709 13:09:00.621443 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.611725 28777 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.010s
I20260709 13:09:00.602298 28440 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:00.629329 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:00.640295 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.657611 25315 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:00.656247 25386 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:00.615751 28505 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:09:00.675938 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.676959 25387 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 5: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:00.658516 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:00.680827 25937 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 7 LEARNER]: Advancing to term 9
I20260709 13:09:00.681780 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:00.684247 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.633872 28716 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:00.666277 28564 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:00.669679 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:00.670673 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:00.671840 28779 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 13:09:00.672344 28762 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:09:00.673511 25939 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
I20260709 13:09:00.674258 28664 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:00.646766 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:00.678705 28604 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:00.682595 25321 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
W20260709 13:09:00.699848 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 53122.5us
I20260709 13:09:00.701084 25209 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:00.658922 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:00.692830 25312 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:09:00.695873 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:00.657984 28766 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:00.703897 25387 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:00.705794 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:00.706845 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:00.706939 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.708734 25312 raft_consensus.cc:2376] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8. Re-sending same reply.
I20260709 13:09:00.708812 25389 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:09:00.710248 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:00.711231 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:00.711467 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:00.712646 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:00.713303 28721 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:00.713976 28721 raft_consensus.cc:697] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:00.715232 28721 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:00.719455 25320 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 13:09:00.721781 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:00.722589 25313 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:09:00.722681 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:00.722905 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.723251 25312 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:09:00.712888 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:00.723798 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:00.745795 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:00.746891 25389 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 9: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:00.747843 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.748445 25389 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:00.723729 25386 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:09:00.750204 28717 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:00.750988 25386 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 13:09:00.752609 28717 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:00.753170 28418 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:00.753779 28418 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:00.733317 25321 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 9: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:00.749074 25243 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:00.759331 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:00.761754 25321 raft_consensus.cc:1275] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:09:00.765991 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.767035 25320 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:09:00.756345 28418 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:00.726344 25937 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 9 LEARNER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:00.774917 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:00.779853 25389 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
W20260709 13:09:00.781464 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 188993us
W20260709 13:09:00.796759 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 104264us
I20260709 13:09:00.798238 25911 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:00.798846 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:00.798908 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:00.796711 25358 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:00.800025 25941 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 9 LEARNER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 9: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:00.797770 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:00.800050 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:00.803153 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:00.721573 28778 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:00.805881 25945 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:09:00.818588 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:00.819141 25387 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:00.819304 25937 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:00.818113 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:00.820348 25387 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: []
I20260709 13:09:00.821012 25911 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:00.822661 25911 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:00.825764 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 62731.1us
I20260709 13:09:00.826821 25938 raft_consensus.cc:2393] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 7: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:00.827164 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:00.828214 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:00.828373 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:00.828491 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:00.776543 28784 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:00.834901 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:00.837533 25943 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:09:00.838374 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:00.839695 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:00.840570 25909 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:00.818655 28778 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:00.793563 28682 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:00.705812 28728 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:00.725900 28604 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:00.823410 28717 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:00.850864 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:00.843382 25358 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:00.845566 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:00.858853 25939 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:00.806170 28781 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 3, Time since last communication: 0.050s
I20260709 13:09:00.860091 25358 leader_election.cc:400] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5.
I20260709 13:09:00.860399 25358 leader_election.cc:403] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:09:00.836377 28768 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:09:00.855201 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:00.831622 28418 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:00.796955 28764 consensus_queue.cc:1059] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
I20260709 13:09:00.841969 28778 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:00.837544 28785 raft_consensus.cc:2764] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 13:09:00.844520 28716 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:00.876317 25939 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:00.876681 25315 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:00.859849 28762 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:00.859529 28682 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:00.863795 28717 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:00.870488 28716 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:00.879055 28440 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:00.835610 28784 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:00.879771 28418 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 4. Current term is 5.
I20260709 13:09:00.892114 28788 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:00.911363 28788 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:00.911643 28784 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:00.912151 28788 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:00.912706 28786 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 13:09:00.895081 28440 raft_consensus.cc:697] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:00.914204 28440 consensus_queue.cc:237] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:00.885078 28743 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:00.900842 28716 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:00.862282 28733 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:00.905463 28717 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:00.890800 28438 consensus_queue.cc:1059] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260709 13:09:00.906163 25387 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 4. Current term is 5. Ops: []
I20260709 13:09:00.883760 28762 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:00.907447 28604 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:00.877460 28712 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:00.953580 25246 raft_consensus.cc:2393] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:00.908653 28771 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:00.967852 28717 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:09:01.015545 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 155278us
W20260709 13:09:01.018573 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 158870us
I20260709 13:09:01.030253 25359 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:01.026203 25358 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:01.033535 25358 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:01.035137 25358 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:01.018848 28664 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:01.019189 28682 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:01.036974 28418 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:09:01.017238 28717 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:01.037993 28785 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:01.063774 28440 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:09:01.075722 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 58955us
I20260709 13:09:01.081655 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:01.082778 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:00.979135 28778 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.085834 25386 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:09:01.085868 25209 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:01.072350 28733 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:01.080394 28757 log.cc:927] Time spent T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 1.077s	user 0.007s	sys 0.000s
I20260709 13:09:01.072264 28785 raft_consensus.cc:697] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:01.075541 28418 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:09:01.096120 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 61071.9us
I20260709 13:09:01.080562 28712 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:01.081092 28805 consensus_queue.cc:1059] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20260709 13:09:01.102423 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:01.108739 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:01.085741 28762 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:01.085700 28721 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:09:01.112288 25358 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:01.114811 28808 raft_consensus.cc:3055] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:01.114701 28785 consensus_queue.cc:237] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:01.115334 28808 raft_consensus.cc:740] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:01.116400 28808 consensus_queue.cc:260] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:01.114723 28762 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:01.117781 28808 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:01.119572 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 279062us
I20260709 13:09:01.121830 28765 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:09:01.121862 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:01.125311 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:01.087340 28794 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.025s
I20260709 13:09:01.102762 28771 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:01.108384 28733 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:01.135452 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:01.137357 25209 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:01.112871 28800 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:01.146853 28712 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:01.147360 28712 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:01.147727 28712 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:01.151930 28716 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.155969 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 510) took 1120 ms. Trace:
I20260709 13:09:01.156404 25316 rpcz_store.cc:276] 0709 13:09:00.035296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:00.037479 (+  2183us) service_pool.cc:224] Handling call
0709 13:09:01.155818 (+1118339us) spinlock_profiling.cc:243] Waited 9.33 ms on lock 0x7b54000f9454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:01.155912 (+    94us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21469440}
I20260709 13:09:01.156741 28762 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:01.157493 28762 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:01.159256 28418 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:01.159724 28418 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:01.160108 28418 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:01.153426 28717 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:01.119586 28778 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:01.165547 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:01.170305 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:01.172004 25389 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:09:01.145287 25209 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:01.175012 25209 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:01.112514 28440 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 13:09:01.179217 28753 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:09:01.179657 28753 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:01.180006 28753 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:01.183188 25284 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:01.184811 28814 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:01.185463 28778 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:01.185774 25284 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:01.185925 28778 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:01.166922 28788 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.142824 28804 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
I20260709 13:09:01.197026 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:01.189734 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:01.196370 25392 raft_consensus.cc:1275] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:09:01.142982 28805 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:01.186484 28778 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:09:01.206550 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:01.209255 28571 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:01.210203 28814 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:09:01.210433 28418 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.212841 25938 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:01.214679 28776 consensus_queue.cc:1059] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:09:01.201306 25284 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:01.215639 28734 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:01.215926 28571 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:01.216337 28571 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:01.216739 28721 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:01.217104 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:01.217274 28721 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.217922 28794 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:01.218000 28554 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:01.216756 28571 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:01.218557 28794 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.219599 28721 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:01.220304 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:01.220678 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:01.218575 28554 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:01.221910 28721 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:01.222441 28721 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.217432 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:01.235105 25240 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:01.222913 28554 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:01.223078 28788 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:01.223742 28794 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:01.225768 28751 consensus_meta.cc:287] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.630s	user 0.012s	sys 0.000s
I20260709 13:09:01.226068 25284 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:01.186519 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:01.244670 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 58202.7us
I20260709 13:09:01.245512 25209 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:01.164141 28768 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:09:01.234022 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2131) took 1133 ms. Trace:
I20260709 13:09:01.247833 25388 rpcz_store.cc:276] 0709 13:09:00.067205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:00.067364 (+   159us) service_pool.cc:224] Handling call
0709 13:09:01.196854 (+1129490us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:09:01.200410 (+  3556us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7282048}
I20260709 13:09:01.236632 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:01.248797 25209 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:01.257438 25388 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:01.266680 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:01.267197 25909 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:01.268745 25240 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:09:01.270644 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:01.272673 25909 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:01.277098 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:01.324581 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 52024.8us
I20260709 13:09:01.356734 25941 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
I20260709 13:09:01.238862 28762 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:01.361433 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:01.367105 25386 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: [9.6-9.6]
W20260709 13:09:01.368955 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 142947us
I20260709 13:09:01.370020 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:01.370388 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:01.369985 25284 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:01.371062 25239 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:01.371757 25284 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:01.247087 28814 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:01.372906 25937 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 13:09:01.373798 25386 raft_consensus.cc:1275] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:09:01.373982 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:01.255432 28554 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:09:01.256623 28222 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:01.375870 25284 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:01.359651 28733 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:09:01.357303 28664 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:01.359565 28788 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:01.250180 28751 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:01.389335 25944 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:09:01.273937 28805 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.107s
I20260709 13:09:01.382874 25388 raft_consensus.cc:2955] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:01.292966 28794 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.088s
I20260709 13:09:01.375900 28812 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:01.378069 28222 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:01.365752 28571 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.376821 28554 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:01.370980 28777 consensus_queue.cc:1059] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.020s
I20260709 13:09:01.379310 28814 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:01.383133 28788 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:01.382834 28811 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:01.394454 28222 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:01.402668 28788 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:01.404114 28571 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:01.404825 28571 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:09:01.405254 28571 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:01.405640 28571 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:09:01.410359 28774 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 1.192s	user 0.002s	sys 0.003s
W20260709 13:09:01.411216 28787 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.539s	user 0.001s	sys 0.001s
I20260709 13:09:01.412312 28447 raft_consensus.cc:3055] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:01.412737 28447 raft_consensus.cc:740] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:01.412957 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:01.413789 28418 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:01.415810 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1537) took 1133 ms. Trace:
I20260709 13:09:01.416222 25387 rpcz_store.cc:276] 0709 13:09:00.282070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:00.824743 (+542673us) service_pool.cc:224] Handling call
0709 13:09:00.824991 (+   248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:00.928152 (+103161us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:01.415553 (+487401us) spinlock_profiling.cc:243] Waited 81 ms on lock 0x7b5400120cd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:01.415747 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":186196352}
I20260709 13:09:01.419461 28791 raft_consensus.cc:3055] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:09:01.419941 28791 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:01.420861 28791 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.428356 25239 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:01.430138 25285 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:01.397876 28554 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:09:01.453768 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:01.441093 28702 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:09:01.458864 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 83490.1us
I20260709 13:09:01.442054 28733 raft_consensus.cc:697] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:01.463419 25284 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:01.464740 28715 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.434s	user 0.008s	sys 0.007s
W20260709 13:09:01.465201 28715 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.434s	user 0.009s	sys 0.007s
I20260709 13:09:01.467103 28797 raft_consensus.cc:3055] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:01.467579 28797 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:01.473666 28702 raft_consensus.cc:697] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:01.474699 28571 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.474331 28814 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:01.475289 28702 consensus_queue.cc:237] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:01.478474 25285 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:01.413794 28447 consensus_queue.cc:260] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.497579 25239 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 13:09:01.502432 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1544) took 1074 ms. Trace:
I20260709 13:09:01.506212 28778 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:01.508518 28778 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:01.509187 25243 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:01.509419 25242 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:01.509971 28753 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:01.511644 28222 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:01.423501 28811 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.457813 28712 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:01.512319 28753 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:01.557766 25240 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:01.490278 28721 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:01.558167 25242 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 13:09:01.560177 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 87881.7us
I20260709 13:09:01.567771 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:01.568442 25939 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:01.568600 25317 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 13:09:01.567981 25316 raft_consensus.cc:1240] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: []
I20260709 13:09:01.528048 28814 raft_consensus.cc:697] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:01.571957 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:01.569839 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:01.574313 25315 raft_consensus.cc:2410] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 13:09:01.570473 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:01.575007 25312 raft_consensus.cc:2393] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 7: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:01.576565 25945 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:01.580530 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:01.563558 28812 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:09:01.620208 25939 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:01.620782 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:01.623001 25393 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 7: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:01.623386 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2132) took 1514 ms. Trace:
I20260709 13:09:01.623588 25393 rpcz_store.cc:276] 0709 13:09:00.109412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:00.109575 (+   163us) service_pool.cc:224] Handling call
0709 13:09:01.582413 (+1472838us) spinlock_profiling.cc:243] Waited 6.49 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:09:01.623343 (+ 40930us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14920064}
I20260709 13:09:01.582224 28814 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.635923 28554 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.640406 28222 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:01.646447 25246 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:01.622042 28751 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:01.666115 28811 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:01.666008 28571 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:01.684113 28776 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:01.685220 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 304191us
I20260709 13:09:01.688022 25241 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
W20260709 13:09:01.692510 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:01.697192 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 218807us
I20260709 13:09:01.701256 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:01.705456 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:01.706571 25939 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 8: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:01.707949 25944 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:09:01.684772 28776 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.622428 28733 consensus_queue.cc:237] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:01.721907 28820 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
I20260709 13:09:01.724623 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:01.720396 28826 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:01.729147 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:09:01.718333 28828 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.157s	user 0.001s	sys 0.000s
I20260709 13:09:01.733403 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:01.735164 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:01.735769 25245 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:01.738075 28712 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:01.739770 28554 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:01.745297 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:01.746137 25394 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2134) took 1635 ms (client timeout 1622 ms). Trace:
W20260709 13:09:01.746500 25394 rpcz_store.cc:269] 0709 13:09:00.110424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:00.111083 (+   659us) service_pool.cc:224] Handling call
0709 13:09:01.745148 (+1634065us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:09:01.746057 (+   909us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46865792}
I20260709 13:09:01.750025 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:01.750684 25944 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:01.756217 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 590) took 1269 ms. Trace:
I20260709 13:09:01.757258 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:01.758217 25944 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:09:01.756616 25389 rpcz_store.cc:276] 0709 13:09:00.486359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:01.173017 (+686658us) service_pool.cc:224] Handling call
0709 13:09:01.173455 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:01.173672 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:09:01.256224 (+ 82552us) log.cc:844] Serialized 14 byte log entry
0709 13:09:01.755738 (+499514us) spinlock_profiling.cc:243] Waited 4.07 ms on lock 0x7b1800078340. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f297685a4bc 00007f2973ba848b 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f297685223a
0709 13:09:01.755913 (+   175us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:01.755957 (+    44us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:01.756003 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:01.756061 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:01.756163 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4402,"prepare.queue_time_us":192489,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":1939,"spinlock_wait_cycles":9357696,"thread_start_us":189909,"threads_started":2,"wal-append.queue_time_us":145009}
I20260709 13:09:01.764230 25317 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7. Ops: []
I20260709 13:09:01.764014 25245 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:01.768656 28788 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.773418 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:01.787176 28788 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:01.787963 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 593) took 1108 ms. Trace:
I20260709 13:09:01.792804 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:01.793687 25320 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:01.796751 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:01.797596 25313 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:01.788553 25392 rpcz_store.cc:276] 0709 13:09:00.679790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:01.203837 (+524047us) service_pool.cc:224] Handling call
0709 13:09:01.204258 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:01.204824 (+   566us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:09:01.369702 (+164878us) log.cc:844] Serialized 14 byte log entry
0709 13:09:01.694869 (+325167us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:01.705335 (+ 10466us) log.cc:844] Serialized 26 byte log entry
0709 13:09:01.705472 (+   137us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:01.705514 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:01.705578 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:01.787754 (+ 82176us) raft_consensus.cc:1685] finished
0709 13:09:01.787779 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:01.787907 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15129,"prepare.queue_time_us":180625,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":226,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":586,"thread_start_us":426958,"threads_started":2,"wal-append.queue_time_us":326351}
W20260709 13:09:01.792429 28718 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.760s	user 0.002s	sys 0.014s
I20260709 13:09:01.808566 25321 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:01.810745 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:01.812328 25321 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:09:01.816699 25358 leader_election.cc:400] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9.
W20260709 13:09:01.810063 28718 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.780s	user 0.002s	sys 0.014s
W20260709 13:09:01.824609 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:01.830646 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:01.834445 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:01.835323 25389 raft_consensus.cc:1275] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:09:01.836431 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:01.836781 25212 leader_election.cc:336] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.538s (SENT)
I20260709 13:09:01.837786 25212 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:01.838569 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:01.838917 25212 connection.cc:256] RPC call timeout handler was delayed by -1.0045618950000517s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18380000000000002s for a response
I20260709 13:09:01.838953 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:01.841501 25212 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:01.841488 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:01.842000 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:01.842625 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:01.844652 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:01.843245 25386 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:01.843945 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:01.831595 28702 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:01.847290 25387 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:01.847929 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:01.848727 25321 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:01.844213 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:01.840513 28571 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:01.850358 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:01.850418 28571 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:01.851671 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:01.852712 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:01.853356 25321 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:01.852813 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:01.854152 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:01.854775 25390 raft_consensus.cc:2393] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:01.855089 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:01.852598 28571 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:01.847235 28733 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:01.854781 25321 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 6: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:01.852577 28728 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:01.837431 28840 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:01.866858 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:01.842948 28712 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:01.872773 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:01.831838 28751 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:01.874953 25387 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:01.875696 25313 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:01.877693 25386 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:09:01.843914 25212 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:01.880162 25320 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:09:01.880256 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:01.881376 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:01.882076 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:01.887917 25312 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:09:01.888895 28447 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:01.889237 28664 raft_consensus.cc:3055] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:01.888748 28751 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.889387 28805 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:01.889721 28664 raft_consensus.cc:740] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:01.880921 28702 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:01.890578 28664 consensus_queue.cc:260] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.888880 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:01.891608 28664 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:01.893100 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:01.894376 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:01.891846 28751 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:01.892066 28702 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:01.888759 25321 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:01.887252 25212 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:01.898236 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:01.898852 25244 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:09:01.900033 25239 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: []
I20260709 13:09:01.900862 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:01.901540 25318 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:09:01.902809 28805 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:01.907284 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:01.902710 28721 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:01.886932 28840 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.910488 28721 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:01.910961 25316 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:01.911032 28721 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:01.902930 28438 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:09:01.913560 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:01.907548 25209 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:01.908614 28835 consensus_queue.cc:1059] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.003s
I20260709 13:09:01.885986 28826 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:01.916450 25209 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:01.919734 25209 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:01.881523 28736 consensus_queue.cc:1059] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260709 13:09:01.924700 25316 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:09:01.926831 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:01.927062 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:01.929838 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:01.930150 28801 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:09:01.930619 25312 raft_consensus.cc:2368] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7.
I20260709 13:09:01.930809 28801 raft_consensus.cc:697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:01.931942 25312 raft_consensus.cc:1275] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:09:01.931751 28801 consensus_queue.cc:237] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:01.933337 28764 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:09:01.933907 28764 raft_consensus.cc:697] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:01.934859 28764 consensus_queue.cc:237] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:01.937652 28835 raft_consensus.cc:3055] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:09:01.938140 28835 raft_consensus.cc:740] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:01.919793 28711 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260709 13:09:01.916836 28840 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:01.940339 25209 leader_election.cc:400] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7.
I20260709 13:09:01.943485 25209 leader_election.cc:403] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 13:09:01.941530 25287 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:01.918406 28438 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:01.944758 28730 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:01.957038 28438 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:01.954375 28736 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260709 13:09:01.958060 28438 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:01.963295 28829 log.cc:927] Time spent T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.469s	user 0.004s	sys 0.000s
I20260709 13:09:01.955340 25212 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:01.965679 28822 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:09:01.967717 25212 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:01.969094 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 937684us
I20260709 13:09:01.969990 28822 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:09:01.970361 25359 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:01.971654 28771 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:09:01.951930 28778 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20260709 13:09:01.976761 25212 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:01.981863 25212 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:01.939060 28835 consensus_queue.cc:260] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:01.951972 28805 consensus_queue.cc:1059] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260709 13:09:01.921413 28826 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:01.982259 28736 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:09:02.002624 28826 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:02.003914 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:02.007963 25359 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:02.010382 28856 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:09:02.012876 25238 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:09:01.993772 28711 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:09:01.988440 28791 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:02.014394 28721 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:02.028841 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 47038us
I20260709 13:09:01.983029 28795 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260709 13:09:02.031776 25212 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:02.033241 25212 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:02.034075 25212 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:02.034682 28733 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:02.034930 28778 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:02.035775 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:02.036466 28733 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 13:09:02.048905 25212 leader_election.cc:336] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.144s (SENT)
I20260709 13:09:02.033659 28728 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:09:01.502872 25391 rpcz_store.cc:276] 0709 13:09:00.428358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:01.170001 (+741643us) service_pool.cc:224] Handling call
0709 13:09:01.170452 (+   451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:01.170736 (+   284us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:09:01.366443 (+195707us) log.cc:844] Serialized 14 byte log entry
0709 13:09:01.498921 (+132478us) log.cc:844] Serialized 14 byte log entry
0709 13:09:01.499839 (+   918us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:01.499886 (+    47us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:09:01.499936 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:01.502290 (+  2354us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:01.502383 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7201,"prepare.queue_time_us":78692,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":71,"raft.queue_time_us":3332,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":3301,"spinlock_wait_cycles":21860608,"thread_start_us":315940,"threads_started":2,"wal-append.queue_time_us":127292}
I20260709 13:09:02.060231 28778 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:09:02.061822 25940 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:09:02.074157 25386 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 13:09:02.080786 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:02.093614 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:02.094692 25394 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: [9.6-9.6]
W20260709 13:09:02.096421 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:02.090099 25908 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:02.064280 28856 raft_consensus.cc:697] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:02.104452 28797 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:02.123780 28437 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:02.124238 28437 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:02.124636 28437 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:09:02.105206 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 178451us
I20260709 13:09:02.124492 28721 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:02.124840 28797 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:02.105779 28664 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:09:02.130954 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:02.131552 25936 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:02.114976 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:02.115309 28776 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:02.118680 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:02.123212 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:02.137490 25941 raft_consensus.cc:2393] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 10: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:02.122556 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:02.131098 25241 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 7. (term mismatch)
I20260709 13:09:02.113888 25943 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:02.138051 25241 pending_rounds.cc:85] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 6
I20260709 13:09:02.138079 25939 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:02.138494 25241 pending_rounds.cc:107] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20260709 13:09:02.133701 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:02.136229 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:02.138322 25386 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:02.130954 25209 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:02.139035 25241 raft_consensus.cc:2889] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:02.139459 25944 raft_consensus.cc:2368] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7.
I20260709 13:09:02.124728 28716 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:02.127511 28755 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:02.181106 28850 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.147s	user 0.000s	sys 0.001s
I20260709 13:09:02.117069 28730 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7.
I20260709 13:09:02.217685 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:02.217880 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 76636.3us
I20260709 13:09:02.226845 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:02.217713 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:02.218678 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:02.218299 25212 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:02.246815 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 105476us
I20260709 13:09:02.247690 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:02.249188 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 107203us
I20260709 13:09:02.217355 28222 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:02.258886 25941 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 13:09:02.255252 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 585) took 1552 ms. Trace:
I20260709 13:09:02.259684 25238 rpcz_store.cc:276] 0709 13:09:00.702835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:00.703780 (+   945us) service_pool.cc:224] Handling call
0709 13:09:00.704202 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:02.013383 (+1309181us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:02.254952 (+241569us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400364a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:02.255181 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3008748672}
I20260709 13:09:02.269914 25237 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:02.274108 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 763) took 2161 ms. Trace:
I20260709 13:09:02.250247 28721 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:02.250756 28822 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:02.251554 28734 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:02.297807 28734 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:02.298228 28734 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:02.251502 28783 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:02.302278 28778 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:02.281013 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:02.303651 25941 raft_consensus.cc:2368] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9.
I20260709 13:09:02.274570 25942 rpcz_store.cc:276] 0709 13:09:00.113010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:00.241925 (+128915us) service_pool.cc:224] Handling call
0709 13:09:00.249875 (+  7950us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:00.503271 (+253396us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:09:00.634640 (+131369us) log.cc:844] Serialized 14 byte log entry
0709 13:09:00.836117 (+201477us) log.cc:844] Serialized 14 byte log entry
0709 13:09:01.116723 (+280606us) spinlock_profiling.cc:243] Waited 1.81 ms on lock 0x7b180003ef20. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b3fc0a 00007f297d27f219 00007f2973bc7a40 00007f2973bc24e4 00007f2973bc2455 00007f2976caa060 00007f2976cbc716 00007f2976ccd287 00007f2976ccd208
0709 13:09:01.169397 (+ 52674us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:01.169466 (+    69us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:01.169536 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:01.871580 (+702044us) spinlock_profiling.cc:243] Waited 3.99 ms on lock 0x7b180003ef20. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:09:02.273876 (+402296us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b540062a2d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:02.273925 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:02.274048 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":264452,"prepare.queue_time_us":190811,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":10913,"raft.queue_time_us":319330,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":220,"spinlock_wait_cycles":595552256,"thread_start_us":349845,"threads_started":3,"wal-append.queue_time_us":257015,"wal-append.run_cpu_time_us":1911,"wal-append.run_wall_time_us":1048630}
I20260709 13:09:02.267894 28730 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:02.226781 28594 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 13:09:02.293711 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:02.295264 25938 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:02.295970 28437 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:02.309760 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 300098us
I20260709 13:09:02.251665 28712 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 13:09:02.310914 25359 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:02.314007 25358 leader_election.cc:400] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9.
I20260709 13:09:02.314428 25358 leader_election.cc:403] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 13:09:02.303150 28778 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:02.336117 25395 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:02.312311 28717 consensus_queue.cc:1059] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20260709 13:09:02.308789 28730 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:02.336817 28858 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:02.335917 28464 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:02.333683 28716 consensus_queue.cc:1059] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 13:09:02.339664 28755 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:09:02.398269 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:02.400346 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 169710us
I20260709 13:09:02.339599 28784 consensus_queue.cc:1059] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.056s
I20260709 13:09:02.404708 28868 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:02.405360 28809 raft_consensus.cc:3055] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 13:09:02.409754 28860 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260709 13:09:02.411160 28778 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:02.380720 28823 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.738s	user 0.033s	sys 0.081s
I20260709 13:09:02.420104 25389 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 13:09:02.424412 28716 raft_consensus.cc:3055] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:02.415818 28809 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:09:02.427837 28861 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.181s	user 0.001s	sys 0.000s
I20260709 13:09:02.375613 28856 consensus_queue.cc:237] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:02.400545 28464 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:02.448417 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 113228us
W20260709 13:09:02.448549 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 203077us
I20260709 13:09:02.450819 25909 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:02.451323 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:02.453451 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:02.455757 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:02.458210 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 147346us
I20260709 13:09:02.459126 25359 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:02.462055 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:02.399245 28858 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:02.447803 28823 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.818s	user 0.033s	sys 0.081s
I20260709 13:09:02.439595 28716 raft_consensus.cc:740] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:02.466953 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:02.477181 25238 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:02.478480 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:02.482527 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:02.486606 25244 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:02.464025 28730 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:02.495388 25243 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 13:09:02.496104 25239 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:09:02.462517 28783 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:09:02.472527 28800 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:02.504968 25243 pending_rounds.cc:85] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 5
I20260709 13:09:02.513494 25243 pending_rounds.cc:107] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 13:09:02.513903 25243 raft_consensus.cc:2889] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:02.490759 25238 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8. Ops: []
W20260709 13:09:02.487828 25908 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.958s (SENT) [suppressed 23 similar messages]
I20260709 13:09:02.480358 28505 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.019s
I20260709 13:09:02.493881 28437 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:02.479877 28754 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260709 13:09:02.528429 28464 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:02.497615 28872 consensus_queue.cc:1059] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.036s
I20260709 13:09:02.530144 28872 raft_consensus.cc:3055] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:02.530560 28872 raft_consensus.cc:740] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:02.531479 28872 consensus_queue.cc:260] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:02.519429 28505 consensus_queue.cc:1243] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 13:09:02.539856 25395 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2130) took 2482 ms (client timeout 2144 ms). Trace:
I20260709 13:09:02.546960 28875 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:02.547660 28875 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:02.510547 28809 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:02.552585 28809 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:02.555282 28875 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:02.555135 28734 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:02.540416 25395 rpcz_store.cc:269] 0709 13:09:00.056973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:00.205847 (+148874us) service_pool.cc:224] Handling call
0709 13:09:01.189575 (+983728us) spinlock_profiling.cc:243] Waited 9.36 ms on lock 0x7b580019fe98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:09:02.539585 (+1350010us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54003237d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:02.539724 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2632374400}
I20260709 13:09:02.558878 28755 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9.
I20260709 13:09:02.515448 28800 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:02.574987 28800 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:02.480089 28727 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:02.577919 28727 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:02.580619 28727 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:02.549037 28721 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:09:02.582041 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 94237.6us
I20260709 13:09:02.583948 25938 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8. Ops: []
W20260709 13:09:02.588405 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 646886us
I20260709 13:09:02.542320 28872 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:02.589805 25287 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:02.593096 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:02.592581 28840 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:09:02.599264 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 482688us
I20260709 13:09:02.587787 28727 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:02.609870 25287 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:02.596998 28755 consensus_queue.cc:1059] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.015s
W20260709 13:09:02.614117 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:02.598145 28840 raft_consensus.cc:697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:02.604125 28702 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:02.638084 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:02.639801 28702 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:02.649876 28868 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:02.657851 28702 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:02.659586 28702 consensus_queue.cc:1059] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:09:02.662706 28835 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:09:02.613238 28727 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:02.677949 28840 consensus_queue.cc:237] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:02.672780 28727 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:02.693061 28734 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:02.697293 28727 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:02.727602 28833 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:02.736099 28727 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:02.729696 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 129523us
I20260709 13:09:02.730396 28897 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 13:09:02.766856 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 69720.8us
I20260709 13:09:02.768919 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:02.769542 25389 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:02.747660 28833 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:02.787249 25395 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:09:02.789086 28440 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:02.761313 28800 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:02.785270 28889 raft_consensus.cc:3055] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:02.785597 28833 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:02.805253 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:02.810850 25241 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:09:02.813337 28727 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:02.791808 28712 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.022s
I20260709 13:09:02.816859 28800 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:02.789674 28440 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:02.819667 28440 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:02.820181 28858 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:02.820521 28898 consensus_queue.cc:1059] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:02.822052 28890 raft_consensus.cc:3055] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:02.822503 28890 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:02.823088 28716 consensus_queue.cc:260] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:02.823494 28890 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:02.824381 28716 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:02.801862 28889 raft_consensus.cc:740] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:02.826079 28889 consensus_queue.cc:260] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:02.827082 28889 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:02.829443 25386 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: []
W20260709 13:09:02.842612 28881 log.cc:927] Time spent T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.196s	user 0.001s	sys 0.003s
W20260709 13:09:02.845055 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 235275us
I20260709 13:09:02.846822 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:02.852046 28880 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.182s	user 0.000s	sys 0.002s
W20260709 13:09:02.856106 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:02.858417 25391 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:02.863704 28440 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:02.864449 28440 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:02.878002 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:02.878136 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:02.880861 25389 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:02.881031 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:02.881970 25387 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:02.883924 25938 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:09:02.883944 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:02.886792 25287 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:02.887154 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:02.888489 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:02.889323 25243 raft_consensus.cc:2410] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 13:09:02.889779 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:02.886870 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:02.891263 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:02.891835 25938 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 5.
I20260709 13:09:02.892624 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:02.893268 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:02.893356 25389 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:02.894155 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:02.896818 25943 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 13:09:02.898866 25243 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 13:09:02.902442 25394 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:02.902925 25238 pending_rounds.cc:85] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 6
I20260709 13:09:02.903252 25238 pending_rounds.cc:107] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20260709 13:09:02.903499 25238 raft_consensus.cc:2889] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:02.904927 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:02.905040 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:02.912035 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:02.912974 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:02.918234 25239 raft_consensus.cc:2393] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:02.911509 25237 raft_consensus.cc:2393] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 10: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:02.914621 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:02.880626 28557 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:02.924657 25287 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:02.926218 28702 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:09:02.926641 28702 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:02.926994 28702 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:09:02.889288 28440 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:02.924813 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:02.930521 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:02.970938 25391 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:09:02.971138 25395 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:02.898808 28899 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:02.976477 28891 consensus_queue.cc:1048] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:09:02.977226 28891 consensus_queue.cc:1243] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer a2a77ecf2c0941618b5bc79d4c04db61 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 13:09:02.979481 25389 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:02.989391 25392 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20260709 13:09:02.990715 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 227375us
I20260709 13:09:02.931439 28840 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:02.974442 25394 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:09:02.974258 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:02.999993 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:03.002306 25239 raft_consensus.cc:2410] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 10 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:09:02.931732 28808 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:03.039311 25212 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:03.011826 25321 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:09:02.934515 28588 consensus_queue.cc:1059] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20260709 13:09:02.931334 28557 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:02.980963 28791 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:02.997205 28447 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:03.089391 28712 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:03.097983 28447 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:03.098579 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:03.093994 25209 leader_election.cc:400] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 6 election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 6. Current term is 7.
W20260709 13:09:03.099388 28902 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.168s	user 0.000s	sys 0.000s
I20260709 13:09:03.100306 28447 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:03.101146 28712 raft_consensus.cc:697] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:09:03.098330 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 59397.1us
I20260709 13:09:02.999365 28801 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:02.934792 28814 consensus_queue.cc:1059] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.074s
I20260709 13:09:03.101009 28438 consensus_queue.cc:1059] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:09:03.001291 28840 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:03.004123 28897 raft_consensus.cc:697] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:03.037653 28727 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:09:03.103415 25285 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:03.112903 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:03.004419 28721 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:03.101027 25209 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:03.083418 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:03.092202 28557 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:03.007732 28717 consensus_queue.cc:1059] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.104s
I20260709 13:09:03.049566 28856 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
I20260709 13:09:03.071935 28740 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20260709 13:09:03.012099 28808 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:03.115192 28727 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:03.086918 28833 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:03.135455 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:03.112903 28840 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:03.141258 25245 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:09:03.164702 25312 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:09:03.142732 25313 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:03.143060 28712 consensus_queue.cc:237] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:03.154575 28808 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:03.143795 28554 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:09:03.110939 28801 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:03.122027 28721 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:03.177529 28702 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:03.178231 28801 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:03.131824 28835 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:09:03.149492 28896 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.174s	user 0.000s	sys 0.002s
I20260709 13:09:03.102099 28892 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.081s
I20260709 13:09:03.150840 28833 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:03.151609 28814 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:03.140362 28727 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:09:03.154505 25318 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 13:09:03.155025 28905 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 13:09:03.102418 28874 consensus_queue.cc:1059] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.063s
I20260709 13:09:03.107800 28791 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:03.193013 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:03.193883 28791 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:03.129908 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.158967 25317 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:03.140596 25314 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:09:03.134428 28755 raft_consensus.cc:3055] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:03.199597 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:03.200348 25318 raft_consensus.cc:2376] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7. Re-sending same reply.
I20260709 13:09:03.188396 28814 raft_consensus.cc:697] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:09:03.206687 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 166287us
I20260709 13:09:03.206866 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:03.208628 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:03.209794 25284 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:03.211131 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:03.197954 28755 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:03.196655 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:03.197295 25358 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:03.215103 25358 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:03.155021 28897 consensus_queue.cc:237] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:03.212958 25392 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 8: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:03.179986 28702 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:03.216156 28874 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:03.206802 28831 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:03.217235 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:03.223182 28682 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:03.223574 28564 consensus_queue.cc:1059] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:09:03.223825 28682 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:03.224653 28564 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:03.225107 28564 raft_consensus.cc:697] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:03.225999 28564 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:03.227241 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:03.228139 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:03.217566 28440 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:03.215322 28801 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.015s
I20260709 13:09:03.241335 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:03.242681 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:03.243630 25391 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 13:09:03.245584 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 130457us
I20260709 13:09:03.230535 28438 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.021s
I20260709 13:09:03.252288 28557 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:09:03.241292 25387 pending_rounds.cc:85] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 3
I20260709 13:09:03.253079 25387 pending_rounds.cc:107] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260709 13:09:03.253379 25387 raft_consensus.cc:2889] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:03.254484 28663 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:09:03.255327 28801 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:03.235910 28831 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:03.252825 28557 raft_consensus.cc:697] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:03.255921 28801 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:03.256716 28557 consensus_queue.cc:237] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:03.257407 28831 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:03.257910 28801 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:03.258078 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:03.251181 28702 consensus_queue.cc:1059] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.015s
I20260709 13:09:03.258466 28682 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:03.249365 28440 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:09:03.265807 28440 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:03.266247 28440 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:09:03.267336 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:03.268560 28874 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:09:03.235646 28791 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.036s
I20260709 13:09:03.249931 28588 raft_consensus.cc:3055] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:03.273209 28791 consensus_queue.cc:1243] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer a2a77ecf2c0941618b5bc79d4c04db61 log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:09:03.273551 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:03.279525 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:03.283489 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:03.284104 25389 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:09:03.287153 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:03.288161 25390 raft_consensus.cc:2393] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:02.982033 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:09:03.235556 28796 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:03.239135 28447 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:09:03.292106 28834 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.843s	user 0.004s	sys 0.000s
I20260709 13:09:03.273588 28588 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:09:03.377022 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 76915.2us
I20260709 13:09:03.410229 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.436926 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.439491 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.440317 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.440732 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:03.440392 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.441576 25321 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:03.300323 28822 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:03.442970 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:03.443717 25389 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:03.415253 28892 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:03.446023 28892 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:03.280949 28755 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:03.448534 28755 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:03.449429 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.450302 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.450508 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.451105 25319 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:03.452076 25392 pending_rounds.cc:85] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 4
I20260709 13:09:03.452443 25392 pending_rounds.cc:107] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260709 13:09:03.452684 25392 raft_consensus.cc:2889] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:03.300848 28796 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:03.443146 28822 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:03.326315 28447 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:03.249541 28814 consensus_queue.cc:237] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:03.411648 28831 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:09:03.461478 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 50648.7us
I20260709 13:09:03.461730 28831 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:03.463845 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.481105 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.481187 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.484707 25287 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:03.443058 28834 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 13:09:03.300405 28222 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:03.491798 28831 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:03.492496 28588 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:03.497066 25318 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:09:03.494072 28222 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:03.510285 28447 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:03.511444 28840 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:03.513473 28222 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:03.515130 28809 raft_consensus.cc:3055] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:03.515698 28809 raft_consensus.cc:740] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:03.522109 25908 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:03.518402 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:03.524927 25386 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:03.519316 25358 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:03.483024 28438 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:03.536401 25321 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:03.542578 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:03.517673 28899 raft_consensus.cc:3055] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:03.517305 28914 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:03.543673 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:03.606051 25386 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:09:03.546177 25244 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:09:03.547564 25359 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:03.552664 25393 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 13:09:03.553443 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 342439us
I20260709 13:09:03.608865 25284 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:03.555269 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:03.559989 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.563005 25241 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: [8.5-9.6]
I20260709 13:09:03.544773 28588 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:03.516855 28809 consensus_queue.cc:260] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:03.567451 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.613036 25318 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:03.529867 28447 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 13:09:03.530429 28833 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
W20260709 13:09:03.569857 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 466513us
W20260709 13:09:03.571293 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:03.540081 28438 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:03.566409 28840 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:03.593559 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 319471us
W20260709 13:09:03.603869 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 61370.4us
I20260709 13:09:03.628137 25358 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:03.629978 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:03.602430 28899 raft_consensus.cc:740] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:09:03.615082 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 67603.7us
I20260709 13:09:03.635403 25359 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:03.593020 28822 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:03.623368 25285 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:03.612986 25284 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:03.606835 28557 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:03.629053 28858 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:03.644819 28718 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:09:03.643958 25358 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:03.635545 28771 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:03.636150 28809 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:03.627936 28897 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:09:03.656502 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 43621.1us
I20260709 13:09:03.665376 25312 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:03.658749 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:03.633387 28892 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:03.642129 28795 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:03.652469 25287 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:03.670210 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:03.670970 25285 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:03.675066 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:03.675990 25386 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:03.668324 25284 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:03.651928 28868 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:03.681299 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 156625us
I20260709 13:09:03.643023 28796 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:03.653417 28858 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:03.684799 25285 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:03.655346 28718 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:03.694584 28898 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:09:03.696463 28892 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 13:09:03.697561 28916 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 13:09:03.698079 28917 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
I20260709 13:09:03.663357 28856 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:03.662819 28874 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:03.669682 28831 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:03.646629 28785 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260709 13:09:03.661219 28918 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:03.693397 28897 raft_consensus.cc:697] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:03.728201 28718 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:09:03.686966 28771 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:03.717581 28858 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:09:03.769021 28921 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 13:09:03.682523 28926 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 13:09:03.771154 28893 log.cc:927] Time spent T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.678s	user 0.000s	sys 0.010s
I20260709 13:09:03.771579 28904 raft_consensus.cc:3055] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:09:03.772101 28904 raft_consensus.cc:740] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:03.772717 28717 raft_consensus.cc:3055] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:03.728858 28814 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:09:03.726955 28898 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:03.733845 28889 raft_consensus.cc:2764] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 13:09:03.753728 28915 consensus_queue.cc:1059] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260709 13:09:03.766387 28438 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:03.748396 28920 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:09:03.766038 28727 consensus_meta.cc:287] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.574s	user 0.013s	sys 0.001s
I20260709 13:09:03.773743 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2077) took 1767 ms. Trace:
I20260709 13:09:03.773198 28717 raft_consensus.cc:740] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:03.779387 28898 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:03.759297 28874 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:03.752998 28222 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:09:03.789512 28440 consensus_meta.cc:287] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.523s	user 0.008s	sys 0.004s
I20260709 13:09:03.686650 28795 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:03.792506 25320 rpcz_store.cc:276] 0709 13:09:02.006345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:02.035197 (+ 28852us) service_pool.cc:224] Handling call
0709 13:09:02.035534 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:02.592245 (+556711us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:02.592277 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:02.600942 (+  8665us) log.cc:844] Serialized 26 byte log entry
0709 13:09:02.876407 (+275465us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:09:02.876446 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:02.876533 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:03.157376 (+280843us) spinlock_profiling.cc:243] Waited 4.78 ms on lock 0x7b4800271ed8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297bbdbaa1 00007f297bbd2b37 00007f297bbd29ca 00007f297bcca26f 00007f297bcc9f37 00007f297bcde55c 00007f297bcde508 00007f297bcde488 00007f297bcde443 00007f297bcdd6ec 00007f297ac07a96
0709 13:09:03.770553 (+613177us) spinlock_profiling.cc:243] Waited 56.4 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:09:03.773447 (+  2894us) raft_consensus.cc:1685] finished
0709 13:09:03.773478 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:03.773665 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":556279,"raft.queue_time_us":560963,"raft.run_cpu_time_us":2202,"raft.run_wall_time_us":559982,"spinlock_wait_cycles":140716672,"thread_start_us":245945,"threads_started":1,"wal-append.queue_time_us":397062}
W20260709 13:09:03.799045 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 138229us
W20260709 13:09:03.806358 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 179124us
I20260709 13:09:03.682906 28899 consensus_queue.cc:260] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:03.785437 28920 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:03.804229 28778 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:03.814519 28932 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:03.804559 28440 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:03.814874 28663 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:03.832688 25239 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:03.838191 25394 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:09:03.797220 28727 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:03.848336 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 180105us
W20260709 13:09:03.848256 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 204379us
I20260709 13:09:03.836503 28899 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:03.815182 28920 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:03.857442 25391 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: []
I20260709 13:09:03.851609 25321 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:09:03.850793 25358 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:03.805389 28222 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:03.853873 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:03.854094 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.858793 25319 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:09:03.856273 25394 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 13:09:03.856524 28924 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 13:09:03.861097 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.866803 25321 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:09:03.867686 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.861752 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.868728 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.862867 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:03.871644 25358 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:03.873394 25317 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:09:03.873409 25209 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:03.877350 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.877667 25313 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:09:03.879288 25315 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:03.880374 25321 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: []
I20260709 13:09:03.881476 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:03.896418 25209 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:03.913043 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:03.840047 28778 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:03.931242 25318 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: []
W20260709 13:09:03.928879 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 57325.8us
I20260709 13:09:03.935118 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:03.943744 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 217367us
I20260709 13:09:03.944492 25285 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:03.872839 28914 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:03.877600 28918 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:03.949586 25244 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:09:03.842996 28932 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:03.954810 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:03.950256 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:03.950325 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:03.952484 25242 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: []
W20260709 13:09:03.948808 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 136863us
I20260709 13:09:03.840555 28717 consensus_queue.cc:260] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:03.953547 28438 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:09:03.954133 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:03.925206 28801 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:09:03.847800 28663 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:03.999501 28801 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:03.999939 28801 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:04.001044 28663 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:03.957331 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:04.002369 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 45123.5us
I20260709 13:09:04.004422 28934 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:04.004834 28934 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:03.957787 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:04.005193 28934 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:04.007138 28721 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:04.008268 28663 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:04.008841 28663 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.010293 28727 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:04.010982 28663 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
W20260709 13:09:04.011638 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:04.011649 28222 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:03.932540 28811 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:03.876546 28889 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.075s
I20260709 13:09:04.012817 28811 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:04.013245 28811 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:03.927760 28822 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.076s
I20260709 13:09:03.941227 28912 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 13:09:03.968775 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:03.839783 28897 consensus_queue.cc:237] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:03.996107 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 61101.4us
I20260709 13:09:04.015080 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:04.016400 25358 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:04.016690 25246 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 13:09:03.996315 28937 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:04.017217 28935 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:09:04.017628 28727 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:03.997495 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:04.018487 25245 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
W20260709 13:09:04.018877 28755 consensus_meta.cc:287] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.570s	user 0.011s	sys 0.002s
I20260709 13:09:04.011742 28936 consensus_queue.cc:1059] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.052s
I20260709 13:09:04.019573 25358 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:04.019749 25395 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:04.020468 28935 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:09:04.021294 28727 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:09:03.995738 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:04.017356 28937 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:04.023689 28937 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:04.024670 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.025525 25238 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 13:09:04.025892 25238 pending_rounds.cc:85] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 5
I20260709 13:09:04.026093 25238 pending_rounds.cc:107] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 13:09:04.026306 25238 raft_consensus.cc:2889] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:04.026844 28778 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:04.027055 28811 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.014858 28920 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:04.027983 28795 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:03.951381 28891 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:09:04.029299 28889 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:04.030066 28920 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:04.029827 28889 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.030493 28811 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:04.030937 25285 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:04.031896 25285 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:04.032359 28564 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:04.032605 28831 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:03.954308 28938 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:03.944835 28913 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.089s
I20260709 13:09:04.034359 28889 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:04.034466 25285 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:03.992734 25237 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:09:03.957890 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:04.033028 28564 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:04.033440 28940 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:04.037642 28889 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:09:04.029335 28801 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:04.030095 28934 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.038957 25394 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:09:04.040766 28937 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.044603 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.045727 25285 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:04.064363 25285 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:04.035316 28868 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:04.065587 25393 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
W20260709 13:09:04.065840 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 348916us
I20260709 13:09:04.067098 28933 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:09:04.067584 28925 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:09:04.069428 25908 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:04.070073 28925 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:04.070462 25908 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:04.072286 25908 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:04.074797 28933 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:09:04.071177 28925 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:09:03.959283 28914 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.091127 28914 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:04.099510 25320 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 13:09:04.103001 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:04.109829 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 170931us
I20260709 13:09:04.073119 28891 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:04.095569 28727 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:04.113596 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:04.058856 28920 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 13:09:04.081001 28890 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:04.126904 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:03.961776 28918 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:04.133610 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:04.134829 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:04.135632 25943 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:09:04.110543 28831 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:04.143810 25942 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 13:09:04.125777 28925 raft_consensus.cc:697] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:04.131774 28791 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:04.163827 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.110786 28940 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260709 13:09:04.166783 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.157037 28564 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:04.140190 28831 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:04.174614 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.129415 28913 consensus_queue.cc:1059] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
I20260709 13:09:04.132922 28797 consensus_queue.cc:1059] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.037s
I20260709 13:09:04.149322 28925 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.178948 25212 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:04.137522 28811 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:04.168818 28795 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:04.171204 28778 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:04.179764 28831 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:04.199321 25943 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:09:04.206218 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:04.210819 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.194840 28734 raft_consensus.cc:3055] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:04.212401 25393 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 13:09:04.221267 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.173087 28791 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:04.179965 28718 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:04.220147 28734 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:04.229055 28801 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:04.229451 28734 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.230497 28734 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:04.188326 28795 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.195938 28800 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:04.233556 28795 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:04.233490 28800 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.192605 28778 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.235884 28778 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:04.236255 28800 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:04.236413 28858 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:04.238354 28718 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:04.240660 28858 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:04.220315 28784 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:09:04.243218 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:04.243588 28784 raft_consensus.cc:697] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:04.243848 25393 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 10
I20260709 13:09:04.244683 28951 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:09:04.196511 28934 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:04.194489 28932 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:04.215576 28785 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:04.257337 28949 log.cc:927] Time spent T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260709 13:09:04.222985 28874 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:04.257445 28785 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:04.259479 28946 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:09:04.259891 28785 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:04.261598 28791 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:04.244488 28784 consensus_queue.cc:237] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.262305 25319 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:04.262576 25394 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:09:04.222950 28911 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:04.264786 28918 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:04.265147 28911 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:04.267402 28911 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:04.254251 28801 raft_consensus.cc:3055] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:04.228430 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.256841 28933 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:04.268747 25389 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:04.267858 28447 consensus_queue.cc:1059] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:09:04.185552 28727 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:04.281154 25911 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:04.282599 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:04.283190 25321 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:04.222618 28948 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:04.268280 28801 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:04.286412 25393 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:04.188199 28811 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:04.288770 28934 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:04.291226 28682 raft_consensus.cc:3055] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:04.291635 28682 raft_consensus.cc:740] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:04.292446 28682 consensus_queue.cc:260] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.292740 28904 consensus_queue.cc:260] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:04.286345 28948 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:04.274575 28858 consensus_queue.cc:1059] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 13:09:04.314570 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:04.315433 28926 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:04.315790 28926 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:04.315972 25393 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:04.316182 28926 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:04.317454 28872 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:04.317966 28872 raft_consensus.cc:697] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:04.318786 28872 consensus_queue.cc:237] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.309024 25321 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:04.322165 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:04.322831 25321 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:09:04.325170 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:04.326849 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:04.328125 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:04.329668 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:04.330300 25242 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:09:04.330835 28944 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:04.331498 28944 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.331915 28447 consensus_queue.cc:1059] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:04.334712 25240 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:04.309407 28932 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:04.324283 28898 consensus_meta.cc:287] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.519s	user 0.007s	sys 0.004s
I20260709 13:09:04.313897 28682 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:04.286988 28831 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.314198 28948 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:04.349946 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 132755us
I20260709 13:09:04.284178 28933 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.291219 28934 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.339807 28904 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:04.340189 28785 raft_consensus.cc:3055] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:04.353922 25212 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:04.335606 28791 consensus_queue.cc:1059] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.020s
I20260709 13:09:04.374428 25212 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:04.350312 28898 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:04.377797 28791 raft_consensus.cc:3055] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:04.323776 28801 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:04.368207 28785 raft_consensus.cc:740] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:04.348577 28932 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.372071 28913 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:04.378327 28791 raft_consensus.cc:740] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:04.397761 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.398641 25940 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 10: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:04.400300 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.401170 28791 consensus_queue.cc:260] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.402413 28791 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:04.404423 25909 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:04.382421 28911 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 13:09:04.383291 28872 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:04.380921 28447 raft_consensus.cc:3055] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:04.383504 28951 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:04.383708 28913 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:09:04.418790 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 44475us
I20260709 13:09:04.410594 28944 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:04.426769 28712 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:04.428416 28712 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.429157 28932 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:04.431289 28712 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:04.432170 28712 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 13:09:04.432610 28966 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 13:09:04.426800 28811 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:04.412003 28943 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:04.417603 28447 raft_consensus.cc:740] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:04.454355 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.423059 28913 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:04.456897 25393 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:04.459651 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.422937 28795 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:04.460742 25393 raft_consensus.cc:2410] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:09:04.462396 28717 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:09:04.462481 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:04.449076 28944 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:04.463501 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.433204 28898 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:04.466565 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:04.470947 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:04.489950 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:04.420899 28872 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:04.493374 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:04.493789 28951 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:04.485854 28795 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.498185 25320 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:09:04.501318 28712 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:04.508026 28712 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.501996 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.501863 28933 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:04.505214 28951 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.506757 28981 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:04.507500 28795 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:04.515142 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 110945us
W20260709 13:09:04.522192 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:04.525997 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.461155 28730 consensus_queue.cc:1059] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.053s
I20260709 13:09:04.528649 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:04.526397 28973 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 13:09:04.437340 28785 consensus_queue.cc:260] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:04.522856 28795 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:04.537739 28795 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.540169 28934 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:04.540613 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.541859 28872 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:04.544761 28785 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:04.539968 28795 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:04.551177 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:04.552341 25909 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:04.448570 28943 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:04.558702 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:04.569921 28969 log.cc:927] Time spent T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 13:09:04.555879 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:04.558393 28872 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:04.531535 28944 consensus_queue.cc:1059] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 13:09:04.592366 28872 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:04.593053 28872 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:04.594296 28712 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:04.593801 28980 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:04.597043 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.599635 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.601379 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.602094 25937 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 9 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:04.527886 28981 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.602454 25941 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: []
I20260709 13:09:04.608937 25941 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:04.613022 28440 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:04.615135 28948 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:04.615535 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.618211 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.620863 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:04.622557 25321 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:09:04.595377 28831 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:04.584405 28934 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:04.627056 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:04.627702 25240 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:09:04.628319 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:04.628721 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:04.628988 25244 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:09:04.633149 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:04.634244 25393 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:04.636439 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:04.637257 28943 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:04.642452 25393 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:04.643494 28981 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:04.645002 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:04.645675 28791 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:04.646185 28791 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.646525 28936 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:04.647064 28936 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.648301 25238 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 13:09:04.649468 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:04.649887 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:04.650080 25242 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
W20260709 13:09:04.650630 28928 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.843s	user 0.003s	sys 0.000s
I20260709 13:09:04.651849 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:04.652716 25241 raft_consensus.cc:2410] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 13:09:04.648240 28791 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:04.654578 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:04.654747 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1928) took 1722 ms. Trace:
I20260709 13:09:04.655066 25243 rpcz_store.cc:276] 0709 13:09:02.932732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:02.932920 (+   188us) service_pool.cc:224] Handling call
0709 13:09:02.933252 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:03.157354 (+224102us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:09:03.157667 (+   313us) log.cc:844] Serialized 14 byte log entry
0709 13:09:03.538625 (+380958us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:03.538688 (+    63us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:03.538733 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:04.653412 (+1114679us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b18002c9ce0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b932a6 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419
0709 13:09:04.654527 (+  1115us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b54001f6dd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:04.654576 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:04.654681 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":4290,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":929,"raft.queue_time_us":7857,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"spinlock_wait_cycles":541992064,"thread_start_us":372827,"threads_started":1,"wal-append.queue_time_us":448100}
I20260709 13:09:04.656579 28936 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:04.657536 25390 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:04.658725 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:04.661764 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:04.662707 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:04.663497 25244 raft_consensus.cc:2393] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:09:04.669140 28979 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 13:09:04.681593 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:04.682433 25241 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:04.685454 28891 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:04.686059 28891 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.691313 28801 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:04.693142 28951 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:04.694309 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:04.695875 25285 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:04.696048 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.697031 28721 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:04.632332 28948 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:04.697594 28721 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.701429 28797 raft_consensus.cc:3055] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:04.701835 28797 raft_consensus.cc:740] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:04.701880 25358 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:04.699725 28721 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:04.702744 28797 consensus_queue.cc:260] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:04.704236 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.705111 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:04.705845 25314 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:09:04.705961 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.706845 28937 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:04.707265 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.702760 28784 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:04.711844 28784 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:04.712057 25908 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:04.712293 28784 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:04.707501 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:04.713553 25321 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:09:04.703760 28797 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:04.688880 28891 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:04.718427 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:04.719904 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:04.730810 25940 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:09:04.745734 25389 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
W20260709 13:09:04.836712 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 140906us
I20260709 13:09:04.838315 25285 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:04.839288 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:04.839913 25241 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:09:04.828980 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:04.840870 25212 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:04.760414 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.812855 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:04.809053 28980 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:04.842816 28937 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:09:04.843130 25312 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:09:04.843088 28980 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.812101 28796 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:04.844688 28796 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.845750 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:04.846323 25241 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:04.847213 28796 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:04.848126 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:04.848688 25243 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:04.855990 25390 raft_consensus.cc:2410] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:09:04.856333 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.846290 25942 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 13:09:04.857168 28872 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.858227 25244 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:09:04.859923 25940 raft_consensus.cc:2393] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 9: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:04.860066 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:04.860319 28872 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:04.708334 28923 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.925s	user 0.004s	sys 0.000s
I20260709 13:09:04.861068 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.778131 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:04.862709 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.858223 25241 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:09:04.863618 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.864112 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.873303 25936 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:04.874440 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.779386 28978 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:04.862097 28980 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:04.865242 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.866129 25243 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:04.871517 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:04.889406 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:04.890177 25393 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:04.870906 25241 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7. Ops: []
W20260709 13:09:04.890877 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:04.881446 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 163466us
I20260709 13:09:04.893021 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:04.871919 25212 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:04.897285 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:04.900384 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:04.901854 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 61548.8us
I20260709 13:09:04.902211 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:04.812402 28717 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:04.903734 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:04.904438 25314 raft_consensus.cc:2368] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7.
I20260709 13:09:04.865325 28981 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:04.867862 28840 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:09:04.899446 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.904458 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.901170 25908 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:04.905921 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.907003 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:04.907284 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:04.907537 25287 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:04.897903 25246 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:09:04.913400 25317 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:09:04.907645 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:04.908748 25241 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 13:09:04.909624 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.913275 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:04.962821 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:04.989042 25320 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:09:04.989229 25939 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:09:04.963826 25312 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:09:04.990244 25312 raft_consensus.cc:606] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:04.990628 25312 raft_consensus.cc:614] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:09:04.992553 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 72458.5us
I20260709 13:09:04.835826 28926 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:04.952104 25393 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
W20260709 13:09:04.997824 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 84636.1us
I20260709 13:09:04.997949 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:04.878278 28914 consensus_queue.cc:1059] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 13:09:04.896517 28978 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:05.003767 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:05.027220 25393 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:05.002736 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:05.004093 25285 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:04.886535 28588 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:05.025422 25317 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:09:04.903653 28785 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:04.908524 28734 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:04.908437 28981 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:04.963184 28777 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:05.039458 25938 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:05.036381 25393 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
W20260709 13:09:05.041344 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:05.090103 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:05.032966 28891 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:05.031230 28588 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:05.031471 28978 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:09:05.068523 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 64507.8us
I20260709 13:09:04.909224 28904 consensus_queue.cc:1059] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.131s
W20260709 13:09:05.068125 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:05.033368 28980 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:09:05.039394 28923 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:09:04.919404 28937 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.124s
I20260709 13:09:04.908099 28717 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:05.056885 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 149437us
I20260709 13:09:05.038354 28557 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:09:05.049226 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 150465us
I20260709 13:09:05.164992 25359 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:05.047844 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1568) took 1499 ms. Trace:
I20260709 13:09:05.087664 28897 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:05.165794 25388 rpcz_store.cc:276] 0709 13:09:03.547253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:03.547992 (+   739us) service_pool.cc:224] Handling call
0709 13:09:03.548298 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:03.548495 (+   197us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:09:03.564049 (+ 15554us) log.cc:844] Serialized 14 byte log entry
0709 13:09:04.632802 (+1068753us) spinlock_profiling.cc:243] Waited 1.36 ms on lock 0x7b1800398bc0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f297685223a 00007f2976cbca2a
0709 13:09:05.037983 (+405181us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:05.038049 (+    66us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:09:05.038132 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:05.046797 (+  8665us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:05.046914 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14542,"prepare.queue_time_us":15378,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":2367,"spinlock_wait_cycles":3125120,"thread_start_us":390232,"threads_started":2,"wal-append.queue_time_us":1428724}
W20260709 13:09:05.110267 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 201565us
I20260709 13:09:05.049510 28981 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:05.035092 28890 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.073s
I20260709 13:09:05.059073 28777 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:09:05.150906 28990 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
W20260709 13:09:05.151211 28991 log.cc:927] Time spent T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 13:09:05.087487 28899 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:09:05.158118 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:05.039544 28942 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.110s
I20260709 13:09:05.222484 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1561) took 1984 ms. Trace:
W20260709 13:09:05.122623 28913 consensus_meta.cc:287] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.656s	user 0.011s	sys 0.000s
I20260709 13:09:05.048213 28785 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:05.150029 28986 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.514s	user 0.000s	sys 0.001s
I20260709 13:09:05.123567 28588 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:05.091523 28734 raft_consensus.cc:697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:05.152755 28980 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:05.223088 25387 rpcz_store.cc:276] 0709 13:09:03.240651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:03.240774 (+   123us) service_pool.cc:224] Handling call
0709 13:09:03.241082 (+   308us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:03.254059 (+ 12977us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:03.254094 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:03.254648 (+   554us) log.cc:844] Serialized 52 byte log entry
0709 13:09:03.662083 (+407435us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:09:03.693833 (+ 31750us) log.cc:844] Serialized 14 byte log entry
0709 13:09:03.694028 (+   195us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:03.694098 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:04.707827 (+1013729us) spinlock_profiling.cc:243] Waited 1 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:09:05.213052 (+505225us) raft_consensus.cc:1685] finished
0709 13:09:05.222246 (+  9194us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:05.222403 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":218,"prepare.queue_time_us":31532,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":71324,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"spinlock_wait_cycles":2305280,"thread_start_us":394659,"threads_started":1,"wal-append.queue_time_us":518433}
I20260709 13:09:05.212671 28777 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:05.152678 28831 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.079s
I20260709 13:09:05.228160 28874 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:05.301704 28874 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:05.303923 28874 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:05.307574 28222 raft_consensus.cc:3055] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:09:05.308174 28222 raft_consensus.cc:740] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:05.250589 28995 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:05.309276 28995 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:05.309326 28222 consensus_queue.cc:260] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:05.310773 28222 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:09:05.311451 28995 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:05.174528 28702 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:05.312408 25390 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: []
I20260709 13:09:05.312341 28702 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:05.313606 25390 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: []
I20260709 13:09:05.314517 25943 raft_consensus.cc:3055] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:05.315016 28702 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:05.261384 28989 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 13:09:05.261566 29000 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:05.318449 29000 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:05.319007 29000 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:05.320166 28717 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:05.262295 28987 log.cc:927] Time spent T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
I20260709 13:09:05.325973 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2185) took 1913 ms. Trace:
I20260709 13:09:05.330858 25394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41830 (request call id 626) took 1110 ms. Trace:
W20260709 13:09:05.263055 28992 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.164s	user 0.000s	sys 0.000s
I20260709 13:09:05.331528 25394 rpcz_store.cc:276] 0709 13:09:04.220820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:04.220993 (+   173us) service_pool.cc:224] Handling call
0709 13:09:05.330642 (+1109649us) spinlock_profiling.cc:243] Waited 40.6 ms on lock 0x7b54001ebc54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:05.330773 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":93487872}
W20260709 13:09:05.263710 28988 log.cc:927] Time spent T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.216s	user 0.001s	sys 0.000s
W20260709 13:09:05.277992 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:05.335047 28962 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.980s	user 0.006s	sys 0.008s
W20260709 13:09:05.339221 28994 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.240s	user 0.002s	sys 0.000s
W20260709 13:09:05.341564 28985 log.cc:927] Time spent T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.242s	user 0.001s	sys 0.000s
I20260709 13:09:05.342545 28734 consensus_queue.cc:237] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:05.291498 28926 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:05.298180 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 529) took 1219 ms. Trace:
I20260709 13:09:05.346318 25315 rpcz_store.cc:276] 0709 13:09:04.078725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:04.078864 (+   139us) service_pool.cc:224] Handling call
0709 13:09:04.080396 (+  1532us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:04.101726 (+ 21330us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:04.101774 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:04.102506 (+   732us) log.cc:844] Serialized 52 byte log entry
0709 13:09:04.180538 (+ 78032us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:09:04.180578 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:05.297768 (+1117190us) spinlock_profiling.cc:243] Waited 9.33 ms on lock 0x7b5400443c54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:05.297814 (+    46us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:05.297859 (+    45us) raft_consensus.cc:1685] finished
0709 13:09:05.297872 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:05.298115 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13742,"raft.queue_time_us":21542,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":1457,"spinlock_wait_cycles":21458304,"thread_start_us":43734,"threads_started":1,"wal-append.queue_time_us":161221,"wal-append.run_cpu_time_us":1758,"wal-append.run_wall_time_us":1001787}
I20260709 13:09:05.348496 28785 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:05.260772 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:05.350260 28898 raft_consensus.cc:3055] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:05.350662 28898 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:05.351450 28898 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:05.352694 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:05.330749 25392 rpcz_store.cc:276] 0709 13:09:03.412225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:03.412351 (+   126us) service_pool.cc:224] Handling call
0709 13:09:03.451871 (+ 39520us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:03.494377 (+ 42506us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:03.494424 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:04.642560 (+1148136us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b2c000e1828. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f29755aba11 00007f29755bb372 00007f2976ceda01 00007f2976cb243a 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2
0709 13:09:04.643057 (+   497us) log.cc:844] Serialized 52 byte log entry
0709 13:09:04.882634 (+239577us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:09:04.908566 (+ 25932us) log.cc:844] Serialized 14 byte log entry
0709 13:09:05.152079 (+243513us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:05.152179 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:05.321238 (+169059us) raft_consensus.cc:1685] finished
0709 13:09:05.325634 (+  4396us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:05.325880 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35395,"prepare.queue_time_us":28240,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":174213,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":32871,"spinlock_wait_cycles":24726784,"thread_start_us":482354,"threads_started":2,"wal-append.queue_time_us":437928}
I20260709 13:09:05.315021 25943 raft_consensus.cc:740] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:05.359426 25943 consensus_queue.cc:260] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:05.360646 25943 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:05.325927 28997 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:05.327297 28995 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:05.361712 28995 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:05.363720 25395 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2180) took 2084 ms (client timeout 1992 ms). Trace:
W20260709 13:09:05.364111 25395 rpcz_store.cc:269] 0709 13:09:03.279127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:03.279254 (+   127us) service_pool.cc:224] Handling call
0709 13:09:05.363534 (+2084280us) spinlock_profiling.cc:243] Waited 739 ms on lock 0x7b5400147b54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:05.363654 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1700538752}
I20260709 13:09:05.364571 28995 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:05.366930 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:05.327945 28953 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:05.369202 28953 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:05.371801 28953 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:05.372237 28976 raft_consensus.cc:3055] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Stepping down as leader of term 5
W20260709 13:09:05.378443 25284 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.691s	user 0.014s	sys 0.031s
W20260709 13:09:05.378871 25284 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.014s	sys 0.031s
W20260709 13:09:05.379132 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:05.165452 28914 raft_consensus.cc:3055] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Stepping down as leader of term 7
W20260709 13:09:05.383343 25313 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.705s	user 0.004s	sys 0.004s
I20260709 13:09:05.232915 28913 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:05.385154 25313 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.005s	sys 0.005s
I20260709 13:09:05.386971 25315 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:09:05.372815 28976 raft_consensus.cc:740] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:09:05.379184 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 692033us
I20260709 13:09:05.146039 28891 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:05.357103 28801 consensus_meta.cc:287] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.665s	user 0.012s	sys 0.000s
I20260709 13:09:05.399150 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:05.401398 25321 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:05.416755 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:05.416917 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:05.380163 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:05.405088 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:05.406366 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:05.419941 25316 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:09:05.406968 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:05.412061 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:05.414126 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:05.361680 28997 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:05.384073 28564 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:05.419297 25314 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:05.426116 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:05.422272 25315 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:09:05.424414 25943 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 7. (term mismatch)
I20260709 13:09:05.427692 25943 pending_rounds.cc:85] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 6
I20260709 13:09:05.428073 25943 pending_rounds.cc:107] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 8.7
I20260709 13:09:05.428417 25943 raft_consensus.cc:2889] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:05.429363 25943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 816) took 1025 ms. Trace:
I20260709 13:09:05.429665 25243 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 7. (term mismatch)
I20260709 13:09:05.430295 25243 pending_rounds.cc:85] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 6
I20260709 13:09:05.430858 25243 pending_rounds.cc:107] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 9.7
I20260709 13:09:05.431205 25243 raft_consensus.cc:2889] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:05.384477 28980 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:05.442415 28997 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:05.442864 28925 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:09:05.443208 28564 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:05.233268 28899 raft_consensus.cc:697] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:05.437476 25240 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:05.439175 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:05.449209 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:05.450403 25358 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:05.429796 25943 rpcz_store.cc:276] 0709 13:09:04.403473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:04.403580 (+   107us) service_pool.cc:224] Handling call
0709 13:09:04.403841 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:05.314385 (+910544us) spinlock_profiling.cc:243] Waited 5.84 ms on lock 0x7b54006d9d20. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976c29dc0 00007f2976cb41c0 00007f2976cb5231 00007f2976cb1d8f 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c
0709 13:09:05.429041 (+114656us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:05.429294 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13432576}
I20260709 13:09:05.384856 28914 raft_consensus.cc:740] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:05.455569 25320 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: []
I20260709 13:09:05.459751 25316 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
W20260709 13:09:05.477102 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:05.497382 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:05.450515 28663 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
W20260709 13:09:05.519706 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:05.514387 28914 consensus_queue.cc:260] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:05.515872 28588 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:05.496878 28990 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 13:09:05.479359 28942 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:05.556459 25321 raft_consensus.cc:2393] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 9: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:05.557410 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:05.505244 28997 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:09:05.570873 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:05.586725 25209 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.065s	sys 0.155s
W20260709 13:09:05.587217 25209 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.065s	sys 0.155s
W20260709 13:09:05.587554 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 657199us
W20260709 13:09:05.617093 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:05.559844 28784 consensus_meta.cc:287] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.846s	user 0.011s	sys 0.001s
W20260709 13:09:05.659850 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 494886us
I20260709 13:09:05.667243 25359 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:05.681222 25287 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:05.618736 28931 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 1.804s	user 0.001s	sys 0.006s
I20260709 13:09:05.711297 28932 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:05.711958 28932 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:05.688172 28937 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:09:05.758104 28999 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 13:09:05.679544 28784 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:05.766732 25391 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
W20260709 13:09:05.774922 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 324578us
I20260709 13:09:05.776034 25358 leader_election.cc:400] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7.
I20260709 13:09:05.776381 25358 leader_election.cc:403] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20260709 13:09:05.756664 28557 consensus_meta.cc:287] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.583s	user 0.005s	sys 0.004s
I20260709 13:09:05.558706 28942 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:05.795620 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:05.792050 25209 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:05.787424 25358 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:05.684226 28981 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:05.773788 28914 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:09:05.820780 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:05.821529 28831 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:05.837447 28913 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:05.838627 28588 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:05.836745 28831 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:05.792025 28564 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:05.832970 28978 consensus_meta.cc:287] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.675s	user 0.004s	sys 0.005s
W20260709 13:09:05.847668 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 60336.1us
I20260709 13:09:05.867230 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:05.821959 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 140896us
I20260709 13:09:05.822371 28892 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:05.878059 25389 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 13:09:05.878574 25389 pending_rounds.cc:85] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 5
I20260709 13:09:05.878867 25389 pending_rounds.cc:107] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 13:09:05.879245 25389 raft_consensus.cc:2889] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:05.822911 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:05.702342 28996 consensus_queue.cc:1059] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.103s
I20260709 13:09:05.761350 28937 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:05.817600 25209 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:05.851326 29000 consensus_meta.cc:287] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.532s	user 0.009s	sys 0.003s
I20260709 13:09:05.826007 25318 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:05.801613 28222 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:05.827888 28911 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:05.800597 28784 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:05.808885 29002 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7.
I20260709 13:09:05.830890 28981 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:05.799495 28801 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:05.683001 28899 consensus_queue.cc:237] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:05.541788 28976 consensus_queue.cc:260] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:05.833637 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:05.867457 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:05.821839 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:05.891551 28937 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:05.868762 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:05.847622 28797 consensus_meta.cc:287] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.093s	user 0.013s	sys 0.000s
I20260709 13:09:05.871276 28717 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:05.872090 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:05.921104 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 49084.8us
I20260709 13:09:05.876085 25387 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:52318
I20260709 13:09:05.926177 25387 raft_consensus.cc:606] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:05.926582 25387 raft_consensus.cc:614] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:09:05.870146 25287 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:05.883306 25240 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
W20260709 13:09:05.890399 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 72885.7us
I20260709 13:09:05.805219 28557 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:05.845302 28942 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:05.843137 28953 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:05.847338 28995 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 13:09:05.903626 25239 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.013s	sys 0.005s
W20260709 13:09:05.932690 25239 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.013s	sys 0.005s
I20260709 13:09:05.910955 25318 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 5. Current term is 6. Ops: []
I20260709 13:09:05.840198 28980 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:05.916514 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:05.862466 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:05.891126 28892 raft_consensus.cc:697] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:05.851788 28978 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:05.899935 28913 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:09:05.917619 25388 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.747s	user 0.059s	sys 0.044s
I20260709 13:09:05.918862 25316 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
W20260709 13:09:05.918260 25319 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2087) took 2058 ms (client timeout 1682 ms). Trace:
W20260709 13:09:05.927697 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 57623.3us
I20260709 13:09:05.929618 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:05.939918 25317 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.011s	sys 0.039s
I20260709 13:09:05.938270 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:05.942261 25937 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.014s	sys 0.038s
W20260709 13:09:05.931008 25313 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.010s	sys 0.011s
W20260709 13:09:05.934444 28777 consensus_meta.cc:287] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.676s	user 0.011s	sys 0.000s
W20260709 13:09:05.944154 25360 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.009s	sys 0.017s
W20260709 13:09:05.944550 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:05.970438 25388 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.799s	user 0.060s	sys 0.045s
W20260709 13:09:05.971664 25313 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.011s	sys 0.012s
I20260709 13:09:05.903391 28222 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:05.972465 25937 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.015s	sys 0.038s
I20260709 13:09:05.973232 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:05.973556 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:05.900629 29000 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:05.947703 25395 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:05.916744 28801 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:05.927639 28976 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:09:05.974010 25241 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:05.984613 25942 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:05.926743 28932 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:05.981423 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:05.987375 25395 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:09:05.973172 25317 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.011s	sys 0.039s
I20260709 13:09:05.945580 25285 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:05.991343 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 45848.2us
I20260709 13:09:05.930979 28784 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:09:05.996942 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:05.975257 25360 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.009s	sys 0.017s
I20260709 13:09:05.977804 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:05.970777 25319 rpcz_store.cc:269] 0709 13:09:03.860296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:03.860424 (+   128us) service_pool.cc:224] Handling call
0709 13:09:05.918021 (+2057597us) spinlock_profiling.cc:243] Waited 384 ms on lock 0x7b5400044a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:05.918170 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":883764864}
I20260709 13:09:05.936107 28890 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:09:05.945367 28717 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:06.003705 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:06.010519 25246 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.014s	sys 0.002s
W20260709 13:09:06.010915 25246 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.015s	sys 0.002s
I20260709 13:09:05.979291 28983 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:05.973587 28702 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:05.981295 28996 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:05.985914 28912 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:06.004657 28913 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:09:05.943548 28814 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.007s	sys 0.004s
I20260709 13:09:05.950594 28942 consensus_queue.cc:1059] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
W20260709 13:09:06.008641 28968 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.545s	user 0.007s	sys 0.032s
I20260709 13:09:05.984009 29005 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:06.007313 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:06.009680 29002 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:05.940032 28953 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.015424 28717 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:05.983601 28826 consensus_queue.cc:1059] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20260709 13:09:06.010421 29004 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:06.049644 25244 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:05.979887 28777 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.013866 28995 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:09:06.036051 25909 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.000s	sys 0.039s
W20260709 13:09:06.054738 25909 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.000s	sys 0.039s
I20260709 13:09:06.034989 25395 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:06.030350 28912 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:06.058044 25938 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 13:09:06.039856 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:06.027959 25241 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:06.031491 28996 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.046618 29002 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.048207 28831 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:06.065249 29002 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:06.065577 28222 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:06.065775 29002 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:06.066313 29002 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.066954 28777 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:06.067613 25945 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:06.035179 28890 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:06.028928 28702 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.068274 28890 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.068837 28980 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:06.045662 28892 consensus_queue.cc:237] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.069751 28702 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:06.070289 28890 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:06.070852 28890 raft_consensus.cc:3055] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:06.056420 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:06.071384 25943 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 9 LEARNER]: Advancing to term 10
I20260709 13:09:06.065600 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 792) took 1165 ms. Trace:
I20260709 13:09:06.073457 25937 rpcz_store.cc:276] 0709 13:09:04.900402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:04.901511 (+  1109us) service_pool.cc:224] Handling call
0709 13:09:04.902047 (+   536us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:04.902777 (+   730us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:04.902804 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:04.903448 (+   644us) log.cc:844] Serialized 26 byte log entry
0709 13:09:05.375714 (+472266us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x7b18003162a0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f29768520d1 00007f2976c6da0f
0709 13:09:05.972850 (+597136us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:05.972895 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:05.974611 (+  1716us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:06.065289 (+ 90678us) raft_consensus.cc:1685] finished
0709 13:09:06.065307 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:06.065533 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30149,"raft.queue_time_us":2698,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":10320,"spinlock_wait_cycles":2558336,"thread_start_us":596948,"threads_started":1,"wal-append.queue_time_us":1072081}
I20260709 13:09:06.054740 29004 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.076942 29004 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:06.077234 28890 raft_consensus.cc:740] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:06.077574 29004 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:06.078008 29004 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.078066 28890 consensus_queue.cc:260] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.079236 28890 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:06.079993 29004 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:06.080493 29004 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:06.082336 28717 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:06.056617 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:06.057380 25212 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:06.085269 28222 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:06.085767 28222 raft_consensus.cc:697] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:06.086589 28222 consensus_queue.cc:237] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.044628 29005 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.062546 28942 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:06.067498 29000 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:06.041237 28968 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.584s	user 0.008s	sys 0.032s
I20260709 13:09:06.089514 28942 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:06.089989 28942 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:06.090205 29005 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:06.091747 29002 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:06.092276 29006 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:06.092794 29006 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.093493 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:06.093969 25943 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 LEARNER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:06.094164 25390 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:06.054414 28995 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.095463 29006 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:06.096244 25212 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:06.096365 28978 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:09:06.061532 28912 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:06.096903 28995 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:06.097383 28911 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:06.097875 28911 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:06.098253 28911 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:06.059592 28826 consensus_queue.cc:1243] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer a2a77ecf2c0941618b5bc79d4c04db61 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 13:09:06.037253 28814 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s	user 0.008s	sys 0.004s
I20260709 13:09:06.103909 28801 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:06.105705 28953 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:06.108521 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:06.080919 29004 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.213693 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:06.215560 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:06.216480 25937 raft_consensus.cc:2393] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 9: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:06.218573 25909 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:06.221782 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 249771us
W20260709 13:09:06.270750 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 52255.7us
W20260709 13:09:06.223070 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:06.229662 25943 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:09:06.234269 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:06.235193 25284 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:06.222151 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 292579us
I20260709 13:09:06.227139 28926 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:06.276407 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 269974us
I20260709 13:09:06.276902 25209 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:06.223568 28998 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:06.223803 28801 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:06.310043 25285 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:06.275859 25390 raft_consensus.cc:2393] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 9: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:06.315022 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:06.332917 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:06.334693 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 57947.7us
I20260709 13:09:06.335712 25209 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:06.234047 28814 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:06.197468 28447 consensus_queue.cc:260] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.348814 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:06.349817 25241 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:09:06.353219 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:06.354801 28913 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:06.355211 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:06.355497 28915 raft_consensus.cc:3055] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:06.355855 28915 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:06.274858 28911 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.356160 25209 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:06.357939 28911 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:06.283209 28734 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:06.278906 28978 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:09:06.374837 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:06.270022 28826 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:06.370610 28438 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:09:06.377159 29011 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 13:09:06.347887 28976 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:06.377359 28911 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:09:06.378579 28913 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:09:06.378785 29022 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:09:06.387444 28734 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.344223 28801 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.407621 29022 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:06.380407 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:06.381801 25238 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:06.383373 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:06.408666 25241 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:06.356600 28915 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.388206 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:06.394716 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:06.277591 28784 raft_consensus.cc:697] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:06.422695 28784 consensus_queue.cc:237] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.375605 28814 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.427356 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2103) took 1019 ms. Trace:
I20260709 13:09:06.386670 28978 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:06.427843 25315 rpcz_store.cc:276] 0709 13:09:05.407530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:05.427285 (+ 19755us) service_pool.cc:224] Handling call
0709 13:09:05.774538 (+347253us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:05.819361 (+ 44823us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:05.819403 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:05.819927 (+   524us) log.cc:844] Serialized 26 byte log entry
0709 13:09:06.118177 (+298250us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:09:06.118235 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:06.118383 (+   148us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:06.426715 (+308332us) raft_consensus.cc:1685] finished
0709 13:09:06.426736 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:06.426882 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44338,"raft.queue_time_us":46369,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"thread_start_us":297880,"threads_started":1,"wal-append.queue_time_us":605315}
I20260709 13:09:06.429333 28978 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:06.429555 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1582) took 1776 ms. Trace:
W20260709 13:09:06.430948 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:06.429983 25388 rpcz_store.cc:276] 0709 13:09:04.652978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:05.167646 (+514668us) service_pool.cc:224] Handling call
0709 13:09:05.167994 (+   348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:05.971340 (+803346us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:05.971372 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:05.971831 (+   459us) log.cc:844] Serialized 26 byte log entry
0709 13:09:06.222643 (+250812us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:06.222681 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:06.222757 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:06.429359 (+206602us) raft_consensus.cc:1685] finished
0709 13:09:06.429389 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:06.429506 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2625,"raft.queue_time_us":153166,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":21365,"thread_start_us":1050362,"threads_started":2,"wal-append.queue_time_us":455973}
I20260709 13:09:06.346905 28926 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.432385 29029 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:06.433068 29029 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.434324 28926 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:06.408242 29022 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:06.347306 28998 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.435783 29022 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.436789 25242 raft_consensus.cc:2410] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
W20260709 13:09:06.389861 28914 consensus_meta.cc:287] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.528s	user 0.009s	sys 0.003s
I20260709 13:09:06.410763 25245 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:09:06.310422 28892 consensus_queue.cc:1059] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.088s
I20260709 13:09:06.413339 28734 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:06.399598 28438 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:06.419742 28840 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 13:09:06.409013 28989 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 13:09:06.402921 28826 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.439604 28996 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:06.362793 28942 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.440387 29008 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:06.441493 29004 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:06.444180 28937 consensus_meta.cc:287] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.513s	user 0.013s	sys 0.007s
W20260709 13:09:06.467322 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:06.468259 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:06.446564 28814 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:06.428444 28912 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.478404 25390 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:06.458648 28438 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:06.439090 28926 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:06.541592 29008 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.449700 28447 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:06.470880 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 235790us
I20260709 13:09:06.445533 29024 consensus_queue.cc:1059] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.117s
I20260709 13:09:06.561012 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:06.495534 28840 raft_consensus.cc:697] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:06.459718 29022 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:06.569839 25284 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:06.559816 25212 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:06.487016 28892 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:06.563014 25245 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
W20260709 13:09:06.577023 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 305686us
I20260709 13:09:06.576884 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:06.580955 25244 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:06.581017 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:06.583696 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:06.584743 25390 raft_consensus.cc:2410] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
W20260709 13:09:06.578718 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:06.589548 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:06.590549 29015 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:06.591107 29015 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.597329 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:06.598683 29009 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.153s	user 0.000s	sys 0.000s
I20260709 13:09:06.598662 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:06.542845 28937 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.570108 29036 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:06.601594 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1804) took 1250 ms. Trace:
I20260709 13:09:06.601536 25284 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:06.615593 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:06.616834 25284 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:06.617300 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:06.618316 25389 raft_consensus.cc:2393] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:06.602264 28890 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:06.603538 29022 consensus_queue.cc:1059] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:09:06.604579 29025 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:09:06.577209 28801 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:06.605590 28982 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:06.606675 28814 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:06.626168 29028 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:06.619781 28588 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:06.627107 28982 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.627552 28912 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:06.628687 29029 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:06.630136 28814 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:09:06.631711 28891 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:06.582351 28734 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:06.634944 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:06.636976 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:06.626197 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:06.637995 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:06.638468 25943 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:09:06.638759 25939 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:06.608461 25287 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:06.609485 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:06.640858 25937 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:06.612193 28796 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:06.646087 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:06.646636 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:06.646865 25937 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:06.647549 25241 raft_consensus.cc:2410] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:09:06.600353 25390 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:06.615250 25246 rpcz_store.cc:276] 0709 13:09:05.351151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:05.351397 (+   246us) service_pool.cc:224] Handling call
0709 13:09:05.351851 (+   454us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:05.352311 (+   460us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:05.352351 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:05.353048 (+   697us) log.cc:844] Serialized 52 byte log entry
0709 13:09:06.011476 (+658428us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:09:06.011528 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:06.011596 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:06.601379 (+589783us) raft_consensus.cc:1685] finished
0709 13:09:06.601398 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:06.601538 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2709,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":102,"thread_start_us":657859,"threads_started":1,"wal-append.queue_time_us":920775}
I20260709 13:09:06.648684 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:06.648806 25386 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:06.621332 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:06.622442 25941 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:09:06.619055 29036 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.641571 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:06.653764 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:06.654649 25242 raft_consensus.cc:2368] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9.
I20260709 13:09:06.655553 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:06.642450 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:06.655857 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:06.657562 25242 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:06.621959 25244 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:06.649767 25944 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 9: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:06.659013 25911 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:06.660557 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 52185.5us
W20260709 13:09:06.664108 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:06.665858 25212 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:06.668531 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:06.668936 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:06.669607 25246 raft_consensus.cc:2393] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 9: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:06.670578 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:06.673386 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:06.674232 25944 raft_consensus.cc:2393] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 9: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:06.570515 28926 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.671450 25389 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:06.679967 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:06.679241 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:06.683486 25394 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:06.684393 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:06.685513 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:06.652931 28734 raft_consensus.cc:697] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:06.600099 28892 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.693035 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:06.674674 29022 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:06.670850 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:06.693015 25314 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 7. Current term is 8. Ops: []
W20260709 13:09:06.698356 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:06.668624 29025 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:06.704010 25390 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:06.704002 25394 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 5.
I20260709 13:09:06.682914 28912 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:06.720141 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:06.723834 25393 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: []
I20260709 13:09:06.726620 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:06.732945 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:06.698168 28890 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:06.667128 28796 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.728397 28912 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:06.751789 28912 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:09:06.859526 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:06.918992 28926 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:06.863619 25286 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.000s	sys 0.022s
W20260709 13:09:06.920363 25286 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.001s	sys 0.023s
I20260709 13:09:06.864907 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:06.866253 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:06.689394 25284 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:06.923290 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 233792us
I20260709 13:09:06.925487 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:06.926587 25317 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:06.926959 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 819) took 2067 ms. Trace:
I20260709 13:09:06.927503 25942 rpcz_store.cc:276] 0709 13:09:04.859399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:04.860768 (+  1369us) service_pool.cc:224] Handling call
0709 13:09:06.926419 (+2065651us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54001b2bd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:06.926711 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2580128000}
I20260709 13:09:06.929589 25392 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:06.930727 25945 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 832) took 1933 ms. Trace:
I20260709 13:09:06.931159 25945 rpcz_store.cc:276] 0709 13:09:04.997469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:05.026193 (+ 28724us) service_pool.cc:224] Handling call
0709 13:09:05.048078 (+ 21885us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:06.068159 (+1020081us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:06.930179 (+862020us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540062e3d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:06.930526 (+   347us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2344303488}
I20260709 13:09:06.932351 28891 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:06.685658 28840 consensus_queue.cc:237] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:06.869946 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:06.871960 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:06.872956 25387 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.000s	sys 0.012s
I20260709 13:09:06.875370 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:06.878839 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:06.878749 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:06.882583 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:06.887372 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:06.890203 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 231092us
I20260709 13:09:06.890903 28898 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:06.726014 29025 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.958690 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:06.959712 29025 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:06.960132 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:06.964795 25316 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:06.968743 25359 leader_election.cc:400] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9.
I20260709 13:09:06.775473 28937 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:06.772312 28982 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:06.781996 28983 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:06.794279 28998 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:06.774619 29015 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:06.769892 29003 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
I20260709 13:09:06.771085 29036 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:06.773494 29008 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:06.770722 28933 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:06.755496 28942 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:06.766229 28971 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:06.786609 28978 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.068331 28971 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.071024 28978 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:07.071249 28971 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:07.072311 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2143) took 2963 ms. Trace:
I20260709 13:09:06.892089 28796 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:07.072698 25313 rpcz_store.cc:276] 0709 13:09:04.109051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:04.666018 (+556967us) service_pool.cc:224] Handling call
0709 13:09:04.666359 (+   341us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:09:04.666749 (+   390us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:04.666775 (+    26us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:09:04.667455 (+   680us) log.cc:844] Serialized 78 byte log entry
0709 13:09:05.330168 (+662713us) spinlock_profiling.cc:243] Waited 4.2 ms on lock 0x7b4800271ed8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297bbdbaa1 00007f297bbd2b37 00007f297bbd29ca 00007f297bcca26f 00007f297bcc9f37 00007f297bcde55c 00007f297bcde508 00007f297bcde488 00007f297bcde443 00007f297bcdd6ec 00007f297ac07a96
0709 13:09:05.395294 (+ 65126us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:09:05.972029 (+576735us) log.cc:844] Serialized 14 byte log entry
0709 13:09:06.220313 (+248284us) log.cc:844] Serialized 14 byte log entry
0709 13:09:06.220695 (+   382us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:06.220789 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:06.568957 (+348168us) spinlock_profiling.cc:243] Waited 7.87 ms on lock 0x7b18003fe020. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:09:07.071795 (+502838us) raft_consensus.cc:1685] finished
0709 13:09:07.071846 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:07.072059 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1463,"prepare.queue_time_us":41574,"prepare.run_cpu_time_us":6444,"prepare.run_wall_time_us":51494,"raft.queue_time_us":51934,"raft.run_cpu_time_us":6656,"raft.run_wall_time_us":670196,"spinlock_wait_cycles":27758336,"thread_start_us":1505325,"threads_started":3,"wal-append.queue_time_us":850279}
I20260709 13:09:07.075217 25389 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:09:07.075379 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:07.076647 25389 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:06.896065 28826 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:06.688055 28734 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:06.742703 28890 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:06.943231 25387 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.001s	sys 0.012s
I20260709 13:09:06.932987 28891 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:06.945839 25911 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:06.946695 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:06.950625 28892 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:06.951323 28953 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:06.952391 25246 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
W20260709 13:09:06.953511 25391 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1584) took 2263 ms (client timeout 2173 ms). Trace:
I20260709 13:09:06.955874 25388 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:09:06.969184 25359 leader_election.cc:403] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 13:09:06.929529 28440 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:09:06.981603 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:06.982445 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:07.000905 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:06.997170 28981 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:07.017577 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:07.036126 28926 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:07.039479 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:07.040946 28438 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.030778 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:07.053084 25316 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:09:07.055740 25317 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:07.057963 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.058336 28858 raft_consensus.cc:3055] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:09:07.064493 29042 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:07.065191 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:07.065544 28983 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:07.085230 28891 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:07.093328 25392 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:07.091892 28858 raft_consensus.cc:740] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:09:07.087119 25391 rpcz_store.cc:269] 0709 13:09:04.676665 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:04.706935 (+ 30270us) service_pool.cc:224] Handling call
0709 13:09:06.924947 (+2218012us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54003d54d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:06.940071 (+ 15124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2426552704}
I20260709 13:09:07.102172 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:07.103230 25238 raft_consensus.cc:2393] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 9: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:07.104122 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:07.098362 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 134100us
I20260709 13:09:07.104976 25238 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 6.
I20260709 13:09:07.105654 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:07.105751 25312 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:07.106899 25945 raft_consensus.cc:2393] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 9: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:07.105559 25241 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 12: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:07.111078 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.111763 28858 consensus_queue.cc:260] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:07.112021 28981 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:07.113478 28858 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:07.113296 28926 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.114634 28953 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.115532 28981 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:07.125468 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:07.126190 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 157996us
I20260709 13:09:07.130347 25318 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:07.138240 28440 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:07.142342 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2212) took 1167 ms. Trace:
W20260709 13:09:07.142450 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:07.142755 25387 rpcz_store.cc:276] 0709 13:09:05.975399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:05.975613 (+   214us) service_pool.cc:224] Handling call
0709 13:09:05.975995 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:05.976242 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:09:06.279040 (+302798us) log.cc:844] Serialized 14 byte log entry
0709 13:09:07.128210 (+849170us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:07.141932 (+ 13722us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:07.142000 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:07.142094 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:07.142271 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":378503,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":11871,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":7772,"spinlock_wait_cycles":36224,"thread_start_us":1086077,"threads_started":2,"wal-append.queue_time_us":548016}
W20260709 13:09:07.145913 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 200343us
I20260709 13:09:07.146899 25911 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:07.150573 28438 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:07.150825 29015 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:07.151649 29015 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:07.155443 28926 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:07.156200 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 46322.1us
I20260709 13:09:07.160612 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:07.161259 25940 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:07.162458 25395 raft_consensus.cc:1217] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Deduplicated request from leader. Original: 6.6->[8.7-8.7]   Dedup: 8.7->[]
I20260709 13:09:07.163007 25395 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
W20260709 13:09:07.162458 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:07.140839 28440 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:07.148695 25911 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:07.157042 25285 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:07.176486 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.181375 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:07.127019 28953 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:07.183452 25315 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:09:07.191009 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.193145 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:07.194629 25246 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:09:07.121903 29042 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:07.200363 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:07.127174 25317 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
W20260709 13:09:07.203235 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 46325.3us
I20260709 13:09:07.213160 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.214167 28872 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:09:07.214934 29036 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:07.216513 28892 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:09:07.216914 28892 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:07.217243 28892 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:07.221237 25317 raft_consensus.cc:606] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:07.221764 25317 raft_consensus.cc:614] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:09:07.222255 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 1949) took 2333 ms. Trace:
I20260709 13:09:07.222642 25243 rpcz_store.cc:276] 0709 13:09:04.888957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:04.889073 (+   116us) service_pool.cc:224] Handling call
0709 13:09:04.889376 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:05.431763 (+542387us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:07.221783 (+1790020us) spinlock_profiling.cc:243] Waited 540 ms on lock 0x7b540028bed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:07.222056 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1241518976}
I20260709 13:09:07.224185 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:07.204528 28447 consensus_meta.cc:287] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.553s	user 0.009s	sys 0.006s
I20260709 13:09:07.225705 28663 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:07.225951 25243 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7. Ops: []
I20260709 13:09:07.226310 28663 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.222959 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.227779 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.231140 28890 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:07.231669 28981 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:07.232420 28981 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:07.232738 29045 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:07.234150 25316 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:07.238760 25238 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:07.242492 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:07.242514 29025 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:07.251842 29025 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:07.253693 29042 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:07.254226 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.257990 25246 raft_consensus.cc:1275] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:09:07.259271 25246 raft_consensus.cc:1240] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:07.255533 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.262944 25316 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:09:07.264209 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.265837 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.266471 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.267339 28899 raft_consensus.cc:3055] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:07.268056 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:07.233456 29045 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:07.268908 25239 raft_consensus.cc:2368] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9.
I20260709 13:09:07.265265 28663 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:07.268074 28899 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:07.270864 28981 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:07.271270 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2104) took 1863 ms. Trace:
I20260709 13:09:07.271792 25320 rpcz_store.cc:276] 0709 13:09:05.407845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:05.441278 (+ 33433us) service_pool.cc:224] Handling call
0709 13:09:05.441566 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:05.473690 (+ 32124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:07.270727 (+1797037us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b540035a7d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:07.271053 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":31763328}
I20260709 13:09:07.272826 25940 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 10: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:07.267038 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:07.281064 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 47270.5us
W20260709 13:09:07.289366 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 140843us
I20260709 13:09:07.290918 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.287925 25312 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:09:07.296164 25315 raft_consensus.cc:2393] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 10: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:07.297538 25313 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
W20260709 13:09:07.301275 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 54014.5us
I20260709 13:09:07.301857 25285 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:07.299504 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.314036 25320 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:07.317848 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.320688 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.323207 25312 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 12: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:07.324549 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.325686 25317 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:09:07.330281 28913 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:07.330940 28913 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:07.331530 29045 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:07.338636 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.274864 28899 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.347441 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.350023 29045 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:07.350617 29045 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:07.351105 29045 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:07.354332 29025 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:07.363763 28983 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.366524 28983 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:07.370837 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:07.370702 28978 consensus_queue.cc:1059] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:07.372761 28978 raft_consensus.cc:3055] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:07.373231 28978 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:07.373558 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:07.374914 28778 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:07.377590 28926 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:07.378364 28926 raft_consensus.cc:697] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:07.379504 28926 consensus_queue.cc:237] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.380621 25911 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:07.382508 28933 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:07.383075 28933 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:07.383453 28933 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:07.386937 25321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2108) took 1971 ms (client timeout 1801 ms). Trace:
W20260709 13:09:07.387409 25321 rpcz_store.cc:269] 0709 13:09:05.415510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:05.416645 (+  1135us) service_pool.cc:224] Handling call
0709 13:09:07.386487 (+1969842us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b54000f9954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:07.386731 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":318580608}
I20260709 13:09:07.357929 28912 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:07.390271 25245 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:09:07.392189 28912 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:07.385084 28913 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:07.385388 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.395097 25320 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:09:07.398108 28976 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:07.398715 28976 raft_consensus.cc:697] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:07.400085 25358 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:07.399803 28976 consensus_queue.cc:237] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:07.400986 28953 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:07.403020 25315 raft_consensus.cc:2393] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 8: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:07.359391 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:07.405622 25314 raft_consensus.cc:3055] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:09:07.382433 25909 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:07.406044 25314 raft_consensus.cc:740] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:07.410130 25241 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:07.415020 25321 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:09:07.420145 25312 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:09:07.421919 25358 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:07.426146 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 66487.1us
I20260709 13:09:07.376613 28978 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.428200 28796 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:07.423024 28899 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:07.436935 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:07.437433 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 155456us
I20260709 13:09:07.439465 28796 raft_consensus.cc:697] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:07.441290 28953 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:07.441769 28953 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:07.442186 28953 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:07.444104 28814 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:07.444471 25314 consensus_queue.cc:260] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.444792 28814 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:07.445883 25314 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:09:07.440922 28796 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.447085 28814 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:07.449563 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1581) took 2783 ms. Trace:
I20260709 13:09:07.449971 25393 rpcz_store.cc:276] 0709 13:09:04.652724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:05.138787 (+486063us) service_pool.cc:224] Handling call
0709 13:09:05.139093 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:06.724424 (+1585331us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:07.412957 (+688533us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b5400282dd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:07.435776 (+ 22819us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3643979136}
I20260709 13:09:07.453492 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:07.456442 25941 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:07.457891 28915 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:07.461287 25359 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:07.456789 28953 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:07.464247 29073 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260709 13:09:07.466472 29029 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:09:07.466919 29029 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:07.467309 29029 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:09:07.468053 28953 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:07.473992 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:07.474244 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:07.474797 25945 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
W20260709 13:09:07.475522 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:07.476333 25359 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:07.477423 28953 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:09:07.478001 28953 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:07.478546 28953 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:07.479980 25139 catalog_manager.cc:5697] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 3 to 9, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 9 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
W20260709 13:09:07.511276 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:07.504922 25139 catalog_manager.cc:5697] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 3 to 7, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 7 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:09:07.513872 28978 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:07.484202 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:07.478755 25941 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 10: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:09:07.497326 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 60522.4us
I20260709 13:09:07.495446 25314 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 13:09:07.516877 25314 pending_rounds.cc:85] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 6
I20260709 13:09:07.481673 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:07.483458 25143 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53264 (request call id 205) took 20412 ms (client timeout 15000 ms). Trace:
I20260709 13:09:07.502442 25287 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:07.522367 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 91516.4us
I20260709 13:09:07.517174 25314 pending_rounds.cc:107] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 13:09:07.525053 25314 raft_consensus.cc:2889] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:09:07.526947 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 67526.4us
I20260709 13:09:07.484874 25285 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:07.531152 28564 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260709 13:09:07.531054 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:07.532328 25945 raft_consensus.cc:2393] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 9: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:09:07.536492 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 51767.3us
I20260709 13:09:07.496140 29036 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9.
I20260709 13:09:07.483667 28891 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.564069 25285 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:07.495324 28953 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:07.541628 28822 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.190s	user 0.018s	sys 0.024s
I20260709 13:09:07.520548 28440 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:07.576452 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:07.584704 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:07.561368 25139 catalog_manager.cc:5697] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
W20260709 13:09:07.577248 28822 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.454s	user 0.019s	sys 0.025s
I20260709 13:09:07.589304 28997 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 13:09:07.565536 28899 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:09:07.521235 28892 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:07.592429 25285 leader_election.cc:400] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9.
I20260709 13:09:07.618140 25312 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:07.618417 25285 leader_election.cc:403] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:09:07.593016 29025 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:07.536047 28840 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:09:07.593364 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 178760us
W20260709 13:09:07.596432 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 67439.4us
W20260709 13:09:07.598680 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:07.591785 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 89559.4us
I20260709 13:09:07.604899 28564 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:07.609457 25316 raft_consensus.cc:2393] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:07.608897 28891 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:07.625299 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:07.600178 25209 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:07.579484 28912 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:07.624267 28564 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.617435 28899 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:07.528370 28933 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.646246 25313 raft_consensus.cc:2368] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7.
I20260709 13:09:07.635303 25139 catalog_manager.cc:5697] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:09:07.648738 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 45182.2us
I20260709 13:09:07.621954 28840 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:07.622160 29025 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:07.662586 25940 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 13:09:07.671974 29047 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9.
I20260709 13:09:07.677762 25285 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:07.678508 28872 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:09:07.679735 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:07.680364 29042 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:07.680572 25243 raft_consensus.cc:2393] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 7: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:09:07.681253 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:07.645669 28899 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:09:07.686410 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:07.651429 28913 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:09:07.625321 28440 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:07.519030 25143 rpcz_store.cc:269] 0709 13:08:47.070785 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:08:47.109239 (+ 38454us) service_pool.cc:224] Handling call
0709 13:09:07.483214 (+20373975us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:08:59.965841 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:08:59.966687 (+   846us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:08:59.966765 (+    78us) write_op.cc:435] Acquired schema lock
0709 13:08:59.966792 (+    27us) tablet.cc:662] Decoding operations
0709 13:08:59.967597 (+   805us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:08:59.967675 (+    78us) write_op.cc:641] Partition lock acquired for write op
0709 13:08:59.967727 (+    52us) tablet.cc:685] Acquiring locks for 4 operations
0709 13:09:00.009570 (+ 41843us) tablet.cc:701] Row locks acquired
0709 13:09:00.018972 (+  9402us) write_op.cc:260] PREPARE: finished
0709 13:09:00.028166 (+  9194us) write_op.cc:270] Start()
0709 13:09:00.028324 (+   158us) write_op.cc:276] Timestamp: P: 1783602540028100 usec, L: 0
0709 13:09:00.028350 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 13:09:00.113779 (+ 85429us) log.cc:844] Serialized 1682 byte log entry
0709 13:09:01.186859 (+1073080us) spinlock_profiling.cc:243] Waited 50.6 ms on lock 0x7b5400470ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cade28 00007f297a6e8581 00007f297a6e7d58 00007f297a6ec9f1 00007f297a6ec9a9 00007f297a6ec939 00007f297a6ec901 00007f297a6ebb2d 000000000038f354 000000000038f199
0709 13:09:01.406016 (+219157us) spinlock_profiling.cc:243] Waited 24.7 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:09:01.530534 (+124518us) op_driver.cc:464] REPLICATION: finished
0709 13:09:02.061293 (+530759us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f2973ba80ac 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297a6e8ef0
0709 13:09:02.464745 (+403452us) write_op.cc:301] APPLY: starting
0709 13:09:02.480909 (+ 16164us) tablet.cc:1367] starting BulkCheckPresence
0709 13:09:02.512146 (+ 31237us) tablet.cc:1370] finished BulkCheckPresence
0709 13:09:02.518079 (+  5933us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:09:02.518854 (+   775us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:09:02.519153 (+   299us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0709 13:09:02.519207 (+    54us) write_op.cc:312] APPLY: finished
0709 13:09:02.520729 (+  1522us) log.cc:844] Serialized 41 byte log entry
0709 13:09:02.636495 (+115766us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:09:02.637961 (+  1466us) write_op.cc:454] Released schema lock
0709 13:09:02.645534 (+  7573us) write_op.cc:341] FINISH: Updating metrics
0709 13:09:02.647232 (+  1698us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:09:02.647269 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1518949,"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":877596,"apply.run_cpu_time_us":41097,"apply.run_wall_time_us":189518,"num_ops":4,"prepare.queue_time_us":214765,"prepare.run_cpu_time_us":18362,"prepare.run_wall_time_us":1221294,"raft.queue_time_us":45461,"raft.run_cpu_time_us":125808,"raft.run_wall_time_us":1021382,"replication_time_us":1451893,"spinlock_wait_cycles":198348288,"thread_start_us":1227131,"threads_started":5,"wal-append.queue_time_us":314650,"wal-append.run_cpu_time_us":11454,"wal-append.run_wall_time_us":3232880}]]}
I20260709 13:09:07.653879 28840 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:07.654532 29025 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:07.680933 29042 raft_consensus.cc:697] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:09:07.697417 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:07.648463 28912 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:07.692657 25139 catalog_manager.cc:5697] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 5 to 6, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 6 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:09:07.702396 25239 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260709 13:09:07.708022 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 143249us
I20260709 13:09:07.715880 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:07.715955 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:07.720662 25242 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:09:07.658257 28891 consensus_queue.cc:1059] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.053s
I20260709 13:09:07.720479 25241 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:09:07.683127 28872 consensus_queue.cc:1059] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 13:09:07.718144 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:07.731583 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:07.731969 25944 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 LEARNER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:09:07.732494 25246 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:07.734176 25940 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 13:09:07.726894 25139 catalog_manager.cc:5697] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 4 to 6, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 6 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:09:07.744180 28976 consensus_queue.cc:1059] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:09:07.744802 25358 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:07.745579 28976 raft_consensus.cc:3055] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:07.745978 28976 raft_consensus.cc:740] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:07.746893 28976 consensus_queue.cc:260] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:07.748203 28976 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:07.750037 25139 catalog_manager.cc:5697] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 6 to 10, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 10 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
W20260709 13:09:07.751473 29075 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260709 13:09:07.751940 25139 catalog_manager.cc:5697] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 5 to 7, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 7 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:09:07.765434 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:07.766271 25388 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:07.769217 28997 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:07.769455 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:07.769742 28997 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:07.770190 25239 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:09:07.770279 28997 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:07.772871 25240 raft_consensus.cc:1217] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Deduplicated request from leader. Original: 6.6->[8.7-8.7]   Dedup: 8.7->[]
I20260709 13:09:07.773479 25240 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 13:09:07.774745 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 630) took 1391 ms. Trace:
I20260709 13:09:07.775451 25908 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:07.720539 29008 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.095s
I20260709 13:09:07.778231 29008 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:09:07.765476 25139 catalog_manager.cc:5697] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 4 to 6, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 6 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:07.779806 29008 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:07.780982 25139 catalog_manager.cc:5697] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 5 to 8, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 8 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:07.781488 29042 consensus_queue.cc:237] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.782598 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:07.783927 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:07.775070 25240 rpcz_store.cc:276] 0709 13:09:06.383098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:06.397760 (+ 14662us) service_pool.cc:224] Handling call
0709 13:09:06.398136 (+   376us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:07.772649 (+1374513us) spinlock_profiling.cc:243] Waited 24.8 ms on lock 0x7b54002af6a0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976c29dc0 00007f2976cb41c0 00007f2976cb5231 00007f2976cb1d8f 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c
0709 13:09:07.773891 (+  1242us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:07.774443 (+   552us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54002d19d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:07.774677 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2839645312}
I20260709 13:09:07.785084 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:07.786337 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:07.787238 25386 raft_consensus.cc:2410] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260709 13:09:07.788394 25239 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:09:07.788548 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:07.788708 28440 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:07.776906 25908 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:07.786053 25139 catalog_manager.cc:5697] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:09:07.784710 25393 raft_consensus.cc:2393] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 9: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:07.784708 28814 raft_consensus.cc:3055] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:07.791028 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:07.792045 25940 raft_consensus.cc:2393] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 11: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:07.799445 25285 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:07.803203 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:07.804054 25242 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:07.769830 28564 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:07.825310 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:07.825814 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:07.817589 25209 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:07.813476 25139 catalog_manager.cc:5697] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:09:07.825816 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:07.829725 25241 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:09:07.793218 28953 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:07.833045 25911 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:07.834131 25285 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:07.794657 28814 raft_consensus.cc:740] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:07.829725 25940 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
W20260709 13:09:07.844049 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 67219.2us
I20260709 13:09:07.811750 28891 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
W20260709 13:09:07.829573 25287 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.173s (SENT) [suppressed 8 similar messages]
I20260709 13:09:07.838478 28447 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:09:07.865535 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 65407.4us
I20260709 13:09:07.840893 28822 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:07.843267 28872 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:09:07.831986 29045 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.859532 28831 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:09:07.854176 29015 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:07.867743 29015 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:07.868173 29015 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:07.865373 25908 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:07.869277 28933 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:07.869570 25139 catalog_manager.cc:5697] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 2 to 5, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 5 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:07.827944 29029 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:07.871240 25139 catalog_manager.cc:5697] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 1 to 7, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 7 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:09:07.872923 29029 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:07.866428 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:07.874277 25209 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:07.875113 28822 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:07.875509 28822 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:07.842618 28892 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:07.875922 28822 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:07.876592 25209 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:07.839159 28564 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:07.856325 28796 raft_consensus.cc:3055] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:07.849622 28440 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:07.865411 28814 consensus_queue.cc:260] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.880162 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:07.872865 25139 catalog_manager.cc:5697] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 5 to 7, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 7 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:07.882953 28913 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:09:07.883790 28976 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:07.903793 28976 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.906919 28976 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:07.907032 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:07.909938 28912 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:07.910359 28913 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:07.909823 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:07.912380 28913 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:09:07.912581 28814 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:07.868882 29045 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:07.912990 28913 raft_consensus.cc:697] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:07.878057 28734 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:07.897920 28840 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.925470 28912 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:07.889400 28440 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:07.877336 28564 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:07.926640 25209 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:07.885583 28914 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:07.928368 25139 catalog_manager.cc:5697] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:07.925881 28913 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.934247 25242 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:09:07.921416 28933 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:09:07.888844 28796 raft_consensus.cc:740] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:09:07.940435 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 247308us
I20260709 13:09:07.877111 29025 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:07.927837 28734 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:09:07.963418 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 218708us
I20260709 13:09:07.885241 29015 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.952316 28564 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:07.952896 28914 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:07.982745 25139 catalog_manager.cc:5697] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:09:07.975464 28915 consensus_meta.cc:287] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.517s	user 0.008s	sys 0.004s
I20260709 13:09:07.981109 25359 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:07.987933 25358 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:07.977239 28734 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:07.935684 28822 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:07.940102 28997 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:08.003564 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 167150us
I20260709 13:09:07.978572 28976 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:09:08.008448 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 81919.4us
I20260709 13:09:07.986624 28914 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:08.011662 25284 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:08.013020 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:07.999289 28926 raft_consensus.cc:670] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } [suppressed 3 similar messages]
I20260709 13:09:07.976783 28912 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:08.005329 25139 catalog_manager.cc:5697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 1 to 7, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 7 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:09:08.061802 25139 catalog_manager.cc:5697] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:08.062713 28912 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:08.063385 25139 catalog_manager.cc:5697] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 6 to 9, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 9 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:09:08.010993 29029 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:08.068681 25284 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:08.020752 28953 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:08.021832 28822 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:08.029874 25393 raft_consensus.cc:2393] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:08.030965 28997 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:08.032780 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:08.045235 28797 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:08.046166 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:08.054615 29042 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:09:08.059343 25394 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:08.020457 28915 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:09:08.071198 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 83350.3us
I20260709 13:09:08.077581 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:08.082888 25941 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 9: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:08.088467 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:08.089098 25242 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:08.075057 28892 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:08.074893 28953 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:08.095182 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:09:08.084703 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 249258us
I20260709 13:09:08.074556 29029 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:08.099241 25358 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:08.099946 28797 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:08.100832 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:08.101284 28912 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:09:08.101569 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:09:08.105738 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:08.106135 25285 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:08.107299 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 277729us
I20260709 13:09:08.107208 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:08.115885 25237 raft_consensus.cc:2368] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7.
I20260709 13:09:08.097239 28953 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:09:08.124748 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:08.131614 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 266365us
W20260709 13:09:08.133689 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:08.104307 28564 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:08.019455 28796 consensus_queue.cc:260] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:08.106606 28997 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:08.095963 28892 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:08.134570 25139 catalog_manager.cc:5697] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:09:08.146798 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 50187.8us
I20260709 13:09:08.117857 29036 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:08.148871 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:08.149511 25389 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:08.150517 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:08.151088 25388 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:08.151402 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:08.152248 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:08.153424 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:08.147243 28892 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:08.153882 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:08.154086 25391 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
W20260709 13:09:08.158329 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 112339us
I20260709 13:09:08.155263 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:08.154520 25237 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:09:08.154969 25139 catalog_manager.cc:5697] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 3 to 8, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 8 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:09:08.164908 25393 raft_consensus.cc:2368] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9.
I20260709 13:09:08.110237 28899 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:08.167672 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:08.171984 25394 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:09:08.141589 29029 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:08.132831 29089 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:09:08.188889 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:08.158638 28872 raft_consensus.cc:697] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:08.192734 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:08.195422 25389 raft_consensus.cc:2393] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:08.151110 25359 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:08.145464 28926 consensus_queue.cc:1059] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
W20260709 13:09:08.204787 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 53748.2us
I20260709 13:09:08.202474 25139 catalog_manager.cc:5697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 7 to 9, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 9 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:09:08.169884 29088 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:09:08.216547 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:08.209862 25390 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:09:08.173849 29091 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:08.189881 28440 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:08.154350 28997 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:08.225308 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:08.225698 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:08.226557 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:08.229022 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:08.203415 29015 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:08.234866 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:08.235505 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:08.236052 25389 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:08.237550 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:08.238349 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:08.215942 29008 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:09:08.274654 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 131284us
I20260709 13:09:08.170250 29036 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:08.223781 29091 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:08.315425 25317 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:08.316722 25312 raft_consensus.cc:2410] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:09:08.311523 25320 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:08.311810 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:08.316948 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:08.317623 25318 raft_consensus.cc:2393] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:08.317759 25316 raft_consensus.cc:2393] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 9: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:09:08.319641 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 126230us
I20260709 13:09:08.320519 25389 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
W20260709 13:09:08.322307 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 113014us
I20260709 13:09:08.281734 25139 catalog_manager.cc:5697] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 6 to 7, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 7 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:08.233541 28840 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:08.327555 25911 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:08.325994 25285 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:08.290196 29008 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:08.331148 25285 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:08.321368 29091 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:08.337061 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:08.336825 28831 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:08.363631 28831 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:08.364045 29029 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:08.364609 28564 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:08.365861 28831 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:08.366879 28914 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:08.367558 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:08.353682 28872 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:08.371254 28899 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:08.337182 29015 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:08.361470 29008 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:08.373364 28797 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:08.374117 29094 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.180s	user 0.000s	sys 0.007s
I20260709 13:09:08.341974 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:08.377666 25312 raft_consensus.cc:2410] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 13:09:08.378851 29015 raft_consensus.cc:697] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:08.378720 28915 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.008s
I20260709 13:09:08.386483 28840 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:08.386976 28840 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:08.387334 28840 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:09:08.388511 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 57452.3us
I20260709 13:09:08.382956 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:08.395061 25312 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7. Ops: []
I20260709 13:09:08.323761 28447 raft_consensus.cc:697] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:08.406114 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:08.406951 25316 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:08.389142 25139 catalog_manager.cc:5697] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 3 to 9. New cstate: current_term: 9 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:09:08.369000 28914 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:08.274989 29088 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:08.409093 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:08.419523 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:08.420394 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:08.424393 28840 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:08.424789 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:09:08.430132 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 217808us
I20260709 13:09:08.401175 28831 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:09:08.445633 25359 leader_election.cc:400] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7.
I20260709 13:09:08.446035 25359 leader_election.cc:403] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 13:09:08.438205 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:08.448282 25240 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:08.387262 29015 consensus_queue.cc:237] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:08.364650 29029 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:08.413228 25139 catalog_manager.cc:5697] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 5 to 8, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 8 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:08.439818 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:08.454524 25316 raft_consensus.cc:2368] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12.
I20260709 13:09:08.461937 25243 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:08.474097 25139 catalog_manager.cc:5697] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:08.477402 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:08.477985 25245 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 10
I20260709 13:09:08.491631 25139 catalog_manager.cc:5697] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 2 to 8, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 8 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:09:08.488683 25358 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:08.497375 28712 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:08.502086 25285 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:08.500442 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 58570.9us
I20260709 13:09:08.478376 28914 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:08.508946 25359 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:08.524868 25908 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:08.528841 25320 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:08.502310 28831 raft_consensus.cc:697] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:08.475674 29029 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:08.550294 29108 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:08.550922 29108 raft_consensus.cc:697] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:08.545871 29109 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:08.551848 29108 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:08.559530 25908 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:08.575088 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:08.574982 29096 consensus_queue.cc:1059] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:08.576454 28891 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:08.576623 29100 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.224s	user 0.001s	sys 0.000s
I20260709 13:09:08.577185 29104 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:08.576999 28891 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:08.577842 29104 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:08.579207 28891 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:08.580051 29104 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:08.580241 28891 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:08.582201 28978 raft_consensus.cc:3055] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:08.582580 28978 raft_consensus.cc:740] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:08.583530 28978 consensus_queue.cc:260] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:08.585228 29061 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:08.585790 29061 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:08.586624 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:08.587885 29061 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:08.561411 29036 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:08.597625 29036 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:08.598186 29036 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
W20260709 13:09:08.599397 29102 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.224s	user 0.001s	sys 0.001s
I20260709 13:09:08.600528 29036 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:08.532313 28914 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:08.563179 28872 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:08.530117 28447 consensus_queue.cc:237] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:08.507954 28712 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:08.601620 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:08.602411 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 83929.2us
I20260709 13:09:08.603425 25242 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:09:08.606549 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:08.626821 25246 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 6.
I20260709 13:09:08.607057 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:08.609666 25390 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:09:08.617066 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:08.632520 25285 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:08.580724 28891 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:08.635639 25940 raft_consensus.cc:2410] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 9 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 13:09:08.565765 28971 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:08.651827 25241 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:08.642375 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:08.603276 28814 consensus_meta.cc:287] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.681s	user 0.005s	sys 0.005s
I20260709 13:09:08.596504 29091 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:08.653872 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 165293us
I20260709 13:09:08.602736 28440 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:08.655849 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:08.657699 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:08.658356 25246 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:09:08.572865 28997 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:08.639783 29047 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 13:09:08.662034 29047 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:09:08.662448 29047 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:08.662827 29047 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:08.665450 28712 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:08.668274 29109 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7.
I20260709 13:09:08.613821 29029 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:08.601111 29036 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:09:08.618568 28564 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:08.615037 28914 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:08.617815 29088 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:08.620685 28840 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:08.656733 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 147837us
I20260709 13:09:08.621598 25944 raft_consensus.cc:2955] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Committing config change with OpId 7.8: config changed from index 2 to 8, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:08.614459 25244 raft_consensus.cc:2955] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Committing config change with OpId 7.8: config changed from index 2 to 8, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:08.654796 25240 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:08.687898 25245 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:08.613004 28892 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:08.607918 28953 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:08.693815 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:08.657727 28440 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:08.694449 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:08.698064 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:08.700623 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:08.700999 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:08.701231 25239 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:09:08.703070 25314 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:08.703873 25241 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
W20260709 13:09:08.705253 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:08.706902 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 130407us
W20260709 13:09:08.695724 28734 consensus_meta.cc:287] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.684s	user 0.008s	sys 0.003s
I20260709 13:09:08.710269 28564 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:08.712119 25393 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:09:08.712692 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:08.735519 25388 raft_consensus.cc:2410] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 13:09:08.602034 28831 consensus_queue.cc:237] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:08.737283 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:08.737792 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 83347.6us
I20260709 13:09:08.713516 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:08.715214 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:08.716447 29103 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260709 13:09:08.716996 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:08.702024 28440 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:08.737911 25391 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:09:08.742403 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:08.717319 28891 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:08.743677 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:08.747294 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:08.747581 25387 raft_consensus.cc:2368] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9.
I20260709 13:09:08.749643 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:08.736304 29029 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:08.750425 25389 raft_consensus.cc:2368] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12.
I20260709 13:09:08.708324 29109 consensus_queue.cc:1059] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.037s
W20260709 13:09:08.766943 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 53494.6us
I20260709 13:09:08.752882 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:08.751940 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:08.757681 25391 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:08.755184 29029 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:08.770830 29029 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:08.773465 29109 raft_consensus.cc:3055] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:08.746899 28712 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:08.773860 29109 raft_consensus.cc:740] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:08.774226 28712 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:08.774705 28712 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:08.774760 29109 consensus_queue.cc:260] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:08.776095 29109 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:08.777802 29091 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:08.779265 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:08.781770 25246 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:08.783630 28892 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:08.610580 29042 raft_consensus.cc:2955] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Committing config change with OpId 7.8: config changed from index 2 to 8, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:08.785477 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:08.718724 29008 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:08.735270 28734 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:08.752000 28914 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:08.796535 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:08.778488 28564 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:08.766594 29025 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:08.746570 29096 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 13:09:08.821971 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:08.822698 25388 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
W20260709 13:09:08.826578 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:08.804857 28914 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:08.829474 28914 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:08.830013 29008 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:08.831053 25237 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:08.832007 28734 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:08.797900 29045 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.000s	sys 0.011s
I20260709 13:09:08.839315 28978 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:08.843294 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:08.844188 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:08.845659 25393 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:09:08.847920 25287 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:08.845323 29045 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.000s	sys 0.012s
I20260709 13:09:08.848703 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:08.855533 29105 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:08.849686 29047 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:08.857772 29047 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:08.860301 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:08.860939 25246 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:08.861464 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:08.862156 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:08.862740 25239 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:08.863617 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:08.863931 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:08.864147 25389 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:08.865069 25209 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:08.869351 25287 leader_election.cc:400] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9.
I20260709 13:09:08.871910 25285 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:08.872880 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:08.873454 25393 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:08.858788 29125 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:08.910809 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 49600.1us
I20260709 13:09:08.911572 25287 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:08.911823 25387 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:09:08.913075 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:08.883600 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:08.919380 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 47541.8us
I20260709 13:09:08.920408 25285 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:08.898911 25246 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
W20260709 13:09:08.929414 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 45375.7us
I20260709 13:09:08.932542 25395 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:09:08.935683 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:08.940047 25239 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:08.942858 25285 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:08.870810 28797 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:09:08.943758 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 78758.5us
I20260709 13:09:08.870981 28734 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:08.885053 29097 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:08.950132 25287 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:08.945505 25285 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:08.945808 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 105780us
I20260709 13:09:08.957405 25393 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:08.958834 25285 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:08.955091 25359 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:08.870544 28440 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:08.945312 28831 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:08.978768 25314 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:09:08.942562 28447 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:08.982056 25387 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:09:08.982980 29122 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:09:09.000782 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:08.967953 28914 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:09.001489 25320 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 6. Current term is 7. Ops: []
I20260709 13:09:08.991009 25314 raft_consensus.cc:606] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:09.001974 29115 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 13:09:09.002410 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:09.003093 25316 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:09:08.946467 28797 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:08.953194 29047 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:08.953724 29045 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:09:08.952183 28734 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:08.953491 29025 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 13:09:09.011756 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 56722.2us
I20260709 13:09:09.015971 25321 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:09:09.016495 25389 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:09.000806 29122 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:09.020411 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:09.015589 28797 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:09.051803 25209 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:08.941965 29125 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:09.042580 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:09.021323 25386 raft_consensus.cc:2368] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9.
I20260709 13:09:09.059370 29120 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:09:09.067600 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 46014.4us
I20260709 13:09:09.079265 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:09.079877 25390 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:09.075100 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:09.017566 29045 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:09.049417 28891 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:09.059150 28734 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:09.049436 28914 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:09.062824 29122 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:09.013489 28447 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:09.089255 25285 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:08.981632 29097 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:09.093189 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 108209us
I20260709 13:09:09.081308 25237 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:09.098054 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:09.098917 25388 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: []
I20260709 13:09:09.099339 25395 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:09:09.100082 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:09.100622 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:09.101342 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:09.101577 25395 raft_consensus.cc:2368] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12.
I20260709 13:09:09.102018 25244 raft_consensus.cc:2368] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12.
I20260709 13:09:09.105701 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:09.107299 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:09.108035 25245 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:09:09.110169 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:09.080293 29025 raft_consensus.cc:697] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:09.119910 28734 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:09.094846 25287 leader_election.cc:400] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12.
I20260709 13:09:09.123294 25287 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:09.124411 25287 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:09.095252 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:09.125599 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:09.126355 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:09.126447 25390 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:09.127063 25387 raft_consensus.cc:2368] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9.
I20260709 13:09:09.127739 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:09.130246 25287 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:09.120955 25237 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:09:09.121819 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:09.070082 29120 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:09.130990 28564 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:09.131556 28892 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:09.127967 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:09.062171 29123 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:09.079144 28814 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:09.078471 29047 raft_consensus.cc:697] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:09.120059 29025 consensus_queue.cc:237] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:09.078775 29108 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:09.084645 29045 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:09.138784 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:09.094358 25389 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:09:09.143978 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:09.153249 25245 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:09.171599 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:09.154744 25359 leader_election.cc:400] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12.
I20260709 13:09:09.173208 25359 leader_election.cc:403] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 13:09:09.171773 25908 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:09.180483 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 50336us
I20260709 13:09:09.180827 25359 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:09.143239 29120 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:09.185873 28971 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:09:09.168339 28831 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:09:09.094521 28914 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:09.168435 29096 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:09.134155 28891 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:09.192132 25287 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:09.192477 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:09.203230 25244 raft_consensus.cc:2393] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 9: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:09.177572 29104 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:09:09.199137 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 147446us
I20260709 13:09:09.176153 28663 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:09.173969 29123 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:09.208763 28440 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:09.191493 25358 connection.cc:256] RPC call timeout handler was delayed by -1.580561031000002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17900000000000002s for a response
I20260709 13:09:09.211519 25209 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:09.211990 25358 connection.cc:256] RPC call timeout handler was delayed by -1.4023610309999412s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1988s for a response
I20260709 13:09:09.170282 28814 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:09.212438 25358 connection.cc:256] RPC call timeout handler was delayed by -1.3792689569999084s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1821s for a response
W20260709 13:09:09.215170 25358 connection.cc:256] RPC call timeout handler was delayed by -1.3689063909999959s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18480000000000002s for a response
W20260709 13:09:09.215576 25358 connection.cc:256] RPC call timeout handler was delayed by -1.3654610309999953s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2029s for a response
W20260709 13:09:09.215842 25358 connection.cc:256] RPC call timeout handler was delayed by -1.3501610309999705s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2046s for a response
W20260709 13:09:09.216207 25358 connection.cc:256] RPC call timeout handler was delayed by -1.3483610309999676s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2048s for a response
W20260709 13:09:09.216601 25358 connection.cc:256] RPC call timeout handler was delayed by -1.2543808670000089s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1636s for a response
W20260709 13:09:09.216897 25358 connection.cc:256] RPC call timeout handler was delayed by -1.2457610310000291s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2162s for a response
W20260709 13:09:09.217221 25358 connection.cc:256] RPC call timeout handler was delayed by -1.2383184220000203s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1965s for a response
W20260709 13:09:09.217571 25358 connection.cc:256] RPC call timeout handler was delayed by -1.184406390999925s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2053s for a response
W20260709 13:09:09.217971 25358 connection.cc:256] RPC call timeout handler was delayed by -1.1501184219999914s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20630000000000004s for a response
W20260709 13:09:09.218339 25358 connection.cc:256] RPC call timeout handler was delayed by -1.1493063909999819s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20920000000000002s for a response
W20260709 13:09:09.218732 25358 connection.cc:256] RPC call timeout handler was delayed by -1.0737063909999733s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21760000000000002s for a response
I20260709 13:09:09.206660 29104 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:09.213223 25287 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:09.190023 29109 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12.
W20260709 13:09:09.219501 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:09.221740 25287 leader_election.cc:400] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12.
I20260709 13:09:09.198737 28734 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:09.181559 29108 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:09.245944 28447 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:09.223832 29117 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:09.220132 28712 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:09.224073 29101 consensus_queue.cc:1059] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 13:09:09.222467 28912 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:09.240588 28814 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:09.209365 29096 raft_consensus.cc:697] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:09.241284 29118 consensus_queue.cc:1059] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.154s
I20260709 13:09:09.241958 29141 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260709 13:09:09.247073 29108 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:09.249053 28953 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:09.223050 25287 leader_election.cc:403] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
W20260709 13:09:09.253647 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 68200.7us
I20260709 13:09:09.209318 28663 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:09.203074 29091 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.023s
W20260709 13:09:09.222158 29087 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 13:09:09.222663 29104 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:09.274062 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:09.222985 28831 consensus_queue.cc:1059] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20260709 13:09:09.275666 28734 raft_consensus.cc:2764] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
W20260709 13:09:09.300465 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 78884.8us
I20260709 13:09:09.315743 25287 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:09.303498 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 87754.5us
I20260709 13:09:09.293588 28814 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:09.282299 28912 raft_consensus.cc:697] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:09.293691 29129 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:09:09.313272 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 93361.2us
I20260709 13:09:09.296593 29117 raft_consensus.cc:697] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:09:09.319607 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 112469us
I20260709 13:09:09.301915 28663 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:09.320329 25209 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:09.299410 29108 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:09.321895 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:09.322185 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:09.323020 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:09.324584 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:09.326967 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:09.311888 29105 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:09.345280 25209 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:09.345729 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:09.327656 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:09.346949 25316 raft_consensus.cc:2393] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:09.263054 29047 consensus_queue.cc:237] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:09.351399 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:09.353543 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:09.353773 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:09.354293 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:09.356000 25312 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
W20260709 13:09:09.377456 29136 log.cc:927] Time spent T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 13:09:09.382149 29140 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:09:09.385437 25209 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:09.388553 28978 consensus_meta.cc:287] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.549s	user 0.005s	sys 0.001s
I20260709 13:09:09.390031 28914 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:09.391631 29042 consensus_meta.cc:287] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.604s	user 0.009s	sys 0.002s
I20260709 13:09:09.390096 29097 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:09.424896 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:09.426894 25387 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:09.352646 25212 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:09.385465 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:09.323969 29125 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:09.393144 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:09.393936 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:09.399258 25395 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:09:09.357982 25319 raft_consensus.cc:2393] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 9: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:09:09.409647 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 82760.9us
I20260709 13:09:09.410988 28891 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:09.429952 28712 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:09.430506 29139 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 0.244s	user 0.000s	sys 0.003s
I20260709 13:09:09.439245 25391 raft_consensus.cc:2410] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
W20260709 13:09:09.439990 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 87451.9us
W20260709 13:09:09.435572 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:09.451318 25358 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:09.411720 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 326413us
I20260709 13:09:09.452301 25321 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:09:09.347712 29108 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:09.411343 29140 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:09.455367 25285 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:09.449250 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 274479us
I20260709 13:09:09.424492 28734 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:09.397843 28797 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:09.302199 29123 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:09.469679 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 84375.2us
I20260709 13:09:09.469790 25285 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:09.458330 29140 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:09.482656 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:09.485998 25944 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:09.496850 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 87958.7us
I20260709 13:09:09.497814 25944 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:09.501952 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:09.502696 25239 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:09:09.474200 29137 log.cc:927] Time spent T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
I20260709 13:09:09.501614 25212 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:09.474541 29092 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.093s
I20260709 13:09:09.473487 28840 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:09.531507 25358 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:09.505390 29095 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:09.417227 29096 consensus_queue.cc:237] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:09.472122 28712 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 13:09:09.510380 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:09.511045 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:09.518625 28831 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:09.511229 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:09.513509 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:09.474066 29147 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:09.504287 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:09.474283 29151 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:09.504263 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:09.474449 29134 raft_consensus.cc:2764] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 13:09:09.538759 28872 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:09.544935 29153 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:09:09.546015 29146 log.cc:927] Time spent T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 13:09:09.470767 29120 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:09.474350 29143 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.202s
I20260709 13:09:09.550694 28797 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:09.551340 28840 raft_consensus.cc:697] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:09.555473 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:09.505419 29149 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:09.456918 28912 consensus_queue.cc:237] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:09.474740 29150 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:09:09.570664 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:09.572238 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:09.572217 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:09.562139 29029 consensus_meta.cc:287] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.790s	user 0.008s	sys 0.004s
I20260709 13:09:09.472297 29157 consensus_queue.cc:1059] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.101s
I20260709 13:09:09.588683 25941 raft_consensus.cc:2393] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:09.586297 29095 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:09.589654 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:09.590297 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:09.599089 25315 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:09:09.629014 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:09.602437 25317 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:09.604084 29138 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:09.605585 29120 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:09.606221 25393 raft_consensus.cc:2393] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:09.463712 29117 consensus_queue.cc:237] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:09.606902 25940 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:09.573580 29147 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:09.551841 28734 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260709 13:09:09.643919 25315 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:09:09.573081 29151 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:09.613970 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:09.591979 28663 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:09.619006 25313 raft_consensus.cc:2393] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 8: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:09.598208 28953 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:09.651501 28953 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:09.651659 29105 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12.
I20260709 13:09:09.651952 28953 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:09.629832 25320 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:09:09.628523 25209 connection.cc:256] RPC call timeout handler was delayed by -2.2038757079999414s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32030000000000003s for a response
I20260709 13:09:09.606881 29153 raft_consensus.cc:697] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:09.636081 25942 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:09.571609 28831 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:09.640321 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:09.646767 25944 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:09.655918 25317 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:09.636974 29138 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:09.649483 29160 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.004s
I20260709 13:09:09.659462 28831 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:09.659353 29153 consensus_queue.cc:237] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:09.660777 29128 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.601s	user 0.001s	sys 0.001s
I20260709 13:09:09.614974 29149 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:09.661273 29157 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:09.661777 29157 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:09.662922 29149 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:09.663924 29157 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:09.666815 25317 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:09.650738 28663 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:09.652408 28712 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:09.643095 29147 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:09.657624 25318 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:09.657308 28913 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:09.667003 29095 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:09.617789 29045 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:09.649847 29161 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.138s	user 0.000s	sys 0.004s
I20260709 13:09:09.593796 29104 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:09.646750 29158 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:09.630048 29029 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:09.674721 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:09.648947 29151 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:09.677650 25285 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:09.668783 29138 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:09.660585 28840 consensus_queue.cc:237] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:09.673614 29108 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:09.687899 29122 consensus_meta.cc:287] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.536s	user 0.011s	sys 0.000s
I20260709 13:09:09.715418 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:09.729799 25317 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:09.729658 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 207416us
I20260709 13:09:09.681916 28712 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:09.748303 25358 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:09.719436 29149 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:09.719094 29123 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:09.758903 25317 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:09.729477 28796 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:09.691462 28913 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:09.770336 25209 connection.cc:256] RPC call timeout handler was delayed by -1.2028486440000279s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1602s for a response
W20260709 13:09:09.772697 25209 connection.cc:256] RPC call timeout handler was delayed by -1.1002486439999757s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1716s for a response
W20260709 13:09:09.813181 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:09.697219 29158 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:09.829941 29155 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.254s	user 0.001s	sys 0.000s
I20260709 13:09:09.831588 28712 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:09.747942 29122 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:09.772892 29134 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:09.837819 25358 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:09.890012 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:09.937760 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:09.956633 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:09.956894 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 132774us
I20260709 13:09:09.775327 29123 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:09.960304 29166 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:09:09.964643 25909 leader_election.cc:336] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 1.701s (SENT)
I20260709 13:09:09.965106 25909 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:09.966516 29097 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:09.969250 25318 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:09.840953 29117 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:09:09.976926 29167 log.cc:927] Time spent T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
I20260709 13:09:09.976898 29029 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:09.977473 28712 consensus_queue.cc:1059] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:09.981597 29140 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:09.991825 28913 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:09.995208 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 166048us
I20260709 13:09:09.999596 28953 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:09.994140 29171 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20260709 13:09:09.970717 29125 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:10.018411 29158 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:10.020496 29129 raft_consensus.cc:3055] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:09:10.020584 29125 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.023965 29122 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:10.026360 28997 raft_consensus.cc:3055] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:10.026774 28997 raft_consensus.cc:740] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:10.020330 28712 raft_consensus.cc:3055] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:10.027350 28712 raft_consensus.cc:740] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:10.027581 28997 consensus_queue.cc:260] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:10.028630 28997 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:10.021098 29129 raft_consensus.cc:740] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:10.031150 29129 consensus_queue.cc:260] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:10.032421 29129 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:10.034765 29140 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:09.992501 29134 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.039485 29134 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:10.031635 28953 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:10.041198 28797 raft_consensus.cc:3055] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:10.041746 28797 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:10.042865 28797 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.047679 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:10.048671 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:10.048755 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:10.052062 25240 raft_consensus.cc:2393] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:10.028270 28712 consensus_queue.cc:260] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.058825 25242 raft_consensus.cc:2393] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 7: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:09:10.062090 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 94174.9us
I20260709 13:09:10.060777 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:10.065237 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:10.065603 25243 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:09:10.065960 25239 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 9: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:09:10.066565 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 54376.4us
I20260709 13:09:10.067232 25244 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
W20260709 13:09:10.067268 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 94947.1us
I20260709 13:09:10.072631 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:10.060199 28797 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:10.060611 29045 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:10.065811 28953 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:10.087569 25944 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:09:10.071756 28978 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:10.083307 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:10.104307 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:10.073853 29104 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:10.080915 29164 log.cc:927] Time spent T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.240s	user 0.001s	sys 0.000s
I20260709 13:09:10.113723 25938 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 8: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:10.105116 25388 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:09:10.116757 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:10.117470 25387 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:09:10.119799 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:10.076834 29088 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:10.118957 25287 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:10.124631 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:10.125415 25392 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:10.085515 28447 raft_consensus.cc:3055] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:09:10.125741 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:10.125463 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:10.117439 28912 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:10.126765 25317 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:09:10.121461 29095 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:10.095366 29125 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:10.130735 25320 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:10.120679 28564 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:10.217825 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:10.099946 28712 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:10.218519 25320 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:09:10.210978 29122 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:10.129807 29149 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:10.130432 28447 raft_consensus.cc:740] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:10.217624 28564 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:10.220893 29122 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.222811 28564 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:10.223666 28447 consensus_queue.cc:260] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.224287 29122 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:10.096124 29138 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:10.226300 29175 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.228s	user 0.003s	sys 0.000s
W20260709 13:09:10.227250 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 110539us
I20260709 13:09:10.131078 25315 raft_consensus.cc:2410] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
W20260709 13:09:10.215319 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:10.214749 25944 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:09:10.204088 25390 raft_consensus.cc:2955] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Committing config change with OpId 7.8: config changed from index 2 to 8, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:09:10.254685 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 171664us
I20260709 13:09:10.258666 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:10.260401 25244 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7. Ops: []
I20260709 13:09:10.261750 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:10.263098 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:10.263782 25244 raft_consensus.cc:2393] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 9: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:10.264695 25244 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:10.265980 25244 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:10.267361 28953 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:10.267921 28953 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.229354 29015 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:10.270524 29134 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:10.271162 29015 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.271307 29134 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:10.273696 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:10.275749 25313 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:09:10.282912 25358 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:10.282852 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 93491.3us
I20260709 13:09:10.285661 25237 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:10.290211 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:10.109371 29151 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:10.117901 28978 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.270596 28953 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:10.301832 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:10.307271 29151 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:10.308233 25316 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:10.308773 25321 raft_consensus.cc:606] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:10.309557 25313 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 13:09:10.284680 28447 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:10.301568 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:10.311054 25239 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 10: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:10.304292 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:10.293967 29147 consensus_meta.cc:287] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.607s	user 0.011s	sys 0.000s
I20260709 13:09:10.323207 25313 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 6. Current term is 7. Ops: []
I20260709 13:09:10.321470 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:10.309326 25321 raft_consensus.cc:614] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:09:10.325275 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:10.327967 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:10.312098 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:10.326352 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:10.252096 29095 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.329991 25358 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:10.292482 29088 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:10.250028 28912 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.333082 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:10.337210 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:10.339531 29095 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:10.336817 29120 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:10.341955 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:10.343638 28912 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:10.335429 29177 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260709 13:09:10.349907 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:10.350556 25390 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:10.341064 29138 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:10.353492 25388 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
W20260709 13:09:10.354375 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:10.356557 25390 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 6. Current term is 7. Ops: []
I20260709 13:09:10.357887 25358 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:10.358446 29089 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:10.361975 29089 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:10.330320 29147 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:10.364872 25944 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20260709 13:09:10.365612 25390 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:09:10.366501 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:10.369314 25944 pending_rounds.cc:85] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 8
I20260709 13:09:10.371498 25944 pending_rounds.cc:107] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260709 13:09:10.371752 25944 raft_consensus.cc:2889] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:10.373730 25911 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:10.377010 28796 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:10.377449 28796 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:10.377827 28796 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:10.378033 28978 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:10.379382 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:10.381182 25942 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:09:10.381230 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:10.382371 25238 raft_consensus.cc:1240] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:10.383132 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:10.383744 29147 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:10.383960 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:10.384505 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:10.384627 25945 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:10.384862 29134 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:10.385927 25386 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2232) took 1674 ms (client timeout 2151 ms). Trace:
I20260709 13:09:10.386570 25940 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:09:10.335836 29015 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:10.386329 25386 rpcz_store.cc:269] 0709 13:09:08.711039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:08.716757 (+  5718us) service_pool.cc:224] Handling call
0709 13:09:10.385724 (+1668967us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b54000dded4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:10.385848 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":635808000}
I20260709 13:09:10.392532 25391 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:10.398140 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:10.400435 25393 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1634) took 1304 ms (client timeout 1563 ms). Trace:
I20260709 13:09:10.415091 25238 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 6. Current term is 7. Ops: []
W20260709 13:09:10.420835 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:10.415655 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:10.421557 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:10.426352 25243 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:10.426421 25246 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:09:10.427245 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:10.427104 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:10.427951 25243 raft_consensus.cc:2393] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:10.427852 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:10.428643 25246 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:10.428580 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:10.429764 29036 consensus_queue.cc:1059] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:09:10.428743 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:10.430613 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:10.403121 28796 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.400940 29120 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:10.432778 25245 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:10.432355 29036 raft_consensus.cc:3055] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:09:10.433456 29036 raft_consensus.cc:740] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:10.434238 29036 consensus_queue.cc:260] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:10.435360 29036 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:10.436718 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:10.437291 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:10.437355 25320 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:09:10.435395 25317 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: []
W20260709 13:09:10.439736 29168 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.450s	user 0.001s	sys 0.002s
I20260709 13:09:10.440604 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:10.440834 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:10.441447 25243 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:10.431034 29154 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:09:10.400892 25393 rpcz_store.cc:269] 0709 13:09:09.096284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:09.097838 (+  1554us) service_pool.cc:224] Handling call
0709 13:09:10.400262 (+1302424us) spinlock_profiling.cc:243] Waited 507 ms on lock 0x7b54001e3f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:10.400381 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1166893440}
W20260709 13:09:10.488973 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:10.450413 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:10.613502 25244 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:09:10.596925 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:10.442178 25245 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:10.565354 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:10.637360 25240 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:10.442623 25238 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:10.614719 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:10.564091 25313 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: [8.10-8.10]
I20260709 13:09:10.546277 25241 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 13:09:10.464164 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:10.612617 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:10.441563 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49638 (request call id 20) took 1012 ms. Trace:
I20260709 13:09:10.562733 25237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1869) took 1044 ms. Trace:
I20260709 13:09:10.408295 25395 raft_consensus.cc:1275] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:09:10.546684 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:10.392832 29088 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:10.554888 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:10.643683 25239 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:10.644088 25908 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:10.637669 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 177307us
I20260709 13:09:10.640126 25321 rpcz_store.cc:276] 0709 13:09:09.429378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:09.429511 (+   133us) service_pool.cc:224] Handling call
0709 13:09:10.441371 (+1011860us) spinlock_profiling.cc:243] Waited 856 ms on lock 0x7b54000f8f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cac853 00007f297bc7a60a 00007f29752007d4 00007f2975200762 00007f2975200691 00007f297520060c 00007f29751ff8c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:10.441484 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1968266624}
W20260709 13:09:10.642426 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 268729us
I20260709 13:09:10.648738 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:10.649520 25318 raft_consensus.cc:2393] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 7: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:10.649338 25911 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:10.645948 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 288125us
I20260709 13:09:10.653571 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:10.660029 25389 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
W20260709 13:09:10.419432 29173 log.cc:927] Time spent T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.402s	user 0.001s	sys 0.001s
I20260709 13:09:10.667392 25908 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:10.667771 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 217455us
I20260709 13:09:10.668787 25287 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:10.658576 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:10.640663 25237 rpcz_store.cc:276] 0709 13:09:09.518467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:09.527831 (+  9364us) service_pool.cc:224] Handling call
0709 13:09:09.538919 (+ 11088us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:10.287048 (+748129us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:10.562403 (+275355us) spinlock_profiling.cc:243] Waited 746 ms on lock 0x7b5400456d54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:10.562633 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1716929280}
I20260709 13:09:10.669878 25393 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:09:10.658677 28914 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:10.671911 28914 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.672710 29162 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:10.673143 29162 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:10.673491 29162 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:10.674185 28914 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:10.403782 29123 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:10.665755 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:10.674471 29105 consensus_queue.cc:1059] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:10.675999 28892 consensus_queue.cc:1059] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:09:10.658634 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:10.677651 25391 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:10.676237 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:10.677758 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:10.680752 25908 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:10.679126 25387 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:10.685743 28971 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:10.687243 28796 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:10.687341 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:10.687469 28971 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:10.687995 28971 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:10.687959 28796 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:10.688616 28971 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:10.547049 28953 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.138s
I20260709 13:09:10.688802 29105 raft_consensus.cc:3055] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:10.689360 29061 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:10.689623 28953 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:10.689719 29105 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:10.690285 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:10.690919 25393 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:10.690810 29105 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.692240 29105 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:10.688313 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:10.693257 29096 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:10.693972 25285 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:10.694439 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:10.695044 25392 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:10.695376 29120 raft_consensus.cc:3055] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:10.695763 29120 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:10.654578 28915 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:10.696532 29152 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:10.697113 29152 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:10.697616 29152 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:09:10.698042 29187 log.cc:927] Time spent T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 13:09:10.696782 28915 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.556190 28891 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.124s
I20260709 13:09:10.711342 25392 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:10.711766 25239 raft_consensus.cc:1275] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:09:10.709167 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:10.702896 28892 raft_consensus.cc:3055] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:10.713967 28892 raft_consensus.cc:740] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:10.714286 28933 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:10.714960 28892 consensus_queue.cc:260] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:10.715498 28933 consensus_queue.cc:1059] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:09:10.716692 29123 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:09:10.721846 25237 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:09:10.718143 28915 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:10.719404 28899 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:09:10.722787 29143 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:09:10.723400 28914 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:10.724817 28914 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.728470 28663 raft_consensus.cc:3055] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:10.728883 28663 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:10.732651 25911 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:10.733675 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:10.734313 25237 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:09:10.735189 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:10.736733 25209 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:10.728960 29123 raft_consensus.cc:697] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:10.737953 29134 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:10.738615 29143 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:09:10.739037 28796 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:10.743918 28892 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:10.728864 28914 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:10.748059 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:10.749184 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:10.750214 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:10.703944 29061 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:10.751302 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:10.752177 25909 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:10.730104 28663 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.764751 28915 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:10.764014 29147 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:09:10.793054 25240 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 13:09:10.797600 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:10.798978 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:10.799382 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 47318.5us
I20260709 13:09:10.759398 25391 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:10.760036 25209 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:10.803668 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 43725.1us
I20260709 13:09:10.758369 25284 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:10.763047 25237 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:10.763325 25239 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:09:10.762039 25359 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:10.804832 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 46624.4us
W20260709 13:09:10.778640 25212 leader_election.cc:336] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.558s (SENT)
I20260709 13:09:10.764225 28814 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:10.770430 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:10.805804 25284 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:10.806142 25212 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:10.807952 25284 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:10.808053 25287 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:10.761435 29129 consensus_queue.cc:1059] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.028s
I20260709 13:09:10.756997 25238 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:10.779440 29061 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:10.787302 29143 raft_consensus.cc:697] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:10.830866 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:10.832672 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 54089.9us
I20260709 13:09:10.769214 29141 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260709 13:09:10.806705 29097 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:10.802301 28796 raft_consensus.cc:697] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:10.827140 28915 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:10.827760 28997 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260709 13:09:10.810005 29147 raft_consensus.cc:697] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:10.825096 28914 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:10.840404 28978 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:10.826094 29154 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:10.847797 28915 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:10.860031 29154 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:10.860165 28915 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:10.860476 29154 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:10.825678 29151 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:09:10.861663 29151 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:10.863958 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:10.870059 29154 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:10.843678 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 178053us
I20260709 13:09:10.843818 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:10.873296 25358 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:10.873687 25940 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:10.875025 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:10.875792 25238 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:10.876034 29154 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:10.876703 29154 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:10.828291 28814 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:10.845726 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 142209us
I20260709 13:09:10.879026 28814 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:10.879722 28814 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:10.880192 28814 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:10.880579 28814 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:10.840670 29163 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.825s	user 0.029s	sys 0.048s
W20260709 13:09:10.881947 29163 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.905s	user 0.029s	sys 0.048s
I20260709 13:09:10.883042 29143 consensus_queue.cc:237] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.885462 29134 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:10.886204 29134 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:10.835259 29091 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260709 13:09:10.894474 28971 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.896667 28971 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:10.899046 25238 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:09:10.838099 29140 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:09:10.840101 29192 raft_consensus.cc:2764] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 13:09:10.900281 28915 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:10.909029 28663 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:10.909189 28915 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:10.910808 25359 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:10.827493 29036 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.024s
I20260709 13:09:10.825989 28912 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:09:10.925515 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:10.888942 29134 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:10.913242 28814 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:10.931718 28912 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:10.851163 29123 consensus_queue.cc:237] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:10.921260 29154 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
W20260709 13:09:10.973052 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 165080us
I20260709 13:09:10.976753 25287 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:10.953222 28997 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:09:10.978519 25287 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:10.983124 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:10.985462 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:10.986335 25391 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:10.993471 29120 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:10.994494 28814 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:10.994902 29120 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:10.997280 25394 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:11.001474 28912 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:10.979743 29141 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:10.941776 28796 consensus_queue.cc:237] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:11.014640 25312 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:09:11.016052 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:11.016616 25312 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:11.018867 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.019593 25393 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:11.025522 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:11.025926 28933 consensus_queue.cc:1059] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:09:11.026204 25318 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:11.026764 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:11.027346 25321 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:10.985248 28914 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:11.028036 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.033875 25394 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:11.033102 25391 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
W20260709 13:09:11.034287 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 230019us
I20260709 13:09:10.981341 29162 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:11.062299 25389 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 13:09:11.062582 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 254682us
I20260709 13:09:11.063686 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:11.064913 25312 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:10.969100 29147 consensus_queue.cc:237] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:11.069505 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.030259 29134 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:11.007645 28997 raft_consensus.cc:697] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:11.066498 29061 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:11.078303 29061 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:11.081343 25212 leader_election.cc:336] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.974s (SENT)
W20260709 13:09:11.080430 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 51305.4us
W20260709 13:09:11.081765 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 273905us
I20260709 13:09:11.057626 29088 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:11.082751 25285 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:11.063244 29097 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:11.096163 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:11.082562 25911 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:11.096999 25388 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:11.090379 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 215133us
I20260709 13:09:11.113818 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:11.127652 25908 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:11.132031 25287 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:11.132128 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:11.133145 25312 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:11.155320 25908 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:11.160730 25312 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:11.166034 25388 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:11.168051 25313 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:09:11.169528 25313 raft_consensus.cc:606] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:11.169859 25313 raft_consensus.cc:614] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:09:11.171540 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 110976us
I20260709 13:09:11.081719 29095 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.070s
I20260709 13:09:11.129887 29096 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:11.149721 29125 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:11.133978 28914 raft_consensus.cc:697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:09:11.191550 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 108867us
I20260709 13:09:11.082270 29134 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:11.197770 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:11.198096 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:11.175225 28971 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:11.173831 29105 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:11.201965 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 45429.3us
I20260709 13:09:11.133369 29088 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:11.200732 29193 log.cc:927] Time spent T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.364s	user 0.004s	sys 0.000s
I20260709 13:09:11.222285 29207 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:11.222787 29091 consensus_queue.cc:1059] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 13:09:11.229266 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:11.224568 29088 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:11.197571 29096 raft_consensus.cc:697] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:11.201172 28814 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:11.200697 28915 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 13:09:11.233487 29207 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:11.234599 29091 raft_consensus.cc:3055] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:11.236403 28914 consensus_queue.cc:237] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:11.202159 28971 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:11.266104 28971 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:11.204129 29129 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:11.190237 29192 consensus_queue.cc:1059] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260709 13:09:11.244802 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:11.273406 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 101204us
I20260709 13:09:11.249642 28892 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:09:11.252158 29089 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:11.254240 29204 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 13:09:11.257614 29200 log.cc:927] Time spent T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 13:09:11.258001 29207 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:11.258937 29134 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:11.271090 28814 raft_consensus.cc:697] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:11.259997 29091 raft_consensus.cc:740] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:11.269241 25936 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:11.248960 29203 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:09:11.192034 28997 consensus_queue.cc:237] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:11.277935 25209 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:11.218058 29105 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:11.316013 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 251046us
I20260709 13:09:11.270751 29129 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:11.258868 25285 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:11.331445 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 72638.2us
I20260709 13:09:11.318984 28912 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:11.328779 25358 leader_election.cc:400] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7.
I20260709 13:09:11.273991 29162 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:11.331676 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 53817.9us
I20260709 13:09:11.335825 25209 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:11.349051 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 217133us
I20260709 13:09:11.352843 25243 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:09:11.355463 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:11.350736 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:11.357820 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:11.359266 25237 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:09:11.359297 25238 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:11.361785 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:11.363577 25909 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:11.368363 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:11.361373 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:11.295394 29206 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20260709 13:09:11.369737 25940 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:09:11.332707 29129 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:11.329250 29207 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:11.348649 29123 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:11.357785 29216 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:11.361876 29138 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:11.358377 29134 raft_consensus.cc:2764] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 13:09:11.398039 29105 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:11.368782 28978 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 13:09:11.408581 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 74393.8us
I20260709 13:09:11.371157 29213 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:11.366154 29201 consensus_queue.cc:1059] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 13:09:11.357846 28912 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:11.421988 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 72152.8us
I20260709 13:09:11.432272 25358 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:11.370975 29191 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260709 13:09:11.413671 29123 raft_consensus.cc:697] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:09:11.446805 29218 log.cc:927] Time spent T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260709 13:09:11.376721 28796 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20260709 13:09:11.405869 28899 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:11.493981 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:11.495684 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:11.503017 25944 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 13:09:11.513800 29223 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 0.097s	user 0.000s	sys 0.006s
W20260709 13:09:11.529052 29224 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 13:09:11.531085 29123 consensus_queue.cc:237] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:11.519578 29120 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:11.534368 29120 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:11.535300 29199 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.459s	user 0.001s	sys 0.002s
I20260709 13:09:11.536526 29120 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:11.538749 28912 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:11.540052 29141 consensus_meta.cc:287] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.531s	user 0.010s	sys 0.001s
I20260709 13:09:11.540469 29141 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:11.428465 28814 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:11.401324 29207 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:11.518558 29201 consensus_queue.cc:1243] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer 2329e428ebbf484fa3ea9452133df567 log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260709 13:09:11.542171 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 110022us
I20260709 13:09:11.542900 25358 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:11.543673 29207 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:11.544761 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:11.560020 25358 leader_election.cc:400] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12.
I20260709 13:09:11.511757 25321 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 13:09:11.511629 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:11.560926 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:11.512059 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:11.512470 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:11.512653 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:11.513262 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:11.366389 29096 consensus_queue.cc:237] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:11.513738 25318 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 13:09:11.514273 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:11.517597 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:11.506971 29152 consensus_meta.cc:287] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.809s	user 0.008s	sys 0.011s
I20260709 13:09:11.566778 25314 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:11.519788 29206 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:11.547431 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:11.509510 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:11.568600 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:11.551877 29151 raft_consensus.cc:3055] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:11.551489 28971 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:11.568323 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 58162.5us
I20260709 13:09:11.570446 25358 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:11.555464 29120 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:09:11.557544 29225 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:09:11.558625 25394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1652) took 1455 ms. Trace:
I20260709 13:09:11.582297 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:11.583693 25317 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:11.549320 29095 raft_consensus.cc:3055] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:11.580572 25394 rpcz_store.cc:276] 0709 13:09:10.102734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:10.341674 (+238940us) service_pool.cc:224] Handling call
0709 13:09:11.558413 (+1216739us) spinlock_profiling.cc:243] Waited 655 ms on lock 0x7b5400481554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:11.558545 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1505702784}
I20260709 13:09:11.433270 29138 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:11.580449 29220 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.072s	user 0.002s	sys 0.000s
I20260709 13:09:11.571233 29195 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:11.589643 29198 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:11.593540 28912 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:11.594518 28971 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:11.594281 28912 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:11.570834 29151 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:11.595762 29151 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:11.596441 28912 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:11.597922 29201 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:09:11.599427 29151 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:11.593640 29222 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 13:09:11.571995 29089 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:11.606410 29089 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:11.606885 29089 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:11.604153 29206 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:09:11.571842 28953 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:11.602139 28912 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 13:09:11.619463 29138 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:11.566205 28899 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:11.638695 29227 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:09:11.639213 28899 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:11.589918 29105 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:11.586709 29095 raft_consensus.cc:740] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:11.639936 29105 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:11.640867 29095 consensus_queue.cc:260] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:11.641355 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 69962.4us
I20260709 13:09:11.641942 29105 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:11.642194 29151 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:09:11.642568 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:11.642879 29129 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:11.570885 29152 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:11.633567 29213 raft_consensus.cc:697] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:11.643752 25287 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:11.643633 29151 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:11.649564 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:11.652725 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:11.653604 25237 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
W20260709 13:09:11.666996 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 82196.5us
I20260709 13:09:11.668076 25209 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:11.642050 29095 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:11.674170 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:11.675127 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:11.677217 25316 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 13:09:11.678364 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:11.680272 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 98097.8us
I20260709 13:09:11.671298 28953 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:09:11.712682 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:11.714632 25391 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260709 13:09:11.715279 25358 connection.cc:256] RPC call timeout handler was delayed by -1.0114975200000345s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17300000000000001s for a response
I20260709 13:09:11.717020 25358 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:11.717267 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.710667 25285 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:11.718111 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:11.718776 29216 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:11.719764 29138 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:11.722527 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:09:11.723431 29138 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:11.716724 28953 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:11.725360 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:11.725826 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:11.726799 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.729156 25393 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:11.730348 25386 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:11.731547 29227 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:11.732023 29227 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:11.682904 29151 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:11.737197 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:11.739971 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:11.740648 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:11.744129 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:11.741351 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:11.745275 25312 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:11.746832 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.746861 25393 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:11.747525 25391 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:09:11.740356 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:11.681602 29238 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:11.678956 28899 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260709 13:09:11.749797 25358 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:11.754041 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:11.754563 25242 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:09:11.733543 29207 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:11.751741 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:11.763095 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:11.752555 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:11.763520 25358 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:11.764042 25241 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:11.751502 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:11.755354 25243 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:11.735584 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:11.765956 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:11.766319 25212 leader_election.cc:336] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.151s (SENT)
I20260709 13:09:11.761411 25394 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:09:11.764930 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:11.766008 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:11.768078 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:11.757043 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:11.766980 25212 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:11.756369 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:11.766315 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:11.769636 25393 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 12
I20260709 13:09:11.770309 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:11.770860 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:11.771493 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:11.769604 25358 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:11.773928 25318 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:11.774775 25318 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:11.760774 29151 raft_consensus.cc:697] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:11.769917 29207 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:11.776321 29207 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:11.776580 29206 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:11.777031 29206 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:11.776695 29151 consensus_queue.cc:237] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:11.777437 29206 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:11.777863 29198 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:11.778302 29198 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:11.778640 29198 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:11.728955 28997 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:11.782971 25209 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:11.786155 25317 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 13:09:11.733811 28814 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:11.716527 29105 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 13:09:11.787166 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.791033 25387 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:09:11.792135 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.792953 25392 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:09:11.797334 25393 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:11.753674 29154 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:11.797946 25314 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:11.798830 25209 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:11.790397 29120 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:11.809507 29120 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:11.768903 29140 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:11.808422 25242 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:11.810429 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:11.781247 28912 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:09:11.837388 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 52412.3us
I20260709 13:09:11.842008 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:11.842608 29248 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:09:11.844413 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:11.855394 29248 raft_consensus.cc:697] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:11.856611 29248 consensus_queue.cc:237] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:11.863071 25936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 849) took 1115 ms. Trace:
I20260709 13:09:11.863533 25936 rpcz_store.cc:276] 0709 13:09:10.747345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:10.757338 (+  9993us) service_pool.cc:224] Handling call
0709 13:09:11.862894 (+1105556us) spinlock_profiling.cc:243] Waited 290 ms on lock 0x7b5400702bd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:11.863011 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":665976064}
I20260709 13:09:11.870636 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.872550 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.787307 29134 consensus_queue.cc:1059] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.014s
I20260709 13:09:11.877146 25312 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:09:11.877377 25394 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20260709 13:09:11.882540 25394 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 13:09:11.809966 29120 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:11.802259 29238 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:11.886543 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.891252 25393 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:11.895366 25395 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:11.897663 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:11.838289 29140 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:11.787020 29230 consensus_queue.cc:1059] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.095s
I20260709 13:09:11.909591 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.910717 25390 raft_consensus.cc:2368] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9.
I20260709 13:09:11.798377 29089 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:11.819509 29221 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:11.838177 29123 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:11.913158 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.914945 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:11.920632 25389 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:11.914324 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.907859 28997 raft_consensus.cc:697] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:11.914916 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.913849 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:11.819852 29162 consensus_queue.cc:1059] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.093s
I20260709 13:09:11.918557 25388 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:09:11.769287 29213 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:11.801529 28814 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:11.908114 29202 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:11.927450 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:11.930015 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.932682 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:11.934353 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 135598us
I20260709 13:09:11.941171 25394 raft_consensus.cc:2376] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11. Re-sending same reply.
I20260709 13:09:11.942418 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:11.943209 25394 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:11.912616 29140 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:11.929265 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.926891 29129 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:11.954840 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.958082 25389 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:11.957331 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.930160 29152 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:11.960234 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:11.960747 25321 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:11.961684 25393 raft_consensus.cc:2368] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12.
I20260709 13:09:11.935525 28953 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:11.955103 29242 log.cc:927] Time spent T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 13:09:11.925032 29221 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:11.970747 25395 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:11.975776 25394 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:11.981597 25390 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:11.930353 29123 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:11.949723 29202 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:11.989810 29248 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.757s
I20260709 13:09:12.746464 28971 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:12.030341 25321 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:12.010663 25316 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: [8.10-8.10]
I20260709 13:09:12.348500 29253 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:12.540335 29246 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:09:12.410284 29198 consensus_meta.cc:287] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.631s	user 0.008s	sys 0.004s
I20260709 13:09:12.001045 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:12.749743 29198 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:12.749229 29246 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:11.996501 29206 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:12.750900 25391 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 13:09:12.751770 28796 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:12.751947 29198 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:12.752610 29206 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:12.752470 28796 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:12.011158 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:09:12.738443 29252 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:12.014134 29089 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:12.756771 29252 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:11.997004 28997 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:12.729516 29088 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:09:11.996654 29236 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
W20260709 13:09:12.079576 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:12.732268 29216 raft_consensus.cc:3055] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Stepping down as leader of term 8
W20260709 13:09:12.013705 29231 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.297s	user 0.000s	sys 0.001s
I20260709 13:09:12.006963 29101 raft_consensus.cc:3055] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:12.759781 29216 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:12.760203 29101 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:11.995674 29207 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:12.761610 25313 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 12. Ops: []
I20260709 13:09:12.761343 29101 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:12.758255 29088 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:12.749006 29253 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:12.762919 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:12.763543 25237 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:12.764788 29253 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:12.765369 25315 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.840s	user 0.005s	sys 0.023s
I20260709 13:09:12.759445 29252 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:12.771368 25237 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
W20260709 13:09:12.755317 25389 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.716s	user 0.006s	sys 0.004s
W20260709 13:09:12.772415 25389 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.006s	sys 0.004s
I20260709 13:09:12.006150 29147 raft_consensus.cc:3055] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:12.760804 29216 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:12.762441 29101 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:12.130895 29143 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:09:12.765903 25315 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.841s	user 0.005s	sys 0.023s
I20260709 13:09:12.769583 28814 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:12.825702 29246 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:12.789268 29147 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:12.826232 29216 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:12.838414 29036 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:12.747788 28971 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:12.859966 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 89653.2us
W20260709 13:09:12.893785 25212 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.025s (SENT) [suppressed 18 similar messages]
W20260709 13:09:12.853924 29120 consensus_meta.cc:287] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.940s	user 0.011s	sys 0.000s
I20260709 13:09:12.817376 29143 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:12.901487 25393 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:09:12.854521 29248 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:12.880515 29253 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:12.879331 29036 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:12.920300 29036 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:12.922017 29091 consensus_queue.cc:260] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:12.923204 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 603) took 1049 ms. Trace:
I20260709 13:09:12.923609 25312 rpcz_store.cc:276] 0709 13:09:11.874042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:11.896443 (+ 22401us) service_pool.cc:224] Handling call
0709 13:09:11.919324 (+ 22881us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:11.935563 (+ 16239us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:11.935601 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:11.955574 (+ 19973us) log.cc:844] Serialized 52 byte log entry
0709 13:09:12.172153 (+216579us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:12.172216 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:12.172288 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:12.922984 (+750696us) raft_consensus.cc:1685] finished
0709 13:09:12.923003 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:12.923141 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1207,"raft.queue_time_us":1451,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":5172,"thread_start_us":174081,"threads_started":1,"wal-append.queue_time_us":949589}
I20260709 13:09:12.926573 28971 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:12.927275 29091 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:12.927984 29276 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:12.928464 29276 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:12.926608 29143 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:12.930282 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:12.930431 29276 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:12.939970 29253 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:12.942375 29253 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:12.943511 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2199) took 1151 ms. Trace:
I20260709 13:09:12.943886 25315 rpcz_store.cc:276] 0709 13:09:11.792064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:11.848825 (+ 56761us) service_pool.cc:224] Handling call
0709 13:09:11.853881 (+  5056us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:11.879716 (+ 25835us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:11.879765 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:11.880335 (+   570us) log.cc:844] Serialized 26 byte log entry
0709 13:09:12.802703 (+922368us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:09:12.802770 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:12.802902 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:12.943314 (+140412us) raft_consensus.cc:1685] finished
0709 13:09:12.943332 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:12.943456 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25265,"raft.queue_time_us":29431,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":20550,"spinlock_wait_cycles":36736,"thread_start_us":877170,"threads_started":1,"wal-append.queue_time_us":1033367}
I20260709 13:09:12.946872 29275 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:09:12.947645 25129 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f75b60146e7d4b9b9954b4fb2c414679 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 46de6919f85341c6a9b574dff112e35a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 8 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:12.949005 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 670) took 1193 ms. Trace:
I20260709 13:09:12.949370 25241 rpcz_store.cc:276] 0709 13:09:11.755570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:11.767795 (+ 12225us) service_pool.cc:224] Handling call
0709 13:09:11.768162 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:11.786242 (+ 18080us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:11.786273 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:11.786313 (+    40us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:11.786343 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:12.948784 (+1162441us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b5400059454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:12.948827 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:12.948939 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41254400}
I20260709 13:09:12.939327 29276 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:12.952602 29276 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:12.931566 29147 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:12.932834 29088 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:12.937803 25246 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:12.937973 29138 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:12.879976 29207 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:12.938449 29248 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:12.947459 29275 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:12.954392 25320 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:12.932122 25237 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:32990
I20260709 13:09:12.965339 25237 raft_consensus.cc:606] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Received request to transfer leadership
I20260709 13:09:12.964160 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:12.964303 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 68567.6us
I20260709 13:09:12.961175 29278 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:12.961180 29147 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:12.976534 29278 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:12.978544 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 714) took 1026 ms. Trace:
I20260709 13:09:12.978598 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2192) took 1242 ms. Trace:
I20260709 13:09:12.907466 29120 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:13.004453 25212 leader_election.cc:336] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.860s (SENT)
I20260709 13:09:13.069332 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:13.079655 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 75251.5us
I20260709 13:09:13.079757 25312 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:13.077742 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1683) took 1283 ms. Trace:
W20260709 13:09:13.077284 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:13.065254 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 60939.4us
I20260709 13:09:13.061940 25909 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:13.048440 25945 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 852) took 2229 ms (client timeout 2025 ms). Trace:
W20260709 13:09:13.083477 25945 rpcz_store.cc:269] 0709 13:09:10.749846 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:10.749974 (+   128us) service_pool.cc:224] Handling call
0709 13:09:12.974252 (+2224278us) spinlock_profiling.cc:243] Waited 551 ms on lock 0x7b5400168854. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:12.979027 (+  4775us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1266154368}
I20260709 13:09:12.998780 25321 rpcz_store.cc:276] 0709 13:09:11.736534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:11.736633 (+    99us) service_pool.cc:224] Handling call
0709 13:09:12.978378 (+1241745us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b5400274f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:12.978518 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":487463680}
W20260709 13:09:13.080613 25212 leader_election.cc:336] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.999s (SENT)
I20260709 13:09:13.089463 25212 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:13.080870 25392 rpcz_store.cc:276] 0709 13:09:11.794812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:11.794970 (+   158us) service_pool.cc:224] Handling call
0709 13:09:11.822367 (+ 27397us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:11.916514 (+ 94147us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:09:12.084672 (+168158us) log.cc:844] Serialized 14 byte log entry
0709 13:09:12.452033 (+367361us) log.cc:844] Serialized 14 byte log entry
0709 13:09:12.521383 (+ 69350us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:12.564902 (+ 43519us) log.cc:844] Serialized 52 byte log entry
0709 13:09:12.565104 (+   202us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:09:12.565185 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:12.565283 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:12.953960 (+388677us) raft_consensus.cc:1685] finished
0709 13:09:12.953986 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:13.077670 (+123684us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":94006,"prepare.queue_time_us":1502934,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":141245,"raft.run_cpu_time_us":472,"raft.run_wall_time_us":154490,"thread_start_us":560743,"threads_started":3,"wal-append.queue_time_us":809002}
I20260709 13:09:12.998374 25389 rpcz_store.cc:276] 0709 13:09:11.952713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:11.972739 (+ 20026us) service_pool.cc:224] Handling call
0709 13:09:11.973215 (+   476us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:11.973480 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:09:12.772794 (+799314us) log.cc:844] Serialized 14 byte log entry
0709 13:09:12.978130 (+205336us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:12.978195 (+    65us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:12.978252 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:12.978333 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:12.978474 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":966004,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":20561,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":14968,"thread_start_us":938191,"threads_started":2}
I20260709 13:09:13.099671 25237 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:32990
I20260709 13:09:13.100307 25237 raft_consensus.cc:606] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Received request to transfer leadership
I20260709 13:09:12.964502 29101 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:12.964216 29138 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:13.065199 29279 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:13.101846 29273 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:09:13.065387 29270 consensus_queue.cc:1059] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
I20260709 13:09:13.093046 29274 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:13.104523 25387 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:13.072518 28796 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:13.152907 29265 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 13:09:13.153239 28796 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:13.077800 29278 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:13.145458 25312 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
W20260709 13:09:13.142818 25212 leader_election.cc:336] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.116s (SENT)
I20260709 13:09:13.149573 29270 raft_consensus.cc:3055] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:13.152642 29276 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:13.147423 29140 consensus_meta.cc:287] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.187s	user 0.013s	sys 0.000s
I20260709 13:09:13.158206 25212 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:13.160431 25908 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:13.151257 29271 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:09:13.169822 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:13.177068 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:13.177524 25386 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:13.178158 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2281) took 1201 ms. Trace:
I20260709 13:09:13.178596 25388 rpcz_store.cc:276] 0709 13:09:11.977170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:13.176699 (+1199529us) service_pool.cc:224] Handling call
0709 13:09:13.178086 (+  1387us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:13.179271 29227 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:13.167994 29143 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:13.181282 25284 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:13.181447 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:13.181686 29283 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:13.182199 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:13.148357 29279 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:13.182878 25284 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:13.158924 29270 raft_consensus.cc:740] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:13.184723 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:13.203528 25238 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 12
I20260709 13:09:13.163872 29278 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:13.162088 28971 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:13.183326 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:13.148311 29101 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:13.182219 25240 raft_consensus.cc:2393] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 9: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:13.206529 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:13.206771 25388 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20260709 13:09:13.205718 25129 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 55588.2us
I20260709 13:09:13.206857 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:13.159657 29154 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.001s	sys 0.024s
I20260709 13:09:13.215166 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:13.213796 25237 raft_consensus.cc:2410] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20260709 13:09:13.217036 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 45023.3us
I20260709 13:09:13.218592 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:13.220177 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:13.220896 25936 raft_consensus.cc:2368] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12.
I20260709 13:09:13.220942 25246 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:09:13.215243 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:13.222294 25386 raft_consensus.cc:2376] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8. Re-sending same reply.
I20260709 13:09:13.223001 25390 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:09:13.223310 29289 consensus_queue.cc:1059] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20260709 13:09:13.211930 29269 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:13.227694 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:13.204140 29152 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:09:13.223613 29256 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 13:09:13.168690 29140 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:13.186312 29143 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:13.228322 25936 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:13.157984 29274 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:13.240733 25246 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:09:13.227916 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:13.241977 25940 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:13.182235 29283 raft_consensus.cc:697] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:09:13.217366 29154 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.002s	sys 0.026s
I20260709 13:09:13.224728 25237 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:13.236174 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:13.239590 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:13.245115 25390 raft_consensus.cc:2368] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9.
I20260709 13:09:13.217756 28997 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:09:13.233991 29141 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.009s	sys 0.008s
I20260709 13:09:13.234357 29286 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:09:13.247802 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 185944us
I20260709 13:09:13.248878 25238 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
W20260709 13:09:13.246943 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 201318us
I20260709 13:09:13.170495 29271 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:13.255172 25242 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:09:13.234899 29248 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:13.278436 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:13.216239 28971 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:13.301687 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 219675us
I20260709 13:09:13.227635 29253 consensus_queue.cc:1059] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20260709 13:09:13.305277 25389 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 13:09:13.314630 29288 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 13:09:13.315673 29138 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:13.303687 29248 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:13.316612 29248 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:13.244129 29143 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:13.227686 29270 consensus_queue.cc:260] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:13.249650 29269 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:13.342419 25245 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:09:13.787495 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:13.787709 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 672) took 2031 ms. Trace:
I20260709 13:09:13.787851 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:13.320680 25320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2150) took 1827 ms (client timeout 1907 ms). Trace:
I20260709 13:09:13.302026 29289 raft_consensus.cc:3055] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:13.788554 25390 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
W20260709 13:09:13.397236 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:13.788641 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:13.555203 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:13.813295 25386 raft_consensus.cc:2376] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7. Re-sending same reply.
I20260709 13:09:13.788163 25243 rpcz_store.cc:276] 0709 13:09:11.756031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:11.776062 (+ 20031us) service_pool.cc:224] Handling call
0709 13:09:11.776320 (+   258us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:11.782544 (+  6224us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:11.782573 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:11.799420 (+ 16847us) log.cc:844] Serialized 26 byte log entry
0709 13:09:11.972235 (+172815us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:11.972301 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:11.972400 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:13.307131 (+1334731us) raft_consensus.cc:1685] finished
0709 13:09:13.307149 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:13.787644 (+480495us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5865,"raft.queue_time_us":6014,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":16267,"thread_start_us":75968,"threads_started":1,"wal-append.queue_time_us":1157727}
I20260709 13:09:13.790891 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:13.824424 25316 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 6.
I20260709 13:09:13.791505 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:13.320374 25388 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 7: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:09:13.364760 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:13.798588 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:13.824242 25320 rpcz_store.cc:269] 0709 13:09:11.493491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:11.493661 (+   170us) service_pool.cc:224] Handling call
0709 13:09:13.320518 (+1826857us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54000627d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:13.320625 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3266087168}
I20260709 13:09:13.714993 25940 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:13.416437 25945 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:13.415007 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:13.830947 25392 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:13.786738 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 526234us
I20260709 13:09:13.297071 29120 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:13.318868 29123 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:13.425957 29290 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20260709 13:09:13.819048 28971 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:13.471302 29140 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:13.318954 29248 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:13.464793 29275 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:13.866014 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2255) took 1995 ms. Trace:
I20260709 13:09:13.866405 25391 rpcz_store.cc:276] 0709 13:09:11.870159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:11.872041 (+  1882us) service_pool.cc:224] Handling call
0709 13:09:11.872490 (+   449us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:12.751248 (+878758us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:13.865753 (+1114505us) spinlock_profiling.cc:243] Waited 878 ms on lock 0x7b5400147b54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:13.865957 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2019612416}
I20260709 13:09:13.867648 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2190) took 2131 ms. Trace:
I20260709 13:09:13.867990 25313 rpcz_store.cc:276] 0709 13:09:11.736018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:11.736124 (+   106us) service_pool.cc:224] Handling call
0709 13:09:11.741003 (+  4879us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:12.761998 (+1020995us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:13.867432 (+1105434us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54000b22d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:13.867583 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2346776832}
I20260709 13:09:13.869413 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2261) took 1980 ms. Trace:
I20260709 13:09:13.415884 29286 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:13.869810 25387 rpcz_store.cc:276] 0709 13:09:11.888663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:11.892533 (+  3870us) service_pool.cc:224] Handling call
0709 13:09:11.892956 (+   423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:13.152059 (+1259103us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:13.869188 (+717129us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54001211d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:13.869354 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2785733888}
I20260709 13:09:13.475539 29274 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:13.889760 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:13.890745 25388 raft_consensus.cc:2393] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:13.796663 29252 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:13.831710 29289 raft_consensus.cc:740] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:13.847635 29270 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:13.840131 29088 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:13.874790 25392 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:13.279425 28997 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:13.838269 29141 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.014s	sys 0.030s
I20260709 13:09:13.848977 29036 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:13.850121 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:13.845990 29154 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:13.854995 29123 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:13.914239 25241 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.014s	sys 0.019s
W20260709 13:09:13.918779 25241 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.015s	sys 0.019s
I20260709 13:09:13.918823 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:13.861708 29120 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:09:13.913368 29141 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.015s	sys 0.031s
I20260709 13:09:13.927134 25246 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:32990
I20260709 13:09:13.910188 29274 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:13.910619 29297 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:13.931375 25389 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:13.471640 29283 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:13.934759 25945 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:13.935168 25246 raft_consensus.cc:606] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Received request to transfer leadership
I20260709 13:09:13.898051 29286 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:13.943836 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:13.944612 25392 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:13.906978 29252 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:13.918347 29036 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:13.913779 29088 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:13.921502 29154 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:13.974068 25321 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:13.965086 28997 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:13.927735 29292 consensus_queue.cc:1059] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260709 13:09:13.965073 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 602) took 2126 ms. Trace:
I20260709 13:09:13.965545 29305 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:13.926295 29123 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:13.942951 29297 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:13.933370 29120 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:13.964740 29036 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:13.981221 25317 rpcz_store.cc:276] 0709 13:09:11.838467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:11.861910 (+ 23443us) service_pool.cc:224] Handling call
0709 13:09:11.862357 (+   447us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:12.003136 (+140779us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:12.003168 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:12.003207 (+    39us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:12.003243 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:13.964840 (+1961597us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b540005ee54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:13.964889 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:13.965011 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":323286272}
I20260709 13:09:13.939688 29274 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:13.998384 29036 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:13.999069 29036 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:14.001556 29036 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:14.002676 29271 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:14.003791 29101 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:14.007072 25246 raft_consensus.cc:3055] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:14.007689 25246 raft_consensus.cc:740] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:14.008797 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1915) took 1078 ms. Trace:
I20260709 13:09:14.009296 25242 rpcz_store.cc:276] 0709 13:09:12.930150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:12.938200 (+  8050us) service_pool.cc:224] Handling call
0709 13:09:12.938513 (+   313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:13.299437 (+360924us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:14.008497 (+709060us) spinlock_profiling.cc:243] Waited 316 ms on lock 0x7b54002d0ad4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:14.008727 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":727564032}
I20260709 13:09:14.010977 29120 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:14.008653 25246 consensus_queue.cc:260] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:14.012539 25246 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:14.014812 29274 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:13.768882 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:09:14.007198 29138 raft_consensus.cc:3055] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:13.985486 29310 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:14.024370 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.026985 25387 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
W20260709 13:09:14.021162 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:14.059607 25244 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:13.940569 29289 consensus_queue.cc:260] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:14.077765 25246 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 5. (term mismatch)
I20260709 13:09:14.078148 25246 pending_rounds.cc:85] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 4
I20260709 13:09:14.078370 25246 pending_rounds.cc:107] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260709 13:09:14.078626 25246 raft_consensus.cc:2889] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:09:14.077291 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 235869us
I20260709 13:09:14.081010 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:14.082933 25358 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:14.083199 25911 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:14.020932 29307 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:14.024370 29201 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:14.022404 29138 raft_consensus.cc:740] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:14.091925 25317 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:09:14.089787 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:14.082309 29143 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:13.990415 29305 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:14.090886 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 187876us
I20260709 13:09:14.084853 29120 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:14.099589 25909 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:14.090164 25358 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:14.102558 25321 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:09:14.106153 29123 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:14.106312 25358 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:14.107944 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:14.111712 29304 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:14.115309 25909 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:14.089273 29319 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:14.090792 29274 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:14.087944 29297 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:14.126545 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.126816 25313 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20260709 13:09:14.130079 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.090621 28796 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:14.088258 29269 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:14.131178 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.131462 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:14.132048 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.132519 25320 raft_consensus.cc:2368] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11.
I20260709 13:09:14.103533 29036 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:14.103801 29101 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:14.100958 29313 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:14.161024 29313 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 13:09:14.161185 29306 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 13:09:14.092784 29201 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:14.165721 29154 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:14.166832 29252 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:14.168876 29286 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:14.155283 29138 consensus_queue.cc:260] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:14.135696 28997 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:14.170317 29138 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:14.120815 28914 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:14.159641 29251 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260709 13:09:14.124780 29129 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:14.164800 29201 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:14.165776 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.171918 25320 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 13:09:14.182499 25315 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:09:14.183271 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1917) took 1252 ms. Trace:
I20260709 13:09:14.175395 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.185581 25317 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:09:14.158530 25312 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:09:14.180644 25321 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:14.131611 29304 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:14.183772 25245 rpcz_store.cc:276] 0709 13:09:12.930732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:12.930845 (+   113us) service_pool.cc:224] Handling call
0709 13:09:12.931110 (+   265us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:13.787852 (+856742us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:14.182977 (+395125us) spinlock_profiling.cc:243] Waited 411 ms on lock 0x7b5400363b54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:14.183200 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":944881536}
I20260709 13:09:14.199952 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.137373 28796 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:14.168804 29305 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:14.202519 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:14.120383 29120 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:14.077453 29310 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:14.091135 29307 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:14.209388 25315 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 13:09:14.222404 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.213852 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.243026 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:14.251936 25321 raft_consensus.cc:2376] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10. Re-sending same reply.
I20260709 13:09:14.254062 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:14.204975 28796 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:14.179569 29036 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:14.137938 29319 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:14.260061 29318 log.cc:927] Time spent T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 13:09:14.261320 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.261881 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.261993 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.262465 29120 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:14.262744 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.209501 29304 raft_consensus.cc:697] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:14.278532 29310 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:14.279265 29201 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:09:14.281400 29251 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 13:09:14.282648 25395 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1688) took 2276 ms (client timeout 2118 ms). Trace:
I20260709 13:09:14.310461 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:14.283506 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.284827 25390 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:09:14.285399 29202 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:14.312299 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:14.286890 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:14.310928 25395 rpcz_store.cc:269] 0709 13:09:12.006638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:14.282381 (+2275743us) service_pool.cc:211] Skipping call since client already timed out
0709 13:09:14.282517 (+   136us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:09:14.289820 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.314381 25394 raft_consensus.cc:2410] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 13:09:14.293761 25945 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 818) took 1075 ms. Trace:
I20260709 13:09:14.294498 25209 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:14.315168 25945 rpcz_store.cc:276] 0709 13:09:13.218221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:13.218370 (+   149us) service_pool.cc:224] Handling call
0709 13:09:14.293545 (+1075175us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b5400014d54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:14.293690 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":453702016}
I20260709 13:09:14.317049 25209 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:14.299535 25391 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:09:14.213156 29088 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:14.201725 29129 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:14.304497 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 198255us
W20260709 13:09:14.324100 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 200383us
I20260709 13:09:14.325184 25358 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:14.325779 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.326396 25945 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:14.325819 25394 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 13:09:14.278031 25315 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 13:09:14.188382 28914 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:14.287966 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.328593 25393 raft_consensus.cc:2410] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20260709 13:09:14.319303 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:14.211576 28997 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:14.326912 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.302124 29305 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:14.334036 29305 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:14.337253 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:14.341418 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.342190 25944 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:14.343084 25316 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:14.306581 29270 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:14.335297 25386 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:14.336074 25945 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 13:09:14.326836 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:14.344184 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.349116 25358 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:14.326551 29271 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:09:14.353561 29221 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:14.358065 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.331259 29088 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:14.330643 29213 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:14.356653 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:14.358703 25942 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:14.354977 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:14.381953 25246 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:14.378741 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.382628 25941 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:14.383901 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:14.382107 25938 raft_consensus.cc:2376] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10. Re-sending same reply.
I20260709 13:09:14.330768 29253 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260709 13:09:14.354084 29312 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:14.384701 25387 raft_consensus.cc:2376] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11. Re-sending same reply.
I20260709 13:09:14.331310 29269 consensus_queue.cc:1059] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 13:09:14.390051 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.357332 29271 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:14.394218 25937 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 13:09:14.342460 28997 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:14.381616 29120 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:14.398808 29271 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:14.356768 29270 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:14.382258 29305 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:14.400516 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.400660 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:14.399878 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:14.426556 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:14.403503 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.464212 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:14.473420 25387 raft_consensus.cc:2410] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 13:09:14.473464 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.476034 25246 raft_consensus.cc:2410] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:09:14.476272 25386 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:09:14.477551 25245 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:09:14.477838 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:14.478145 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:14.478440 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:14.477475 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:14.479151 25242 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:14.480306 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:14.481040 25246 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:09:14.484828 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:14.478868 25239 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:14.390659 29213 raft_consensus.cc:697] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:14.486883 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:14.486099 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 81071.4us
I20260709 13:09:14.489923 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.490877 29213 consensus_queue.cc:237] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:14.398283 29312 raft_consensus.cc:697] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:14.493633 25387 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:09:14.402954 29129 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:14.493973 25209 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:14.487157 29322 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 13:09:14.506791 25358 leader_election.cc:400] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11.
I20260709 13:09:14.507157 25358 leader_election.cc:403] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 13:09:14.507915 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:14.398015 29154 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.100s
I20260709 13:09:14.509668 29216 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.005s
W20260709 13:09:14.510344 29324 log.cc:927] Time spent T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
W20260709 13:09:14.512148 29301 log.cc:927] Time spent T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.575s	user 0.004s	sys 0.000s
I20260709 13:09:14.403438 29304 consensus_queue.cc:237] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:14.512967 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 658) took 2762 ms. Trace:
I20260709 13:09:14.513293 29307 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:14.513460 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:14.513536 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.527842 25393 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:09:14.505456 29201 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:14.514292 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.514935 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.515012 29319 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:14.516634 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.529098 25395 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:09:14.518954 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1934) took 1260 ms. Trace:
I20260709 13:09:14.488523 29323 consensus_queue.cc:1059] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 13:09:14.514148 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.533778 25389 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:14.533926 29323 raft_consensus.cc:3055] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:14.534271 29323 raft_consensus.cc:740] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:14.526950 25244 rpcz_store.cc:276] 0709 13:09:11.750862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:11.751854 (+   992us) service_pool.cc:224] Handling call
0709 13:09:11.782602 (+ 30748us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:14.060258 (+2277656us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:14.512730 (+452472us) spinlock_profiling.cc:243] Waited 2.28 s on lock 0x7b54003eb054. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:14.512909 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5235954432}
I20260709 13:09:14.535293 29323 consensus_queue.cc:260] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:14.494432 29138 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:14.524048 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.529160 25943 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:14.537518 25387 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:09:14.536924 29323 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 12
I20260709 13:09:14.528461 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:14.538030 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.538682 25943 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:14.538980 25358 leader_election.cc:400] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12.
I20260709 13:09:14.539613 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.540318 25943 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:14.541859 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.542860 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:14.543327 25943 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:14.545289 29319 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:14.518044 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.532862 29333 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11.
I20260709 13:09:14.537575 29138 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:14.527262 25390 raft_consensus.cc:2376] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13. Re-sending same reply.
I20260709 13:09:14.546877 25386 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:09:14.546957 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.562623 25940 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:09:14.530135 25238 rpcz_store.cc:276] 0709 13:09:13.259039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:13.261533 (+  2494us) service_pool.cc:224] Handling call
0709 13:09:13.261891 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:13.336441 (+ 74550us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:13.336487 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:13.336545 (+    58us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:09:13.336592 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:14.518792 (+1182200us) spinlock_profiling.cc:243] Waited 74.2 ms on lock 0x7b540034cbd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:14.518825 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:14.518912 (+    87us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":170769792}
W20260709 13:09:14.550357 25358 connection.cc:256] RPC call timeout handler was delayed by -1.0696718379998629s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19870000000000002s for a response
W20260709 13:09:14.553406 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:14.553704 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.513717 29036 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:14.520859 29253 consensus_queue.cc:1059] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260709 13:09:14.548794 28796 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:14.558631 29270 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:14.565761 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.547533 29201 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:14.578275 25390 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:14.538893 28914 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:14.569087 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.592089 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:14.545723 29319 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:14.573552 25285 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:14.577335 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.577518 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.572521 29221 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20260709 13:09:14.603940 25944 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 8. (term mismatch)
I20260709 13:09:14.604393 25944 pending_rounds.cc:85] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 7
I20260709 13:09:14.604606 25944 pending_rounds.cc:107] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 9.8
I20260709 13:09:14.604830 25944 raft_consensus.cc:2889] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:09:14.608419 25212 leader_election.cc:336] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.104s (SENT)
I20260709 13:09:14.609550 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.610598 25945 raft_consensus.cc:2410] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 10 because replica has last-logged OpId of term: 8 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 9.
I20260709 13:09:14.585584 29201 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:14.613361 29312 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:14.570472 29138 raft_consensus.cc:697] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:14.618777 29138 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:14.568320 29141 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20260709 13:09:14.621533 29279 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:14.625269 29319 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:14.627830 29274 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:14.590750 28796 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:14.634681 29269 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:14.635093 29143 raft_consensus.cc:3055] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:14.635273 29269 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:14.637871 25389 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:09:14.638536 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1932) took 1388 ms. Trace:
I20260709 13:09:14.628501 29274 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:14.603708 25940 raft_consensus.cc:1240] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:14.597764 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:14.608976 25212 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:14.635637 29143 raft_consensus.cc:740] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:14.638978 25240 rpcz_store.cc:276] 0709 13:09:13.250051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:13.250207 (+   156us) service_pool.cc:224] Handling call
0709 13:09:13.250499 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:13.850026 (+599527us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:09:14.092730 (+242704us) log.cc:844] Serialized 14 byte log entry
0709 13:09:14.123035 (+ 30305us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:14.123069 (+    34us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:09:14.123110 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:14.638312 (+515202us) spinlock_profiling.cc:243] Waited 599 ms on lock 0x7b5400364054. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:14.638355 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:14.638471 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19669,"prepare.queue_time_us":146236,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":56,"raft.queue_time_us":4925,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":13086,"spinlock_wait_cycles":1378291968,"thread_start_us":252756,"threads_started":2,"wal-append.queue_time_us":66593}
W20260709 13:09:14.645591 25212 connection.cc:256] RPC call timeout handler was delayed by -1.4073795760000394s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1728s for a response
W20260709 13:09:14.645244 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 48858.5us
I20260709 13:09:14.651407 25284 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:14.654671 25212 connection.cc:256] RPC call timeout handler was delayed by -1.2408795759999975s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19130000000000003s for a response
W20260709 13:09:14.664209 25212 connection.cc:256] RPC call timeout handler was delayed by -1.2229175290000285s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2036s for a response
I20260709 13:09:14.676329 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:14.678157 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 170287us
I20260709 13:09:14.679247 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:14.689463 25943 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:14.691358 28997 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:14.696540 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:14.697784 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:14.701061 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:14.702064 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 54565.6us
I20260709 13:09:14.705785 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.702471 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.711931 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:14.712037 25940 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:14.713038 25945 raft_consensus.cc:2368] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11.
I20260709 13:09:14.663116 29273 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:14.662923 29129 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:14.729167 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.717928 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.720036 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.731642 25315 raft_consensus.cc:2393] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:14.722035 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.734428 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.723456 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.736799 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:14.738615 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 174139us
I20260709 13:09:14.723896 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:14.778635 25358 connection.cc:256] RPC call timeout handler was delayed by -1.0381718379999256s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2022s for a response
I20260709 13:09:14.779011 25936 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:14.780076 25317 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 13:09:14.780026 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:14.781227 25319 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:09:14.777894 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 126652us
I20260709 13:09:14.782198 25284 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:14.689033 29274 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:09:14.715273 29292 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:14.670822 29221 consensus_queue.cc:1059] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20260709 13:09:14.785017 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:15.236387 25386 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:14.881162 29323 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:15.216018 29251 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:15.036371 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 334509us
I20260709 13:09:14.837751 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:15.239223 25141 catalog_manager.cc:5697] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:09:15.240509 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:15.005251 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:14.784686 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.815548 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:15.241616 25391 raft_consensus.cc:2368] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12.
W20260709 13:09:14.842113 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:14.915392 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:14.816208 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:15.243420 25389 raft_consensus.cc:2410] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:09:15.243567 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:15.245623 25240 raft_consensus.cc:1275] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:09:15.237067 29323 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:15.247001 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:14.849705 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:14.888888 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:14.815948 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:14.796429 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 100261us
I20260709 13:09:14.816892 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:15.256107 25394 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
W20260709 13:09:15.262547 29252 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.002s	sys 0.010s
I20260709 13:09:15.263360 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:15.260911 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:15.264202 25321 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:09:15.235255 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:15.265900 25391 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:09:15.266279 25392 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:14.982985 29319 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:15.267278 29129 raft_consensus.cc:697] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:09:15.268352 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 486239us
I20260709 13:09:15.268762 29129 consensus_queue.cc:237] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:15.269178 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:15.260321 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:15.269097 25284 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:15.257602 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:15.262212 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:15.272923 25313 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:15.257082 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:15.273474 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:15.273238 25315 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
W20260709 13:09:15.272161 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 492377us
W20260709 13:09:15.060667 29336 log.cc:927] Time spent T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.099s	user 0.002s	sys 0.000s
I20260709 13:09:14.790103 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:15.062729 29253 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:15.277882 29253 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:15.280671 29253 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:15.281505 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:15.287880 25393 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:15.243181 25141 catalog_manager.cc:5697] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 6 to 9, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 9 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:15.288846 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:15.293100 25284 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:15.259742 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:15.285921 25908 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:14.796097 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:15.306872 25209 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.009s	sys 0.009s
W20260709 13:09:15.307571 25909 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.004s	sys 0.014s
W20260709 13:09:15.307950 25909 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.004s	sys 0.014s
W20260709 13:09:15.034793 29271 consensus_meta.cc:287] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.539s	user 0.007s	sys 0.003s
W20260709 13:09:15.309468 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 631273us
I20260709 13:09:15.330634 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:15.331377 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:15.331691 25238 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
W20260709 13:09:15.007148 29334 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.231s	user 0.000s	sys 0.001s
I20260709 13:09:14.781445 29269 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:15.339933 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:14.862315 29273 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:09:15.309237 25209 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.009s	sys 0.009s
W20260709 13:09:15.309657 29213 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.006s	sys 0.018s
W20260709 13:09:15.323397 25942 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.003s	sys 0.013s
W20260709 13:09:15.346349 25942 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.003s	sys 0.013s
I20260709 13:09:15.272310 29289 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:14.819437 29270 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:14.881717 29337 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:14.728941 28971 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.564s
W20260709 13:09:15.366008 25139 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39572 (request call id 290) took 12485 ms (client timeout 15000 ms). Trace:
I20260709 13:09:14.796584 28997 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:15.269261 29310 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:14.960366 29201 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:15.323529 25388 raft_consensus.cc:2393] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:09:15.342743 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:15.053733 29298 raft_consensus.cc:3055] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 13:09:15.345292 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 666068us
I20260709 13:09:14.738775 29286 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.585s
I20260709 13:09:15.349223 25908 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:15.351284 25909 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:15.287389 28914 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:14.819725 29320 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.544s
I20260709 13:09:14.781759 29295 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.556s
W20260709 13:09:15.366580 25139 rpcz_store.cc:269] 0709 13:09:02.872071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:02.884920 (+ 12849us) service_pool.cc:224] Handling call
0709 13:09:15.356965 (+12472045us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:09:08.900609 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:09:08.901580 (+   971us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:09:08.901658 (+    78us) write_op.cc:435] Acquired schema lock
0709 13:09:08.901687 (+    29us) tablet.cc:662] Decoding operations
0709 13:09:09.002819 (+101132us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:09:09.002925 (+   106us) write_op.cc:641] Partition lock acquired for write op
0709 13:09:09.002955 (+    30us) tablet.cc:685] Acquiring locks for 28 operations
0709 13:09:09.005469 (+  2514us) tablet.cc:701] Row locks acquired
0709 13:09:09.005530 (+    61us) write_op.cc:260] PREPARE: finished
0709 13:09:09.005738 (+   208us) write_op.cc:270] Start()
0709 13:09:09.005899 (+   161us) write_op.cc:276] Timestamp: P: 1783602549005669 usec, L: 0
0709 13:09:09.005922 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 13:09:09.098814 (+ 92892us) log.cc:844] Serialized 10998 byte log entry
0709 13:09:09.821932 (+723118us) op_driver.cc:464] REPLICATION: finished
0709 13:09:10.361728 (+539796us) spinlock_profiling.cc:243] Waited 81.6 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f2973ba80ac 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297a6e8ef0
0709 13:09:10.972672 (+610944us) write_op.cc:301] APPLY: starting
0709 13:09:10.983366 (+ 10694us) tablet.cc:1367] starting BulkCheckPresence
0709 13:09:11.078692 (+ 95326us) tablet.cc:1370] finished BulkCheckPresence
0709 13:09:11.078745 (+    53us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:09:11.225488 (+146743us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:09:11.225972 (+   484us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=28
0709 13:09:11.226045 (+    73us) write_op.cc:312] APPLY: finished
0709 13:09:11.226597 (+   552us) log.cc:844] Serialized 188 byte log entry
0709 13:09:11.386067 (+159470us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:09:11.431387 (+ 45320us) write_op.cc:454] Released schema lock
0709 13:09:11.446369 (+ 14982us) write_op.cc:341] FINISH: Updating metrics
0709 13:09:11.533229 (+ 86860us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:09:11.533262 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":154694,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":1113230,"apply.run_cpu_time_us":33812,"apply.run_wall_time_us":564763,"num_ops":28,"prepare.queue_time_us":97356,"prepare.run_cpu_time_us":33719,"prepare.run_wall_time_us":266490,"raft.queue_time_us":166454,"raft.run_cpu_time_us":78860,"raft.run_wall_time_us":1152188,"replication_time_us":747042,"spinlock_wait_cycles":187783040,"thread_start_us":1236156,"threads_started":5,"wal-append.queue_time_us":266038,"wal-append.run_cpu_time_us":51574,"wal-append.run_wall_time_us":2582161}]]}
I20260709 13:09:15.337396 25141 catalog_manager.cc:5697] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 7 to 10, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 10 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:15.243952 29251 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:14.980806 29140 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:15.321980 29312 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:09:15.346196 29213 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.007s	sys 0.018s
I20260709 13:09:15.352716 29221 raft_consensus.cc:3055] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:15.402330 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:15.386567 25908 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:15.339839 29271 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:15.391167 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:15.391855 29281 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:15.392545 28814 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:15.287130 29292 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:15.324298 29279 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.064s
I20260709 13:09:15.396766 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:15.390532 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 97540.4us
I20260709 13:09:15.365876 29278 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:15.321933 29307 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:09:15.401652 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:15.391259 29283 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:15.406067 25908 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:15.407102 25240 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:15.404688 25141 catalog_manager.cc:5697] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 6 to 10, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 10 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:15.375614 29310 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:15.439635 25942 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:15.438274 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:15.468048 25141 catalog_manager.cc:5697] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 5 to 8, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 8 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:15.381651 29298 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:15.469686 25141 catalog_manager.cc:5697] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 7 to 8, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 8 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:15.470942 29298 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:15.471427 25141 catalog_manager.cc:5697] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 5 to 9, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 9 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:15.472261 29298 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:14.861627 29143 consensus_queue.cc:260] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:15.444919 25284 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:15.449200 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:15.476161 25909 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:15.450356 29271 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:15.397873 29320 consensus_queue.cc:1243] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 11.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260709 13:09:15.429840 29341 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 13:09:15.395598 29323 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:15.479949 29323 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:15.480690 29298 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:15.441454 29138 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:15.481236 29140 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:09:15.480422 29323 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:15.462910 25389 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:15.372598 29337 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:15.484047 29319 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:15.484529 29326 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:15.374168 29273 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:15.484979 29326 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:15.485045 29337 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:15.485397 29326 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:15.486740 29273 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:15.486985 29201 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:15.406800 29221 raft_consensus.cc:740] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:15.488332 29221 consensus_queue.cc:260] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:15.460018 29307 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:15.489603 29307 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:15.489925 29091 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:15.394402 28997 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:15.490424 29091 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:15.464195 29283 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:15.490897 29091 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:15.491111 29283 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:15.495332 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:15.496030 25239 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:15.496650 25238 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 12. Ops: []
W20260709 13:09:15.464833 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 58806.6us
I20260709 13:09:15.498837 25908 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:15.503535 25908 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:15.438400 29304 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:15.540686 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:15.460752 29278 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:15.402402 28914 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:15.542027 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:15.541370 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:15.549979 25945 raft_consensus.cc:2376] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9. Re-sending same reply.
I20260709 13:09:15.542639 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:15.542587 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:15.551179 25239 raft_consensus.cc:2410] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 13 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 13:09:15.541074 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:15.497877 29310 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:15.551070 25244 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:09:15.428447 29312 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:15.541013 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:15.548287 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:15.542948 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:15.557337 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:15.527289 29274 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:15.527513 29279 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:15.557921 25246 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:09:15.559900 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:15.565958 25237 raft_consensus.cc:2410] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:09:15.547521 25141 catalog_manager.cc:5697] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 5 to 10, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 10 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:15.543282 29251 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:15.557251 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 117797us
I20260709 13:09:15.549266 29278 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:15.583905 29319 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:15.585111 25943 raft_consensus.cc:2376] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11. Re-sending same reply.
I20260709 13:09:15.573153 29286 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260709 13:09:15.557900 29362 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:15.584705 29319 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:15.626817 29361 log.cc:927] Time spent T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.059s	user 0.005s	sys 0.000s
W20260709 13:09:15.622176 29353 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.180s	user 0.004s	sys 0.007s
I20260709 13:09:15.646291 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:15.647085 25945 raft_consensus.cc:2368] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11.
W20260709 13:09:15.647529 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 172455us
W20260709 13:09:15.637377 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 160345us
W20260709 13:09:15.650148 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 374819us
I20260709 13:09:15.656214 25940 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 9 index: 10. (term mismatch)
I20260709 13:09:15.656733 25940 pending_rounds.cc:85] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 9
I20260709 13:09:15.657027 25940 pending_rounds.cc:107] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 8.10
I20260709 13:09:15.657490 25940 raft_consensus.cc:2889] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:15.656922 25141 catalog_manager.cc:5697] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 6 to 10, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 10 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:15.659061 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:15.660215 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:15.660089 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:15.661532 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:15.660152 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:15.672127 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:15.675729 25358 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:15.678730 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:15.679081 25285 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:15.680891 25358 leader_election.cc:400] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11.
I20260709 13:09:15.680797 25284 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:15.659972 29270 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:15.687557 25284 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:15.688947 25284 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:15.690137 25141 catalog_manager.cc:5697] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 5 to 8, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 8 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:15.664673 29251 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:15.626091 29362 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:15.677280 29337 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:15.699183 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 124939us
I20260709 13:09:15.676893 28796 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:09:15.702754 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:09:15.690759 29371 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:15.719342 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:15.686684 29366 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:15.659778 29305 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260709 13:09:15.682925 29140 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:15.688055 29304 consensus_queue.cc:1059] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:09:15.668641 29286 consensus_queue.cc:1059] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20260709 13:09:15.732985 25246 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260709 13:09:15.751762 25246 pending_rounds.cc:85] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 8
I20260709 13:09:15.752023 25246 pending_rounds.cc:107] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 13:09:15.752285 25246 raft_consensus.cc:2889] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:15.734212 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:15.737447 25238 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:09:15.738250 25239 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:15.737491 25209 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:15.747879 29129 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:15.756093 29129 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:15.756650 29129 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:15.756651 29273 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:15.761912 29273 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:15.762403 29273 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:15.764308 25209 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:15.748237 29376 log.cc:927] Time spent T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 13:09:15.765671 29346 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:15.765275 29253 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:15.777768 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:15.778661 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 99670.1us
I20260709 13:09:15.729581 29270 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:15.788811 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:15.751157 29371 raft_consensus.cc:697] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:15.804014 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:15.783759 29253 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:15.782733 28997 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:15.708163 29251 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:15.790292 29270 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:15.767575 29140 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:15.819902 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:15.728226 29337 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:15.814369 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 96622us
W20260709 13:09:15.846556 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 78295.6us
I20260709 13:09:15.822503 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:15.901309 29371 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:15.818141 29253 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:09:15.879305 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 94768us
I20260709 13:09:15.888649 29358 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:15.902521 29221 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:09:15.906299 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:15.875303 29213 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:15.913712 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:15.807209 28914 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:15.864187 29319 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:15.898547 29281 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:15.840999 29347 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:15.847050 29312 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:15.933208 25287 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:15.841068 29304 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:15.938954 25287 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:15.807742 29383 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:15.942322 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:15.922406 29366 raft_consensus.cc:697] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:15.944851 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:15.944765 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:15.947301 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:15.956934 25320 raft_consensus.cc:2393] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 9: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:15.952664 25319 raft_consensus.cc:2410] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:09:15.922500 29279 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20260709 13:09:15.934768 29358 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:15.921187 29270 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:15.938237 29319 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:15.965936 25359 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:15.969477 25316 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 13:09:15.947853 29386 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:15.975225 25359 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:15.977684 25942 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 13:09:15.936429 28914 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 13:09:15.931901 29213 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:15.980149 25359 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:15.932034 29372 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260709 13:09:16.005167 25313 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 7.
I20260709 13:09:16.002691 25359 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:15.941147 29304 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:15.974332 29320 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:16.002496 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 49361.7us
I20260709 13:09:16.011569 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:15.978786 29383 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:15.939780 29281 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.002264 29390 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:16.007042 29386 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:16.030426 29386 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:16.030987 29386 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:16.016252 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:16.016788 25359 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:15.990473 28997 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:15.978581 29319 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.032488 25359 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:16.032675 28997 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:15.968400 29129 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:15.989565 29312 consensus_queue.cc:1048] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260709 13:09:16.033427 29397 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:16.033888 29397 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:16.034293 29397 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:16.035075 29129 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:16.035312 29326 consensus_meta.cc:287] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.549s	user 0.004s	sys 0.005s
I20260709 13:09:16.018110 29320 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.019767 29399 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:16.035729 29326 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.036106 29399 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:09:16.036320 29387 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 13:09:16.036643 29399 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:16.037536 29320 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:16.020406 29380 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:09:16.039448 25943 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 13:09:16.019886 29366 consensus_queue.cc:237] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.021497 29304 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:16.017436 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 78549.8us
I20260709 13:09:16.043882 25287 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:16.032325 29091 consensus_meta.cc:287] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.541s	user 0.006s	sys 0.006s
I20260709 13:09:15.974856 29286 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:16.041989 25359 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:15.939970 29347 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.031026 29281 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:16.044319 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:16.045097 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:15.998368 29368 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:16.012495 29273 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:16.079919 29385 log.cc:927] Time spent T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.158s	user 0.002s	sys 0.000s
I20260709 13:09:16.058811 25287 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:16.066516 25246 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:16.088016 25287 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:16.067216 25359 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:16.062469 29320 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:16.070508 29319 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:16.072055 29307 consensus_meta.cc:287] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.582s	user 0.008s	sys 0.003s
I20260709 13:09:16.066007 29286 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:15.974855 29358 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:15.997042 29367 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
I20260709 13:09:16.133977 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:16.120723 25287 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:16.051995 29352 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:16.126792 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:16.128229 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:16.128829 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:16.130069 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:16.131042 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:16.133196 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:16.049794 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:16.080119 29373 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:16.140282 25287 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:16.134644 25243 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:16.137578 25393 raft_consensus.cc:2376] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11. Re-sending same reply.
I20260709 13:09:16.141956 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:16.138113 25242 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:16.143566 25287 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:16.138846 25241 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:16.138491 25245 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:16.146560 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:16.139122 25314 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:16.148382 25287 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:16.149322 25359 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:16.152753 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:16.156574 29358 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:16.157719 29380 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:16.158313 29380 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.159008 29323 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:16.160439 29380 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:16.160946 29393 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:16.139290 29403 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:16.161628 29389 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:16.161707 29403 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:16.139374 28997 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:16.162302 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:16.163002 25942 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:16.162883 28997 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:16.162312 29403 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:16.064002 29091 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.073885 29368 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:16.206977 25287 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:16.162199 29389 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:16.133236 29319 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:16.150336 28914 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:09:16.224800 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 166077us
I20260709 13:09:16.135779 29374 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:16.151304 29320 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:16.139523 29373 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:16.139410 29400 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:09:16.228401 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 446296us
I20260709 13:09:16.232511 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:16.149744 29312 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:16.132455 29307 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.237110 25318 raft_consensus.cc:1240] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:16.192615 28997 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:16.247303 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:16.210107 29337 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:16.197767 29308 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:16.227780 29383 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:16.251305 25393 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:16.219379 29394 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:16.243163 29269 consensus_queue.cc:1059] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:09:16.227180 29389 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:16.250414 25394 raft_consensus.cc:2393] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 10: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:16.201488 29393 raft_consensus.cc:697] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 13 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:16.234573 28914 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:16.267285 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:16.240315 29320 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:16.240182 29373 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:16.246788 29400 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:16.227119 29129 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.048s
I20260709 13:09:16.247408 29326 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:16.238009 29374 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:16.327857 25313 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:09:16.417466 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:16.327792 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:16.329617 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:16.270370 29289 consensus_meta.cc:287] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.907s	user 0.001s	sys 0.014s
W20260709 13:09:16.272347 29274 consensus_meta.cc:287] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.704s	user 0.013s	sys 0.000s
W20260709 13:09:16.337414 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 102197us
I20260709 13:09:16.435669 25209 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:16.351425 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:16.352047 25320 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:09:16.237936 29319 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.209833 29399 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:16.270735 29394 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:16.374564 25908 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:16.378219 25314 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1884b7159da148dbb1c19c424d667792"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:09:16.458317 25314 raft_consensus.cc:606] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:16.458894 25314 raft_consensus.cc:614] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:09:16.253809 29312 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:16.433779 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:16.459822 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 85402.1us
W20260709 13:09:16.418820 29310 consensus_meta.cc:287] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.854s	user 0.007s	sys 0.006s
I20260709 13:09:16.460688 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:16.461709 25908 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:16.345927 29298 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:16.358999 28914 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:16.442977 25938 raft_consensus.cc:2410] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 13:09:16.447423 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:16.357704 29320 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:09:16.417011 29283 consensus_meta.cc:287] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.886s	user 0.007s	sys 0.020s
I20260709 13:09:16.358171 29366 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:16.440796 29374 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:16.441422 29326 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:16.368407 29400 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:16.449071 29274 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:16.464347 29289 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:16.465302 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:16.475291 29298 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.464448 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:16.464015 29394 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:16.543569 29253 consensus_meta.cc:287] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.616s	user 0.004s	sys 0.004s
W20260709 13:09:16.502141 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 352864us
I20260709 13:09:16.496402 29326 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:16.508129 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:16.567001 25246 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
W20260709 13:09:16.565634 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 100606us
I20260709 13:09:16.505618 29366 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:16.537266 28814 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:16.537570 29402 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.327s	user 0.001s	sys 0.000s
I20260709 13:09:16.582408 25285 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:16.533233 29354 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:16.545902 29368 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:16.546382 29410 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260709 13:09:16.548007 29399 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:16.548693 29417 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:09:16.549306 29286 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:16.549510 29091 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:16.551786 29347 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:16.553784 29307 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:16.488430 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:16.500625 29278 consensus_meta.cc:287] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.895s	user 0.010s	sys 0.002s
I20260709 13:09:16.557268 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2021) took 1186 ms. Trace:
I20260709 13:09:16.569078 25287 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:16.579031 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:16.580638 25394 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:16.581362 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:16.565841 29221 consensus_meta.cc:287] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.640s	user 0.002s	sys 0.008s
I20260709 13:09:16.537309 29289 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:16.583837 28814 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.553004 28796 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:16.585206 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:16.587543 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:16.668864 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:16.669902 25246 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:16.479611 29310 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:16.589280 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:16.566386 29326 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:16.495960 29283 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.574445 29101 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:16.592726 29354 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.575065 29380 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:16.593151 29129 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:09:16.712634 29403 consensus_meta.cc:287] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.518s	user 0.009s	sys 0.000s
W20260709 13:09:16.605613 29409 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.274s	user 0.001s	sys 0.000s
I20260709 13:09:16.712934 29380 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:16.714574 25317 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: []
W20260709 13:09:16.607232 29413 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 13:09:16.608600 29377 raft_consensus.cc:3055] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:16.717178 29377 raft_consensus.cc:740] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:16.718195 29377 consensus_queue.cc:260] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.609612 29417 raft_consensus.cc:697] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:16.720340 29417 consensus_queue.cc:237] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:16.611801 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:16.584823 29138 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:16.613059 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:16.613967 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:16.615989 25938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 756) took 2228 ms (client timeout 2002 ms). Trace:
I20260709 13:09:16.618950 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:16.619264 29386 consensus_meta.cc:287] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.588s	user 0.007s	sys 0.005s
I20260709 13:09:16.620235 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:16.620697 29352 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:16.376019 29393 consensus_queue.cc:237] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:16.593277 25942 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:16.629369 25240 rpcz_store.cc:276] 0709 13:09:15.370987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:15.383384 (+ 12397us) service_pool.cc:224] Handling call
0709 13:09:15.383737 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:15.482002 (+ 98265us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:16.556939 (+1074937us) spinlock_profiling.cc:243] Waited 23.2 ms on lock 0x7b54002d14d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:16.557192 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":53333760}
W20260709 13:09:16.639237 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 372012us
I20260709 13:09:16.727725 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:16.694422 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:16.579440 29253 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.680114 28796 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:09:16.708312 29404 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260709 13:09:16.708750 25314 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:16.709174 29101 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.605212 29298 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:16.724798 25320 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:09:16.723717 29138 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:09:16.724506 25938 rpcz_store.cc:269] 0709 13:09:14.387840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:14.393862 (+  6022us) service_pool.cc:224] Handling call
0709 13:09:16.615795 (+2221933us) spinlock_profiling.cc:243] Waited 1.97 s on lock 0x7b540062e3d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:16.615925 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4523265408}
I20260709 13:09:16.671275 29278 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.605286 29384 raft_consensus.cc:3055] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:16.730775 25246 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
W20260709 13:09:16.729054 25212 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.006s	sys 0.005s
I20260709 13:09:16.728005 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:16.798897 25212 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.006s	sys 0.005s
I20260709 13:09:16.734637 25315 raft_consensus.cc:1275] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
W20260709 13:09:16.736527 29397 consensus_meta.cc:287] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.702s	user 0.008s	sys 0.003s
I20260709 13:09:16.737040 29366 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:16.737286 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:16.739266 25284 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:16.739367 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:16.737962 28814 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:16.763307 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:16.764406 25387 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:16.723798 29377 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:16.712957 29129 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:16.725216 25936 raft_consensus.cc:2410] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 13:09:16.724643 29310 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:16.730713 25940 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: []
W20260709 13:09:16.795495 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 226477us
W20260709 13:09:16.799219 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 712187us
I20260709 13:09:16.799340 25315 pending_rounds.cc:85] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 4
I20260709 13:09:16.848955 25315 pending_rounds.cc:107] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 13:09:16.849278 25315 raft_consensus.cc:2889] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:16.800230 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:16.802155 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 340501us
I20260709 13:09:16.839161 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:16.850472 25246 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:16.843135 25945 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 12: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:16.726257 29403 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:16.798192 29138 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:16.844115 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 104895us
I20260709 13:09:16.857117 25240 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:09:16.799800 29384 raft_consensus.cc:740] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:16.858318 25284 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:16.796620 29151 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:16.858795 25240 pending_rounds.cc:85] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 5
W20260709 13:09:16.860172 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:16.731227 29386 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:16.803673 29426 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:16.863327 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:16.864423 25240 pending_rounds.cc:107] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 11.6
I20260709 13:09:16.895574 25240 raft_consensus.cc:2889] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:09:16.862954 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:16.865845 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:16.851770 28814 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:16.899917 25312 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:16.866784 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:16.867169 29403 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:16.868649 29292 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:16.846015 29129 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:16.912775 29384 consensus_queue.cc:260] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.913954 29384 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:16.914081 28796 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.861864 25212 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:16.901108 25941 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:16.901028 25246 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:09:16.851799 29319 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:16.807283 29397 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:16.837234 29320 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:16.897483 29426 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:16.910737 29253 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:16.911337 28814 raft_consensus.cc:697] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:16.864621 29312 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:16.918437 25243 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 7. Current term is 8. Ops: []
W20260709 13:09:16.919013 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 57243.6us
W20260709 13:09:16.898459 29270 consensus_meta.cc:287] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.913s	user 0.011s	sys 0.000s
W20260709 13:09:16.929003 25287 leader_election.cc:336] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.555s (SENT)
I20260709 13:09:16.889317 29358 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:16.938846 25212 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:16.934679 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:16.937053 25287 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:16.896291 29307 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:16.925477 29216 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:16.941092 25393 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
W20260709 13:09:16.946148 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:16.927635 28796 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:16.947152 25315 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "292e92613a634963a5de1f016eb951ea"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:09:16.955572 25315 raft_consensus.cc:606] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:16.954133 25212 leader_election.cc:400] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9.
I20260709 13:09:16.949610 25394 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:09:16.956379 25212 leader_election.cc:403] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:09:16.962513 25212 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:16.963891 25943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 773) took 1013 ms. Trace:
I20260709 13:09:16.962460 25287 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:16.963909 25212 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:16.964391 25943 rpcz_store.cc:276] 0709 13:09:15.950474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:16.018937 (+ 68463us) service_pool.cc:224] Handling call
0709 13:09:16.019315 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:16.039895 (+ 20580us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:16.963641 (+923746us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x7b54007044d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:16.963833 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":45581568}
I20260709 13:09:16.945410 29381 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:16.980485 25212 leader_election.cc:400] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12.
I20260709 13:09:16.981627 25212 leader_election.cc:403] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:09:16.972909 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:16.953771 29403 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:16.982543 25394 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:16.960875 29253 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:16.939530 29270 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:16.981474 29373 consensus_meta.cc:287] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.564s	user 0.003s	sys 0.008s
I20260709 13:09:16.946234 29347 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260709 13:09:16.992435 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:16.943341 29358 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:16.969507 29216 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9.
I20260709 13:09:16.961387 29352 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
I20260709 13:09:16.978628 29411 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:09:16.971295 29421 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:16.972795 29283 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:17.033073 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2200) took 1089 ms. Trace:
I20260709 13:09:17.033185 25319 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 12: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:17.033682 25943 raft_consensus.cc:1240] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:17.038142 25317 rpcz_store.cc:276] 0709 13:09:15.943981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:15.944103 (+   122us) service_pool.cc:224] Handling call
0709 13:09:15.944359 (+   256us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:16.715137 (+770778us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:17.032810 (+317673us) spinlock_profiling.cc:243] Waited 770 ms on lock 0x7b5400044554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:17.033014 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1770890112}
I20260709 13:09:16.947266 29408 consensus_queue.cc:1059] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.145s
I20260709 13:09:16.969599 28914 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.014603 29312 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:17.048153 25394 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:17.030683 29373 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.030958 29253 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12.
I20260709 13:09:16.971462 29430 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
W20260709 13:09:17.049259 29400 consensus_meta.cc:287] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.527s	user 0.010s	sys 0.000s
I20260709 13:09:17.044224 29278 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:17.038064 29421 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:16.981199 29201 consensus_queue.cc:1059] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
W20260709 13:09:17.059540 29423 log.cc:927] Time spent T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260709 13:09:17.059844 29417 raft_consensus.cc:3055] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:17.059808 29427 consensus_queue.cc:1059] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.080s
I20260709 13:09:17.060439 29417 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:17.060352 29419 consensus_queue.cc:1059] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.080s
I20260709 13:09:17.061599 29419 raft_consensus.cc:3055] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:17.062044 29419 raft_consensus.cc:740] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:17.061522 29417 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:17.062755 29429 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 13:09:17.063009 29417 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:17.048384 29411 raft_consensus.cc:697] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:16.983923 29381 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:17.067067 29420 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
I20260709 13:09:17.032632 29415 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.202s
I20260709 13:09:17.072896 29426 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:17.063894 29421 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:17.059608 29418 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:17.081332 29380 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:17.085234 29362 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:17.085780 29358 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:17.085990 29373 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:17.087244 29382 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 1.189s	user 0.006s	sys 0.007s
W20260709 13:09:17.084396 29394 consensus_meta.cc:287] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.542s	user 0.006s	sys 0.004s
I20260709 13:09:17.078338 29397 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:17.089499 29394 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.091670 29394 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:17.064057 29400 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:17.093338 29374 consensus_meta.cc:287] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.572s	user 0.007s	sys 0.003s
I20260709 13:09:17.093714 29374 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.089357 29411 consensus_queue.cc:237] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.095494 29374 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:17.095597 29407 log.cc:927] Time spent T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.253s	user 0.000s	sys 0.001s
I20260709 13:09:17.062785 29419 consensus_queue.cc:260] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.101717 29419 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:17.093034 29389 consensus_meta.cc:287] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.713s	user 0.000s	sys 0.007s
I20260709 13:09:17.103507 29389 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.108348 29421 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.111371 29421 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:17.142518 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 161926us
I20260709 13:09:17.147574 25212 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:17.151158 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:17.152127 25240 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:17.152909 25244 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:17.153965 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:17.157522 25212 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:17.148667 25237 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:09:17.166333 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 872) took 1510 ms. Trace:
I20260709 13:09:17.168746 25386 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260709 13:09:17.170390 25390 raft_consensus.cc:2393] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 8: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:17.171602 25391 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:17.173804 25314 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:09:17.175061 25314 raft_consensus.cc:606] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:17.166989 25937 rpcz_store.cc:276] 0709 13:09:15.655584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:15.663147 (+  7563us) service_pool.cc:224] Handling call
0709 13:09:15.663586 (+   439us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:15.664049 (+   463us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:15.664103 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:15.667105 (+  3002us) log.cc:844] Serialized 26 byte log entry
0709 13:09:15.760365 (+ 93260us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:15.760612 (+   247us) log.cc:844] Serialized 14 byte log entry
0709 13:09:16.195698 (+435086us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:16.195775 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:17.086594 (+890819us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:09:17.165494 (+ 78900us) raft_consensus.cc:1685] finished
0709 13:09:17.165669 (+   175us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:17.165988 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26336,"prepare.queue_time_us":399,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":35798,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":10367,"spinlock_wait_cycles":404640000,"thread_start_us":478027,"threads_started":2,"wal-append.queue_time_us":207923}
I20260709 13:09:17.015296 28814 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.183862 25212 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:17.185992 25314 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:09:17.186970 25314 raft_consensus.cc:606] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:17.187474 25314 raft_consensus.cc:614] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:09:17.189325 29091 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:17.190244 29347 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:17.191012 29347 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:17.190603 29091 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.193110 29349 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:17.193813 29349 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:17.194406 29349 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:17.195039 29270 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:17.196178 29270 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:17.053844 29312 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.189363 29253 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:17.199252 29253 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:17.191679 29347 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:17.201529 29312 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:17.203758 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:17.204900 25394 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:17.197070 29270 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.205698 29352 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:17.206233 29352 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:17.206696 29352 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:17.207212 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:17.208132 25392 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 12
I20260709 13:09:17.200649 29253 consensus_queue.cc:1059] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
I20260709 13:09:17.192785 28914 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:17.100667 29383 consensus_meta.cc:287] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.776s	user 0.008s	sys 0.000s
I20260709 13:09:17.220239 25314 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:09:17.227005 29270 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:17.244315 25314 raft_consensus.cc:606] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:17.235347 29377 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:17.248482 25314 raft_consensus.cc:614] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:09:17.260375 29442 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
I20260709 13:09:17.262317 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2196) took 1320 ms. Trace:
I20260709 13:09:17.262187 25359 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:17.262933 25316 rpcz_store.cc:276] 0709 13:09:15.942181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:15.942860 (+   679us) service_pool.cc:224] Handling call
0709 13:09:15.943225 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:15.972627 (+ 29402us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:17.261577 (+1288950us) spinlock_profiling.cc:243] Waited 25.7 ms on lock 0x7b5400115654. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:17.262025 (+   448us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":59168128}
I20260709 13:09:17.266773 29091 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:17.244982 29253 consensus_queue.cc:1243] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer 2329e428ebbf484fa3ea9452133df567 log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:09:17.248121 29383 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:17.264616 25359 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:17.269713 29374 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:17.269796 25359 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:17.270210 29374 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:17.270696 29374 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:17.271217 29390 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:17.271862 29390 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:17.272390 29390 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:17.272333 29377 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.266418 29408 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:17.277604 29377 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:17.292614 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2197) took 1350 ms. Trace:
I20260709 13:09:17.293231 25318 rpcz_store.cc:276] 0709 13:09:15.942506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:15.942753 (+   247us) service_pool.cc:224] Handling call
0709 13:09:15.958742 (+ 15989us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:16.241826 (+283084us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:17.291879 (+1050053us) spinlock_profiling.cc:243] Waited 278 ms on lock 0x7b5400276354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:17.292339 (+   460us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":639622400}
I20260709 13:09:17.296593 29408 consensus_queue.cc:1059] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 8.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:09:17.297976 29253 raft_consensus.cc:3055] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:17.298684 29253 raft_consensus.cc:740] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:17.300523 29253 consensus_queue.cc:260] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.297390 29377 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:17.304770 29377 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.303959 29253 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:17.317297 25359 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:17.318125 29138 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.320104 25941 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:17.321702 25313 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:17.324122 29390 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.321755 29312 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:17.337234 25392 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:17.339182 29349 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.341825 29352 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.343115 29373 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:17.355821 29373 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:17.343547 25359 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:17.356443 29373 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:17.343896 29394 consensus_queue.cc:1059] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:09:17.334282 25394 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:09:17.357329 25240 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 13:09:17.371194 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:17.367676 29129 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.391763 29362 raft_consensus.cc:3055] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:17.392375 29362 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:17.351938 29312 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:17.359885 29349 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:17.357560 29358 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:17.375324 29390 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:17.361970 29352 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:17.395373 25392 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:17.388525 29138 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:17.397964 29362 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.403141 29213 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:17.404035 29362 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:17.373979 29347 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.406023 29273 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:17.394961 29312 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:17.409654 25359 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:17.407182 29347 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:17.396533 29358 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:17.408427 29377 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:17.409026 29373 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.413956 29358 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:17.427346 29326 consensus_meta.cc:287] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.704s	user 0.011s	sys 0.011s
I20260709 13:09:17.428603 29216 raft_consensus.cc:3055] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:17.429333 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:17.429881 29307 raft_consensus.cc:3055] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:17.430325 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:17.431978 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:17.433560 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:17.436657 29307 raft_consensus.cc:740] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:17.439957 29143 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:17.440132 29307 consensus_queue.cc:260] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.421829 29374 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.425853 29390 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:09:17.436705 29216 raft_consensus.cc:740] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:17.444900 25316 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:17.447376 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:17.445062 29129 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:17.447719 29307 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:17.428078 29326 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:17.456228 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:17.451735 29390 raft_consensus.cc:697] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:17.463833 29394 consensus_queue.cc:1059] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:17.471310 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 626) took 1129 ms. Trace:
I20260709 13:09:17.454545 29386 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:17.475832 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:17.476193 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:17.473368 29394 raft_consensus.cc:3055] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:17.476897 29394 raft_consensus.cc:740] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:17.477638 29380 consensus_queue.cc:1059] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:09:17.478777 29426 raft_consensus.cc:3055] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:17.479275 29426 raft_consensus.cc:740] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:17.479705 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:17.479930 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:17.480434 25389 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:17.480542 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2219) took 1132 ms. Trace:
I20260709 13:09:17.481047 29374 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:17.481364 25313 rpcz_store.cc:276] 0709 13:09:16.348607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:16.464173 (+115566us) service_pool.cc:224] Handling call
0709 13:09:17.480313 (+1016140us) spinlock_profiling.cc:243] Waited 727 ms on lock 0x7b540005ee54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:17.480485 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1671643008}
I20260709 13:09:17.481913 29374 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:17.482609 29374 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.483397 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:17.484570 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:17.484858 29374 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:17.480376 29426 consensus_queue.cc:260] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:17.486187 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:17.487018 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:17.487613 25388 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:17.480544 25393 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:09:17.473871 29449 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.254s	user 0.002s	sys 0.000s
I20260709 13:09:17.489715 29426 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:17.472492 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:09:17.478003 29394 consensus_queue.cc:260] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.497722 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:17.498261 29394 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:17.498598 29326 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:17.499373 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:17.501199 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:17.501947 25313 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:17.502359 29389 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:17.502699 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:17.503542 25317 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:17.506982 25313 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: []
W20260709 13:09:17.508536 29441 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.436s	user 0.003s	sys 0.000s
I20260709 13:09:17.474490 25320 rpcz_store.cc:276] 0709 13:09:16.342183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:16.553889 (+211706us) service_pool.cc:224] Handling call
0709 13:09:16.554252 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:16.791221 (+236969us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:17.470765 (+679544us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b5400043dd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:17.471092 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":391390080}
I20260709 13:09:17.508844 25393 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:17.514223 25389 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:17.516212 29400 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:17.454530 29216 consensus_queue.cc:260] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.518671 29216 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:17.521697 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:17.527338 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:17.533733 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:17.534934 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:17.535341 25394 raft_consensus.cc:2410] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 12 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 13:09:17.536093 25387 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:17.467559 29373 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:17.536790 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:17.541908 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:09:17.545657 29140 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:17.543081 25390 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1718) took 1414 ms (client timeout 1555 ms). Trace:
W20260709 13:09:17.546284 25390 rpcz_store.cc:269] 0709 13:09:16.128481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:16.128597 (+   116us) service_pool.cc:224] Handling call
0709 13:09:17.542793 (+1414196us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400120cd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:17.542999 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2373182336}
I20260709 13:09:17.542507 25394 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 12: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:17.549453 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:17.550585 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:17.551304 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:17.552260 25245 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
W20260709 13:09:17.552672 29446 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.131s	user 0.001s	sys 0.000s
I20260709 13:09:17.552564 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:17.553143 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:17.554818 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:17.554913 25241 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:17.555558 25316 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:17.556126 29347 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:17.556806 29347 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:17.558315 29439 log.cc:927] Time spent T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.463s	user 0.001s	sys 0.000s
I20260709 13:09:17.559846 29347 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:17.561386 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:17.562525 29354 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:17.564196 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:17.566193 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:17.567006 25239 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:09:17.570686 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:17.572083 25239 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:17.577936 25392 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:09:17.579141 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1947) took 1328 ms. Trace:
I20260709 13:09:17.579604 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 1951) took 1327 ms. Trace:
I20260709 13:09:17.579610 25243 rpcz_store.cc:276] 0709 13:09:16.244869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:16.848727 (+603858us) service_pool.cc:224] Handling call
0709 13:09:16.918277 (+ 69550us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:16.935343 (+ 17066us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:17.572442 (+637099us) spinlock_profiling.cc:243] Waited 48.3 ms on lock 0x7b540048f650. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:17.572630 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":111096704}
I20260709 13:09:17.591679 29251 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:17.580022 25246 rpcz_store.cc:276] 0709 13:09:16.251735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:16.925994 (+674259us) service_pool.cc:224] Handling call
0709 13:09:16.926376 (+   382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:16.927038 (+   662us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:09:16.938872 (+ 11834us) log.cc:844] Serialized 14 byte log entry
0709 13:09:17.160771 (+221899us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:17.161861 (+  1090us) log.cc:844] Serialized 26 byte log entry
0709 13:09:17.162143 (+   282us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:09:17.162192 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:17.162330 (+   138us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:17.557440 (+395110us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:09:17.572909 (+ 15469us) raft_consensus.cc:1685] finished
0709 13:09:17.579248 (+  6339us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:17.579430 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":13666,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":39877,"raft.run_cpu_time_us":312,"raft.run_wall_time_us":8326,"spinlock_wait_cycles":278546816,"thread_start_us":221140,"threads_started":1,"wal-append.queue_time_us":137142}
I20260709 13:09:17.557544 29390 consensus_queue.cc:237] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:17.609289 29411 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.004s	sys 0.019s
W20260709 13:09:17.617750 25945 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 877) took 1856 ms (client timeout 2142 ms). Trace:
W20260709 13:09:17.618185 25945 rpcz_store.cc:269] 0709 13:09:15.761322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:15.819220 (+ 57898us) service_pool.cc:224] Handling call
0709 13:09:17.617285 (+1798065us) spinlock_profiling.cc:243] Waited 932 ms on lock 0x7b54005d0054. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:17.617549 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2143026048}
W20260709 13:09:17.618820 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:17.619710 25938 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12. Ops: []
W20260709 13:09:17.612239 29411 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.514s	user 0.004s	sys 0.019s
I20260709 13:09:17.621165 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 846) took 1248 ms. Trace:
I20260709 13:09:17.621512 25938 rpcz_store.cc:276] 0709 13:09:16.372343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:16.940197 (+567854us) service_pool.cc:224] Handling call
0709 13:09:17.619226 (+679029us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:17.620189 (+   963us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:17.620899 (+   710us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x7b54005d0050. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:17.621036 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1525844224}
I20260709 13:09:17.622305 29312 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.625772 25313 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:09:17.629212 29358 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.630720 25386 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:17.632268 29358 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:17.637868 29417 raft_consensus.cc:3055] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:17.638413 29417 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:17.639503 29417 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.640758 29417 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:17.643484 29354 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:17.644241 29354 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.651118 25387 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:17.653824 29348 consensus_queue.cc:1059] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:17.658735 29281 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:17.659456 29281 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.657716 29101 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:17.662856 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:17.663086 29281 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:17.663341 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:17.664537 25284 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:17.669451 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:17.669572 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:17.670341 25320 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:17.671015 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:17.671257 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:17.671552 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:17.671970 25319 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 8: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:17.672536 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:17.673506 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:17.674674 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:17.675256 25387 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:17.669023 29354 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:17.672860 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:17.676298 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:17.676783 25319 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:17.648769 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2205) took 1701 ms. Trace:
I20260709 13:09:17.669711 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:17.679773 25243 raft_consensus.cc:2393] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 10: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:17.683099 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:17.672717 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:17.676313 25285 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:17.655287 29348 raft_consensus.cc:3055] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:17.717886 29348 raft_consensus.cc:740] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:17.677676 25321 rpcz_store.cc:276] 0709 13:09:15.947469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:15.959078 (+ 11609us) service_pool.cc:224] Handling call
0709 13:09:15.959344 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:16.000623 (+ 41279us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:09:16.009004 (+  8381us) log.cc:844] Serialized 14 byte log entry
0709 13:09:16.920873 (+911869us) spinlock_profiling.cc:243] Waited 940 us on lock 0x7b18001fa080. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f297685223a 00007f2976cbca2a
0709 13:09:17.220277 (+299404us) log.cc:844] Serialized 14 byte log entry
0709 13:09:17.249798 (+ 29521us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:17.249918 (+   120us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:17.250063 (+   145us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:17.646815 (+396752us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b18001fa080. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b932a6 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419
0709 13:09:17.648273 (+  1458us) spinlock_profiling.cc:243] Waited 41 ms on lock 0x7b540005e954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:17.648322 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:17.648583 (+   261us) inbound_call.cc:177] Queueing success response
0709 13:09:17.654457 (+  5874us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b18001fa080. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b3fc0a 00007f297d27f219 00007f297ab8658f 00007f2973bc5571 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd
Metrics: {"mutex_wait_us":435058,"prepare.queue_time_us":61923,"prepare.run_cpu_time_us":125,"prepare.run_wall_time_us":3893,"raft.queue_time_us":87082,"raft.run_cpu_time_us":534,"raft.run_wall_time_us":7658,"spinlock_wait_cycles":186944768,"thread_start_us":291736,"threads_started":2,"wal-append.queue_time_us":1057822}
I20260709 13:09:17.719324 29312 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:17.720340 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:17.723846 25316 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:17.717916 25392 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:09:17.718938 29348 consensus_queue.cc:260] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.727596 25388 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
W20260709 13:09:17.730293 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 110806us
I20260709 13:09:17.731463 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:17.730017 29286 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:17.738245 25391 raft_consensus.cc:2393] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 10: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:17.745541 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:17.745857 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:17.746557 25945 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:09:17.745551 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 68004.8us
I20260709 13:09:17.747771 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:17.747864 25358 connection.cc:256] RPC call timeout handler was delayed by -1.416196538000122s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1695s for a response
I20260709 13:09:17.743077 29348 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:17.750602 25358 connection.cc:256] RPC call timeout handler was delayed by -1.2928965380000363s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18320000000000003s for a response
W20260709 13:09:17.755128 25358 connection.cc:256] RPC call timeout handler was delayed by -1.2397965380000642s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18910000000000002s for a response
I20260709 13:09:17.743537 29427 consensus_queue.cc:1059] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
W20260709 13:09:17.759004 25358 connection.cc:256] RPC call timeout handler was delayed by -1.1366791610000746s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
W20260709 13:09:17.759510 25358 connection.cc:256] RPC call timeout handler was delayed by -1.0091794130000835s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1878s for a response
I20260709 13:09:17.749341 29418 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:17.760962 29320 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:17.761797 29281 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:17.761598 29418 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.762468 29281 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.767988 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:17.768847 25241 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:17.768152 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:17.769739 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:17.776379 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:17.776379 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:17.779343 29308 raft_consensus.cc:3055] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:17.779515 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:17.779853 29308 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:17.779963 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:17.781070 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:17.781484 25241 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:17.780226 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:17.753324 29415 consensus_queue.cc:1059] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
I20260709 13:09:17.780754 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:17.765519 29418 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:17.783551 25284 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:17.784761 29418 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:17.784969 25940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 842) took 2109 ms. Trace:
I20260709 13:09:17.783277 29381 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:17.785456 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:17.785290 29364 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:17.785974 29418 raft_consensus.cc:697] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:09:17.775413 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:17.790364 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:17.791596 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:17.791723 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:17.787902 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:17.788717 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:17.788939 25318 raft_consensus.cc:1240] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:17.786441 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:17.792984 25321 raft_consensus.cc:2393] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 10: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:17.765340 29281 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:17.785416 25940 rpcz_store.cc:276] 0709 13:09:15.661292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:15.661401 (+   109us) service_pool.cc:224] Handling call
0709 13:09:15.661649 (+   248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:16.851887 (+1190238us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:17.744582 (+892695us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540062a2d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:17.770379 (+ 25797us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2458110848}
I20260709 13:09:17.794487 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:17.794639 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:17.798060 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:17.798346 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 122299us
I20260709 13:09:17.801659 25318 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:17.801851 25241 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:17.801828 25243 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:17.802868 25321 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 10: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:09:17.806197 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:17.808661 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:17.810015 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:17.811039 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:17.811426 25318 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:17.819509 25315 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:17.821501 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:17.823665 25319 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 13:09:17.780973 29308 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.825826 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:17.825194 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:17.828094 25945 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 6. Current term is 7. Ops: []
I20260709 13:09:17.827888 25316 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:09:17.830430 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:17.831758 25319 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:17.833184 25391 raft_consensus.cc:2393] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:17.831555 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2207) took 1819 ms. Trace:
I20260709 13:09:17.838598 25359 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:17.839536 25313 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 12: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:17.834056 25312 rpcz_store.cc:276] 0709 13:09:15.948136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:15.964393 (+ 16257us) service_pool.cc:224] Handling call
0709 13:09:16.051541 (+ 87148us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:16.900466 (+848925us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:17.739202 (+838736us) spinlock_profiling.cc:243] Waited 848 ms on lock 0x7b5400040bd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:17.767393 (+ 28191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1950487936}
I20260709 13:09:17.841749 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:17.843343 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:17.844092 25388 raft_consensus.cc:2393] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 10: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:17.844143 29427 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:17.844831 29427 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.847453 29251 consensus_queue.cc:1059] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:09:17.847594 25209 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:17.849200 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:17.851298 25938 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:17.851823 25909 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:17.850510 29216 raft_consensus.cc:3055] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:17.853506 29143 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:17.853814 29216 raft_consensus.cc:740] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:17.854080 29143 raft_consensus.cc:697] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:17.854832 29216 consensus_queue.cc:260] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.855130 29143 consensus_queue.cc:237] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.855772 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:17.856113 29216 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:09:17.858127 29453 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
W20260709 13:09:17.852977 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 44305.1us
I20260709 13:09:17.861508 29101 raft_consensus.cc:3055] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:17.862015 29101 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:17.862135 25909 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:17.863090 29101 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.863864 25909 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:17.864460 29101 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:17.864796 29281 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:17.865223 29281 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:17.865599 29281 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:17.866008 29304 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:17.869594 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:17.869910 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:17.869911 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:17.870935 25392 raft_consensus.cc:2410] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 12 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 13:09:17.870007 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:17.871915 25388 raft_consensus.cc:2368] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8.
I20260709 13:09:17.872011 29297 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:17.873422 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:17.874274 25395 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:17.875823 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:17.877696 29308 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:17.885290 29390 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:17.885890 29390 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:17.887725 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:17.901702 25394 raft_consensus.cc:2393] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:17.827070 29418 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.902858 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:17.860910 25241 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:17.876979 25908 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:17.900080 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:17.904394 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 45236.8us
I20260709 13:09:17.902781 29390 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:17.884853 29427 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:17.886101 29221 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
I20260709 13:09:17.922447 25386 raft_consensus.cc:2393] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:09:17.923635 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:17.924149 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:17.937800 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:17.941380 25285 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:17.945118 25192 auto_rebalancer.cc:893] Removing replace marker failed: Replica 46de6919f85341c6a9b574dff112e35a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:17.945309 29352 consensus_queue.cc:1059] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:09:17.945513 29416 consensus_queue.cc:1059] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:09:17.946300 29251 consensus_queue.cc:1059] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:09:17.947235 29251 raft_consensus.cc:3055] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:17.947631 29251 raft_consensus.cc:740] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:17.947691 29416 raft_consensus.cc:3055] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:17.948282 29416 raft_consensus.cc:740] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:17.949313 29140 raft_consensus.cc:3055] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:17.949739 29140 raft_consensus.cc:740] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:17.950294 25940 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:17.950678 29140 consensus_queue.cc:260] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.952132 29140 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:09:17.953047 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 76307.5us
I20260709 13:09:17.953859 25312 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:17.954982 29101 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:17.966034 25314 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 13:09:17.945678 25389 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
W20260709 13:09:17.945675 25192 auto_rebalancer.cc:901] Could not move replica: Remote error: Service unavailable: leadership transfer for f75b60146e7d4b9b9954b4fb2c414679 already in progress
W20260709 13:09:17.968053 25192 auto_rebalancer.cc:330] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for f75b60146e7d4b9b9954b4fb2c414679 already in progress
I20260709 13:09:17.972366 29304 consensus_queue.cc:1059] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:17.973562 29304 raft_consensus.cc:3055] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:17.973958 29304 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:17.974898 29304 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.976028 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:17.977293 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:17.977617 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:17.979107 29367 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:17.979506 29367 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:17.979853 29367 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:17.981676 29418 consensus_queue.cc:1059] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:09:17.983191 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:17.956790 25908 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:17.939595 25287 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:17.944725 29337 consensus_queue.cc:1059] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:09:17.909534 29281 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:17.948776 29251 consensus_queue.cc:260] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:17.991983 29251 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:17.937726 29326 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:17.992820 29281 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:17.993422 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:17.994133 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 54654.3us
I20260709 13:09:17.995028 25938 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:17.988762 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:17.995787 29427 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:09:17.996444 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:17.996708 25911 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:17.997278 29337 raft_consensus.cc:3055] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:17.988746 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:17.994985 25287 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:17.990118 25359 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:17.997730 29337 raft_consensus.cc:740] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:18.010989 25315 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 7.
I20260709 13:09:18.013898 25393 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:18.007965 29312 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:09:18.028025 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:18.028940 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:18.030978 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:18.032621 25393 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: []
W20260709 13:09:18.028885 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 72311.6us
I20260709 13:09:18.013309 29101 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:18.022284 29326 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:09:18.036262 29297 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:18.046989 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:18.044759 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:18.051204 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:18.049353 25387 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 13:09:18.049695 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:18.049438 25393 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:18.049465 25391 raft_consensus.cc:1240] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:09:18.032577 29304 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:18.041674 29312 raft_consensus.cc:697] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:18.058549 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:18.059515 25939 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: []
W20260709 13:09:18.060930 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 66049us
I20260709 13:09:18.029358 29427 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:18.068068 25938 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 12: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:18.038106 29101 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:18.007534 29416 consensus_queue.cc:260] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.072086 25386 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:18.064726 25394 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:18.077450 25287 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:18.078771 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:18.073688 29101 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:09:18.080108 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:18.081065 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:18.086180 29337 consensus_queue.cc:260] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.086740 29358 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:18.058648 29390 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.087574 29358 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.089291 25387 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:18.089570 29390 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:18.089854 29358 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:18.090721 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:18.091358 25938 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:18.091679 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:18.092372 25241 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:18.092957 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:18.092525 29312 consensus_queue.cc:237] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.093753 25242 raft_consensus.cc:2393] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 10: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:18.087427 29337 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:18.074997 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:18.105068 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:18.104882 25241 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:18.112129 25387 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:18.081367 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:18.090490 29390 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:18.114267 25359 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:18.129549 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:18.130860 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:18.131707 25242 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 12: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:18.131798 25238 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:18.131021 25240 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:18.134574 25358 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:18.135458 29394 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:18.137261 29390 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:18.139748 29390 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:18.140292 29390 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:09:18.141455 25244 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:18.142724 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:18.143568 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:18.147419 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:18.156131 25240 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 9: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:18.121796 25942 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:09:18.148195 29221 raft_consensus.cc:3055] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:18.148262 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:18.149012 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 78388.6us
I20260709 13:09:18.149516 25245 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260709 13:09:18.150079 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:18.151566 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:18.161468 25242 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
W20260709 13:09:18.153738 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:18.111225 29308 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:18.166960 25238 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:18.169490 25239 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:18.172349 25243 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 13:09:18.168020 29216 consensus_queue.cc:1059] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.006s
I20260709 13:09:18.173662 25312 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:18.174242 29216 raft_consensus.cc:3055] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:18.174642 29216 raft_consensus.cc:740] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:18.168285 29358 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:09:18.175685 29216 consensus_queue.cc:260] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.159795 29221 raft_consensus.cc:740] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:18.167021 29101 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:18.177866 25942 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:09:18.173501 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:18.179183 25244 raft_consensus.cc:2410] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 12 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 13:09:18.179224 25320 raft_consensus.cc:1240] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:18.179433 29101 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:18.180543 29216 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:18.177870 29221 consensus_queue.cc:260] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.182143 29221 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:18.187387 29367 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:18.189756 29367 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:18.189950 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:18.190779 25391 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 9: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:18.160480 25245 pending_rounds.cc:85] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 7
I20260709 13:09:18.198630 25237 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:18.195858 25245 pending_rounds.cc:107] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 13:09:18.202587 25245 raft_consensus.cc:2889] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:18.209715 25941 raft_consensus.cc:2410] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 13:09:18.214445 29216 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
W20260709 13:09:18.242753 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:18.242532 25945 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:18.247326 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:18.263391 25245 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:18.263665 25942 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 7: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:18.267035 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:18.267938 25243 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:18.276620 25938 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:09:18.277432 25287 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:18.271250 29458 log.cc:927] Time spent T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:09:18.289953 29393 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.079s
I20260709 13:09:18.313055 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:18.313791 25938 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:18.288957 29364 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 13:09:18.319227 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 101459us
W20260709 13:09:18.364336 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:18.356212 29474 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:18.379976 29390 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:18.380717 29390 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:18.385624 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:18.376910 29364 raft_consensus.cc:697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:09:18.394114 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:18.392611 29473 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:18.405738 29473 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:18.406145 29474 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:18.405599 29364 consensus_queue.cc:237] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.469543 25133 maintenance_manager.cc:419] P 4d591857b6ac45ad9cab82445b07ae03: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033389
W20260709 13:09:18.489876 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 219999us
I20260709 13:09:18.501302 25315 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:09:18.508128 25140 catalog_manager.cc:5697] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 8 to 9, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 9 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:09:18.511351 25140 catalog_manager.cc:5697] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:09:18.512617 29417 raft_consensus.cc:3055] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:18.513196 29304 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:18.513797 29304 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:18.514186 25140 catalog_manager.cc:5697] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:18.515785 29416 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:18.516760 29304 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:18.513170 29417 raft_consensus.cc:740] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:09:18.523952 25141 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53264 (request call id 206) took 14932 ms (client timeout 15000 ms). Trace:
I20260709 13:09:18.531894 25243 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
W20260709 13:09:18.536273 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 401765us
I20260709 13:09:18.537017 29394 raft_consensus.cc:697] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:09:18.545543 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:18.515802 25140 catalog_manager.cc:5697] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 2 to 9, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 9 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:18.548594 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:18.553768 25285 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:18.568018 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:18.568809 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:18.569469 25941 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:18.571864 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:18.579427 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:18.580037 25243 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:18.575407 25358 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:18.541980 29221 consensus_queue.cc:1059] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.031s
I20260709 13:09:18.543149 29337 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 13:09:18.558130 29411 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:18.590842 25941 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:18.595479 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:18.569352 29381 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:18.545471 29418 consensus_queue.cc:1059] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 13:09:18.596230 25241 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:18.597290 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:18.597812 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:18.557798 29308 consensus_queue.cc:1059] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 13:09:18.602763 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:09:18.603068 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:18.595543 29251 consensus_meta.cc:287] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.603s	user 0.020s	sys 0.002s
W20260709 13:09:18.601907 29140 consensus_meta.cc:287] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.647s	user 0.020s	sys 0.004s
I20260709 13:09:18.597769 29143 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:18.573468 29286 consensus_queue.cc:1059] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 13:09:18.598506 29381 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:18.611879 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:18.597981 29367 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 13:09:18.615332 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:18.618813 29143 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:18.619697 29417 consensus_queue.cc:260] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.620811 29417 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:18.604857 29304 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:18.597548 29312 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:09:18.607646 29415 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:18.608727 29411 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:18.622319 25241 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:18.612536 25209 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:18.626507 29143 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:18.611136 29364 consensus_queue.cc:1059] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
W20260709 13:09:18.629835 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:18.635150 25909 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:18.611758 29474 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:18.636991 25909 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:18.622529 29381 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:18.542543 25141 rpcz_store.cc:269] 0709 13:09:03.591470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:03.617680 (+ 26210us) service_pool.cc:224] Handling call
0709 13:09:18.523710 (+14906030us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:09:16.085312 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:09:16.085908 (+   596us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:09:16.085948 (+    40us) write_op.cc:435] Acquired schema lock
0709 13:09:16.085963 (+    15us) tablet.cc:662] Decoding operations
0709 13:09:16.200137 (+114174us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:09:16.210851 (+ 10714us) write_op.cc:641] Partition lock acquired for write op
0709 13:09:16.227201 (+ 16350us) tablet.cc:685] Acquiring locks for 10 operations
0709 13:09:16.487639 (+260438us) tablet.cc:701] Row locks acquired
0709 13:09:16.487665 (+    26us) write_op.cc:260] PREPARE: finished
0709 13:09:16.487868 (+   203us) write_op.cc:270] Start()
0709 13:09:16.488003 (+   135us) write_op.cc:276] Timestamp: P: 1783602556487806 usec, L: 0
0709 13:09:16.488022 (+    19us) op_driver.cc:348] REPLICATION: starting
0709 13:09:16.488735 (+   713us) log.cc:844] Serialized 4047 byte log entry
0709 13:09:17.286783 (+798048us) op_driver.cc:464] REPLICATION: finished
0709 13:09:17.418370 (+131587us) write_op.cc:301] APPLY: starting
0709 13:09:17.420539 (+  2169us) tablet.cc:1367] starting BulkCheckPresence
0709 13:09:17.450734 (+ 30195us) tablet.cc:1370] finished BulkCheckPresence
0709 13:09:17.452399 (+  1665us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:09:17.467038 (+ 14639us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:09:17.558406 (+ 91368us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0709 13:09:17.558460 (+    54us) write_op.cc:312] APPLY: finished
0709 13:09:17.558936 (+   476us) log.cc:844] Serialized 77 byte log entry
0709 13:09:17.559149 (+   213us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:09:17.559379 (+   230us) write_op.cc:454] Released schema lock
0709 13:09:17.559452 (+    73us) write_op.cc:341] FINISH: Updating metrics
0709 13:09:17.560059 (+   607us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:09:17.560095 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":433582,"reported_tablets":34,"child_traces":[["op",{"apply.queue_time_us":123722,"apply.run_cpu_time_us":6752,"apply.run_wall_time_us":141731,"num_ops":10,"prepare.queue_time_us":208976,"prepare.run_cpu_time_us":26753,"prepare.run_wall_time_us":747018,"raft.queue_time_us":201978,"raft.run_cpu_time_us":15581,"raft.run_wall_time_us":105172,"replication_time_us":798359,"thread_start_us":627915,"threads_started":4,"wal-append.queue_time_us":456246,"wal-append.run_cpu_time_us":15378,"wal-append.run_wall_time_us":1619389}]]}
I20260709 13:09:18.639787 25242 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:18.640233 25237 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:09:18.645289 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:18.634099 29367 raft_consensus.cc:697] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:18.611402 29394 consensus_queue.cc:237] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.638806 29390 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:18.642928 29326 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:18.654573 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:18.641862 29286 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:18.655705 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:18.663661 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:18.636066 29308 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20260709 13:09:18.641633 29143 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:18.650584 29337 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:18.652900 29101 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:18.715346 29101 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:18.715919 29101 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:18.663007 29326 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:18.639194 29304 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:18.707221 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:18.711483 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 139676us
I20260709 13:09:18.713652 25359 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:18.709278 25212 leader_election.cc:336] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.519s (SENT)
I20260709 13:09:18.721812 25212 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:18.661312 29140 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:18.717876 25358 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:18.719619 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:18.723013 25287 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:18.702281 29473 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:18.707351 29143 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:18.653368 29312 consensus_queue.cc:1059] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.068s
I20260709 13:09:18.707294 29474 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:18.717247 29390 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:09:18.749363 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 170345us
I20260709 13:09:18.724119 29326 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:18.752316 29472 log.cc:927] Time spent T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 13:09:18.752314 29143 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:18.778723 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 167546us
I20260709 13:09:18.754529 29427 raft_consensus.cc:3055] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:18.757078 25909 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:18.781447 29281 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:18.786551 29281 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:18.787106 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:18.757647 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:18.788846 29469 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.097s
I20260709 13:09:18.789597 25241 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 9: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:18.790604 29469 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:18.791036 29469 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:18.791381 29469 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:18.792605 29320 raft_consensus.cc:3055] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Stepping down as leader of term 11
W20260709 13:09:18.758836 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 45283us
I20260709 13:09:18.748005 25140 catalog_manager.cc:5697] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 2 to 8, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 8 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:18.793045 29320 raft_consensus.cc:740] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:18.793984 29320 consensus_queue.cc:260] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.795388 29320 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:18.736874 29364 raft_consensus.cc:3055] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:18.796599 29364 raft_consensus.cc:740] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:18.797598 29364 consensus_queue.cc:260] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.798789 29364 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:18.800530 29393 raft_consensus.cc:3055] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:18.801002 29393 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:09:18.801467 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 92230.8us
W20260709 13:09:18.802147 25212 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.551s (SENT) [suppressed 35 similar messages]
I20260709 13:09:18.801956 29393 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.757036 29347 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:18.708817 29468 consensus_queue.cc:1059] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.081s
I20260709 13:09:18.734030 29337 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.750680 29140 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:09:18.804171 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 47192.6us
W20260709 13:09:18.807384 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:18.751327 29418 raft_consensus.cc:3055] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:18.724438 29286 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:18.761608 25284 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:18.760946 29390 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:18.813335 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:18.820855 25941 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:18.822007 25285 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:18.814077 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:18.707314 29367 consensus_queue.cc:237] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:18.785039 29221 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:09:18.824250 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:18.803236 29393 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:18.825829 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:18.829614 25241 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:18.788932 29281 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:18.785488 29427 raft_consensus.cc:740] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:18.811574 29348 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:18.832546 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:18.837013 25945 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:18.815179 29418 raft_consensus.cc:740] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:18.837184 25244 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:09:18.823725 29390 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:18.782191 29474 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:18.852208 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 90727.1us
I20260709 13:09:18.854496 25287 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:18.856148 25287 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:18.858494 29482 consensus_queue.cc:1059] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:09:18.859366 29482 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:18.859805 29482 raft_consensus.cc:697] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:18.861524 29485 raft_consensus.cc:3055] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:18.861948 29485 raft_consensus.cc:740] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:18.862340 29143 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:18.862797 29485 consensus_queue.cc:260] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.864025 29485 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:18.864423 29474 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:18.866227 29286 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:18.860656 29482 consensus_queue.cc:237] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.869181 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:18.874662 25242 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:18.874850 29308 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:18.875349 29308 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:18.875778 29308 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:18.852957 25284 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:18.878054 29286 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:09:18.862826 29487 raft_consensus.cc:3055] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:09:18.878809 29487 raft_consensus.cc:740] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:18.879241 25284 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:18.879707 29487 consensus_queue.cc:260] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:18.850721 29304 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:18.877610 29418 consensus_queue.cc:260] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.848240 29488 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:18.881481 29304 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:18.831103 29424 consensus_queue.cc:1059] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.054s
I20260709 13:09:18.881754 29418 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:18.885313 29221 raft_consensus.cc:697] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:18.885468 29483 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:09:18.886135 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:18.886374 29221 consensus_queue.cc:237] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.888156 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:18.839303 29383 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:18.858713 29484 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:18.886778 25315 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:18.884565 29486 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:09:18.888811 25314 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
W20260709 13:09:18.890450 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 172685us
I20260709 13:09:18.905684 25358 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:18.891525 25287 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:18.885308 29480 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:09:18.838126 29411 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:18.907660 29480 raft_consensus.cc:3055] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:18.908036 29480 raft_consensus.cc:740] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:18.908654 29337 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:18.908865 25287 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:18.909092 29481 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:09:18.895937 29101 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:18.881919 29304 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:18.881001 29487 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:18.881816 29488 raft_consensus.cc:697] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:18.838438 29348 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:18.897037 29427 consensus_queue.cc:260] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.917398 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:18.920269 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 118435us
I20260709 13:09:18.928444 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:18.929294 25395 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:18.933007 25911 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:18.934149 29491 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:18.934845 29491 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:18.935388 29491 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:18.937852 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 135713us
I20260709 13:09:18.940304 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:18.921922 29304 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:18.922509 29358 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:18.926206 29348 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:18.962843 25212 leader_election.cc:336] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.634s (SENT)
W20260709 13:09:18.968966 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:18.960692 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:18.976339 25212 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:18.980489 29101 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:18.939425 29484 raft_consensus.cc:697] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:18.943931 29411 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:18.957437 29286 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:18.956974 29140 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:09:18.993258 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 168048us
I20260709 13:09:18.994514 25909 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:18.995847 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:18.972445 29383 raft_consensus.cc:697] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:18.943462 29486 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260709 13:09:18.985963 29411 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:19.012681 29481 raft_consensus.cc:697] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:19.000304 29140 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:09:19.010993 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 80637.9us
I20260709 13:09:19.010062 29286 raft_consensus.cc:697] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:19.023828 29286 consensus_queue.cc:237] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:18.999099 29474 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:09:19.026528 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 104381us
W20260709 13:09:19.026473 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 63681.7us
I20260709 13:09:19.026473 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:18.908913 29480 consensus_queue.cc:260] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:19.010927 29427 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:19.046506 25285 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:19.071018 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:19.032042 29486 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:19.033449 29381 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:19.034576 29497 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 13:09:19.002586 29482 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20260709 13:09:19.016185 29101 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 13:09:19.038699 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:19.085942 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:19.014886 29281 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:19.072739 25395 raft_consensus.cc:3055] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:19.074836 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:19.016588 29411 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:19.026146 29140 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:19.083784 25242 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:09:19.074728 29486 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:19.096163 25395 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:19.103184 25391 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 13:09:19.076292 29483 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:19.118813 29483 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:19.119313 29483 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:19.074175 29488 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.078159 29143 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:19.125200 29383 consensus_queue.cc:237] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.117240 25320 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:19.118054 29480 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:19.127835 25387 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:19.029232 29390 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:19.145807 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 101138us
W20260709 13:09:19.156486 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 69358.5us
I20260709 13:09:19.075524 29424 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.051s
I20260709 13:09:19.157552 25908 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:19.125787 29143 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.111450 29486 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:19.159224 25395 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.175314 25908 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:19.176920 25358 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:19.126543 29101 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:19.182366 29101 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.131117 29474 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:19.186290 29474 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:19.189375 29394 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:19.189998 29394 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.192387 29394 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:19.194904 29390 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:19.195724 29390 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:19.196573 25395 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:19.085738 29484 consensus_queue.cc:237] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:19.201490 25391 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 13:09:19.188887 29474 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:19.172987 29424 consensus_queue.cc:1243] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 10.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:09:19.213639 29358 raft_consensus.cc:697] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:19.223731 29358 consensus_queue.cc:237] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.193011 29394 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:19.226316 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:19.176645 29469 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:19.230706 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:19.228369 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 220407us
I20260709 13:09:19.226831 25395 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 11. (term mismatch)
I20260709 13:09:19.180192 29498 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:19.234256 25395 pending_rounds.cc:85] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 10
I20260709 13:09:19.239987 25395 pending_rounds.cc:107] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 11.11
I20260709 13:09:19.240388 25395 raft_consensus.cc:2889] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:19.242121 25909 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:19.085294 29481 consensus_queue.cc:237] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:19.247570 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 72504us
W20260709 13:09:19.248790 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 72045.1us
I20260709 13:09:19.236830 29101 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:19.250142 29505 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:19.246435 25209 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:19.232301 29394 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.330928 29512 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:09:19.346540 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 100168us
I20260709 13:09:19.350986 29469 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:19.351707 29320 raft_consensus.cc:3055] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:19.352240 29320 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:19.352308 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:09:19.351358 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:19.356545 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:19.357357 25387 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:19.350862 29348 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.358955 25911 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:19.360278 29348 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:19.351117 29474 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:19.361840 29474 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:19.362301 29474 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:19.364653 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:19.367179 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:19.367807 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:19.246440 29513 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:09:19.390249 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:19.339964 29510 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 13:09:19.417022 25395 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:09:19.250170 29143 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:19.422621 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:19.427062 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:19.409967 29509 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 0.115s	user 0.002s	sys 0.000s
I20260709 13:09:19.432049 25387 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:09:19.432858 25243 raft_consensus.cc:1217] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Deduplicated request from leader. Original: 6.7->[8.8-11.10]   Dedup: 8.8->[9.9-11.10]
I20260709 13:09:19.433451 25243 pending_rounds.cc:85] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 8
I20260709 13:09:19.427918 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:19.433691 25243 pending_rounds.cc:107] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260709 13:09:19.433959 25243 raft_consensus.cc:2889] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:19.434576 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:19.427748 25315 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:19.409796 29485 consensus_meta.cc:287] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.501s	user 0.013s	sys 0.000s
I20260709 13:09:19.437405 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:19.365918 29427 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:19.425395 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:19.442900 25321 raft_consensus.cc:2393] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 10: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:19.428530 25389 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:19.445564 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:19.446238 25319 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:19.411325 29513 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:19.428157 29143 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:19.459406 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:19.470070 25319 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:19.420382 29482 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:19.418394 29512 raft_consensus.cc:697] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:19.471061 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:19.473713 29348 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:09:19.473906 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 135579us
I20260709 13:09:19.480135 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:19.488353 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:19.499542 25315 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:19.437616 29393 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:19.416321 29304 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.410530 29308 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.421094 29491 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.465947 29513 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:19.512099 29348 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:19.549674 25312 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 10: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:19.556000 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:19.552368 29348 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:19.464001 29427 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.531070 29381 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:19.513834 29482 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:19.572340 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 213451us
I20260709 13:09:19.534375 29485 consensus_queue.cc:1059] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 13:09:19.583192 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:19.583868 25242 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:19.559896 29337 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:19.563848 29221 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:19.520998 29393 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.584062 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:19.581197 29417 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:19.582055 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 153970us
I20260709 13:09:19.534842 29143 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:19.596503 29519 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:19.596935 29508 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:19.597877 25945 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:09:19.606956 29417 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:19.598707 29337 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:19.629629 29508 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:19.630828 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:19.608700 29514 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:19.613435 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:19.641377 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:19.646656 25237 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:19.615126 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:19.668901 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 240439us
I20260709 13:09:19.641793 25936 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:19.688133 25943 raft_consensus.cc:2393] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 10: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:09:19.611157 29326 consensus_meta.cc:287] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.803s	user 0.011s	sys 0.000s
W20260709 13:09:19.695150 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 56962us
I20260709 13:09:19.697640 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:19.698489 25240 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:19.690779 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 167149us
W20260709 13:09:19.641734 29483 consensus_meta.cc:287] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.522s	user 0.011s	sys 0.000s
W20260709 13:09:19.706637 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 67692us
I20260709 13:09:19.612550 29143 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:19.717571 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:19.691761 29515 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.211s	user 0.001s	sys 0.001s
I20260709 13:09:19.711493 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:19.720953 25940 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:19.613233 29308 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:19.723130 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:19.648206 29508 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:19.723867 25936 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:19.725530 25242 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:19.735706 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:19.615916 29381 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.104s
I20260709 13:09:19.604779 29221 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:19.636749 29411 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.614527 29486 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:19.740594 25943 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:19.739888 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:19.708657 29418 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:19.723263 29394 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:19.761173 25240 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:19.762024 25936 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:19.792309 25937 raft_consensus.cc:2393] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 10: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:19.759485 25238 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
W20260709 13:09:19.784485 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 49547.5us
I20260709 13:09:19.613718 29519 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.593262 29512 consensus_queue.cc:237] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.705952 29326 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.727259 29143 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:19.741518 29523 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:19.740242 29337 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:19.723433 29424 consensus_queue.cc:1059] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.061s
I20260709 13:09:19.756139 29468 raft_consensus.cc:3055] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:19.746483 29522 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:19.726089 29286 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.067s
W20260709 13:09:19.814864 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 92452.9us
I20260709 13:09:19.720184 29483 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:19.842116 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 134715us
I20260709 13:09:19.797067 29491 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:19.854171 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 215741us
I20260709 13:09:19.760927 29502 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
W20260709 13:09:19.864490 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 120369us
W20260709 13:09:19.847733 29495 log.cc:927] Time spent T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 13:09:19.799307 29418 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:19.820386 29393 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:19.818701 29427 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:19.884492 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:19.885188 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:19.819564 29468 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:19.879776 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:19.817596 29143 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:19.864764 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 70557.2us
I20260709 13:09:19.886108 25321 raft_consensus.cc:2368] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10.
W20260709 13:09:19.888055 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:19.889645 25246 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:09:19.881575 25909 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:19.872712 29140 consensus_meta.cc:287] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.742s	user 0.013s	sys 0.000s
I20260709 13:09:19.909435 25314 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:19.817840 29523 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.929800 25318 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:19.887615 29482 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:19.937094 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:09:19.937955 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 215273us
I20260709 13:09:19.939848 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:19.941460 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:19.944144 25238 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10. Ops: []
W20260709 13:09:19.921687 29487 consensus_meta.cc:287] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.997s	user 0.013s	sys 0.000s
I20260709 13:09:19.946449 25395 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:09:19.947441 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:19.949870 25908 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:19.952299 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:19.852813 29522 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:19.951861 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:19.954032 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:19.908061 29491 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:19.962911 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:19.964057 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:19.968025 25393 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260709 13:09:19.952189 29480 consensus_meta.cc:287] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.806s	user 0.012s	sys 0.000s
I20260709 13:09:19.893353 29500 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.061s
I20260709 13:09:19.975392 29221 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:19.975926 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 103999us
I20260709 13:09:19.976612 29536 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:19.980757 29367 consensus_queue.cc:1059] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:19.982502 29522 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:19.984488 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:19.985678 29485 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:19.987128 29308 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:19.988440 29468 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.989732 29417 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:19.977012 29525 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.234s
I20260709 13:09:19.924831 29535 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:19.971740 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:20.003664 29486 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:20.006423 25237 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:19.911528 29140 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:19.961982 29415 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:20.037801 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:20.045735 25395 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:20.046180 29411 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:20.047144 29308 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.048450 25358 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:20.049539 29308 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:19.976655 29143 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:20.052603 29487 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:20.054706 29140 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:20.055428 29485 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:20.056373 29519 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:09:20.055450 29286 raft_consensus.cc:3055] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:20.057575 29485 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:20.023392 29390 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:20.025653 29492 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:19.973562 29482 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.070s
I20260709 13:09:20.036873 29221 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:20.084553 29499 raft_consensus.cc:3055] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:20.053269 29535 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:20.057873 29286 raft_consensus.cc:740] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:20.080938 29390 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:20.074466 29491 raft_consensus.cc:697] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:20.090184 25395 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:20.101178 25911 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:20.079145 25209 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:20.090446 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:20.026947 29536 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.079413 25246 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
W20260709 13:09:20.113951 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:20.080559 29468 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:20.116640 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:20.094549 29487 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:20.093016 29499 raft_consensus.cc:740] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:20.095466 29418 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:20.124039 29546 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:20.124478 29474 consensus_meta.cc:287] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.762s	user 0.009s	sys 0.003s
I20260709 13:09:20.124419 29529 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:20.121114 29143 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:20.121460 29483 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:20.125883 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:20.126977 29499 consensus_queue.cc:260] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:20.122764 29487 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:20.129498 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:20.172391 29491 consensus_queue.cc:237] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.130429 29535 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:20.173878 29498 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:20.174273 29529 raft_consensus.cc:697] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:20.174393 29498 raft_consensus.cc:697] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:20.175208 29529 consensus_queue.cc:237] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.175376 29498 consensus_queue.cc:237] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.177291 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:20.138650 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:20.178213 25388 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:09:20.146117 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:20.096390 29221 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:20.182870 25388 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:09:20.182928 29221 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:20.148703 25908 leader_election.cc:400] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10.
I20260709 13:09:20.184468 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:20.171272 29469 consensus_queue.cc:1048] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260709 13:09:20.185509 29555 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:20.186105 29320 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:20.187285 29320 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:20.187753 29555 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:20.188206 29555 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:20.149453 25395 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:09:20.149926 29417 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:20.191134 29417 raft_consensus.cc:3055] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:20.191534 29417 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:20.150179 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:20.150406 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:20.151019 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:20.185503 29390 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:20.193676 25387 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:20.153107 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:20.194759 25393 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:20.154702 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:20.155705 25386 raft_consensus.cc:1275] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:09:20.155928 29554 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:20.156404 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:20.157351 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:20.191543 29499 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:20.146384 29286 consensus_queue.cc:260] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.157476 29418 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:09:20.103384 29348 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.192495 29417 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:20.202183 29417 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:20.160764 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:20.091885 29492 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.168323 25316 raft_consensus.cc:2393] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 10: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:20.205518 25321 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:20.169879 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:20.165961 29143 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:20.158054 25287 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:20.144357 29101 consensus_queue.cc:1059] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.061s
I20260709 13:09:20.211645 25393 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:20.202953 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:20.167061 29474 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.227478 29557 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:20.238117 29558 log.cc:927] Time spent T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.068s	user 0.002s	sys 0.000s
I20260709 13:09:20.238016 29557 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.239364 29474 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:20.240206 25239 raft_consensus.cc:2393] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:20.240545 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:20.212601 29143 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:20.231865 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:20.242004 25393 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:20.243542 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:20.228303 29326 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:20.233819 25390 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:20.234673 29492 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:20.235289 25943 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:20.204535 29415 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:20.240640 29557 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:20.241221 25938 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:20.169214 29546 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:20.232719 29523 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:20.247664 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 89685.5us
I20260709 13:09:20.278056 25287 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:20.279175 29418 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:09:20.279911 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:20.249231 29484 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:20.270277 29415 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.282691 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:20.286441 25388 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:20.256639 29514 raft_consensus.cc:2764] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election decision vote started in defunct term 9: won
W20260709 13:09:20.284042 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 81163.3us
W20260709 13:09:20.287410 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:20.277168 29526 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:20.296413 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:20.286882 29418 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.010s
I20260709 13:09:20.298866 25358 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:20.305240 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:20.288300 29483 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:20.287217 29320 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260709 13:09:20.287346 29553 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:20.343467 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:20.350998 25387 raft_consensus.cc:2393] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 10: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:20.352550 25321 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
W20260709 13:09:20.354342 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 47540.3us
I20260709 13:09:20.310577 29487 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:20.342011 29304 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:20.362603 29415 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:20.342612 29394 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:20.368422 25238 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:09:20.306561 29312 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:09:20.369598 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 64478us
I20260709 13:09:20.302316 29390 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:20.381870 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:20.311575 29508 consensus_meta.cc:287] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.545s	user 0.009s	sys 0.002s
I20260709 13:09:20.384675 25246 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:20.342356 29483 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:20.350695 29553 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:20.384011 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:20.389364 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:20.402536 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:20.393152 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:20.403312 25393 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:20.393028 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:20.389021 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:20.389088 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:20.355607 29348 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:20.391639 25285 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:20.381593 29320 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:20.408285 25244 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:09:20.408893 25238 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
W20260709 13:09:20.409375 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 150689us
I20260709 13:09:20.409770 25285 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:20.416642 25243 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:09:20.383476 29312 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:20.382651 29474 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:20.426035 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:20.398587 29424 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:20.422158 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:20.434573 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:20.426095 25911 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:20.432740 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:20.432927 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:20.341769 29545 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.130s
I20260709 13:09:20.435767 25241 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:20.304883 29526 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.438800 25285 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:20.435627 25238 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:20.444995 25358 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:20.404954 29553 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:20.455260 25908 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:20.456081 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:20.457027 25238 raft_consensus.cc:2410] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 12 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
W20260709 13:09:20.457959 29566 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
W20260709 13:09:20.460054 29552 log.cc:927] Time spent T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260709 13:09:20.461776 25238 raft_consensus.cc:1275] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260709 13:09:20.462368 25238 pending_rounds.cc:85] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 4
I20260709 13:09:20.462648 25238 pending_rounds.cc:107] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 13:09:20.462893 25238 raft_consensus.cc:2889] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:20.431311 29312 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:20.420073 29485 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:20.429723 29393 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:20.450881 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:20.452275 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:20.434296 29390 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:20.394125 29143 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:20.449617 29513 consensus_meta.cc:287] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.903s	user 0.013s	sys 0.000s
I20260709 13:09:20.488920 29568 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:20.489360 29513 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.490485 25941 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:20.471410 25244 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:20.475407 25285 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:20.479995 29394 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:20.470661 29337 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:20.483796 29415 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:20.486013 29564 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:09:20.487527 25908 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:20.519203 25285 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:20.447693 29367 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 13:09:20.474390 29390 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:20.522994 25244 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:20.522933 25241 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:20.399304 29508 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.495944 25938 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:20.442267 29383 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:20.500175 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:20.501142 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 75096.7us
I20260709 13:09:20.428570 29320 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:20.473183 29569 log.cc:927] Time spent T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260709 13:09:20.506628 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:20.537245 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:20.504181 25359 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:20.514515 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:20.434401 29474 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:20.424456 29486 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 13:09:20.525874 29546 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:20.444418 29424 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:20.500337 29554 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:09:20.509933 29565 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 13:09:20.533597 25209 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:20.536121 25246 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:20.504755 29485 raft_consensus.cc:697] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:09:20.657995 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 124469us
W20260709 13:09:20.655681 25212 leader_election.cc:336] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.551s (SENT)
I20260709 13:09:20.537272 29383 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:20.518867 29393 raft_consensus.cc:697] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:20.499604 29536 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:20.580906 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 76784.7us
W20260709 13:09:20.726033 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:20.508559 29337 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.522228 29415 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.613534 29513 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:20.551417 29394 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:20.529604 29390 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:20.526846 29500 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:20.536929 29544 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:20.603318 29518 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:09:20.553174 25391 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:20.508168 29567 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:20.529417 29499 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.120s
I20260709 13:09:20.649056 25285 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:20.650637 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:20.739424 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 90462.3us
I20260709 13:09:20.527379 29143 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:20.551541 25238 raft_consensus.cc:2368] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10.
I20260709 13:09:20.506116 29483 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:20.661192 29535 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:20.654315 29411 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:20.548144 29484 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.109s
I20260709 13:09:20.778362 25238 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:09:20.664008 29530 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:20.781250 29535 raft_consensus.cc:697] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:20.784065 29535 consensus_queue.cc:237] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:20.784737 29573 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.075s	user 0.002s	sys 0.009s
I20260709 13:09:20.786769 29542 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:20.786979 29541 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:20.787343 29542 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:20.787541 29541 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:20.791445 25209 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:20.721202 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:20.651984 29101 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:20.724038 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 68398.7us
I20260709 13:09:20.702102 29468 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:20.739465 25241 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
W20260709 13:09:20.651587 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 119004us
W20260709 13:09:20.729674 29417 consensus_meta.cc:287] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.516s	user 0.007s	sys 0.005s
I20260709 13:09:20.773906 29546 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:20.856036 29415 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:20.855773 29546 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:20.858255 29546 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:20.859355 29474 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:20.859710 29320 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:20.860176 29101 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.861212 29492 raft_consensus.cc:3055] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:20.861642 29492 raft_consensus.cc:740] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:09:20.862149 29576 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
I20260709 13:09:20.862419 29101 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:20.862567 29492 consensus_queue.cc:260] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:20.863354 29574 log.cc:927] Time spent T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.074s	user 0.000s	sys 0.003s
I20260709 13:09:20.774183 29393 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.775405 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:20.867151 25245 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:20.776163 29312 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:20.778535 25244 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 13:09:20.720095 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:20.774425 29411 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:20.737787 29567 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:20.738610 29544 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:20.738451 29500 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:09:20.853305 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:20.853022 25358 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:20.880133 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:20.863687 29492 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:20.849852 29418 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:20.850512 29530 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:20.990054 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:20.961010 25387 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:09:20.986871 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:20.892520 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:20.976675 25285 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:20.984724 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 131784us
I20260709 13:09:21.017787 25237 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:21.051453 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 800) took 1014 ms. Trace:
I20260709 13:09:21.120546 25391 rpcz_store.cc:276] 0709 13:09:20.036721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:20.037266 (+   545us) service_pool.cc:224] Handling call
0709 13:09:20.037565 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:20.738855 (+701290us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:21.051204 (+312349us) spinlock_profiling.cc:243] Waited 514 ms on lock 0x7b5400147154. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:21.051390 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1181885184}
I20260709 13:09:21.074025 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 798) took 1490 ms. Trace:
I20260709 13:09:20.864542 29143 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:21.122735 25359 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:21.114100 25358 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:21.121985 25941 rpcz_store.cc:276] 0709 13:09:19.583591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:19.583813 (+   222us) service_pool.cc:224] Handling call
0709 13:09:21.073826 (+1490013us) spinlock_profiling.cc:243] Waited 896 ms on lock 0x7b54004ad8d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:21.073961 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2060792064}
I20260709 13:09:21.113905 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:20.861261 29570 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.206s	user 0.001s	sys 0.006s
I20260709 13:09:21.138290 25244 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
W20260709 13:09:21.111404 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:21.133911 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 157307us
I20260709 13:09:20.851924 29468 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:21.134591 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:21.113065 25238 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:21.146457 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:21.152812 25241 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:21.149066 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:21.149677 25246 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: []
W20260709 13:09:21.033739 29578 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.243s	user 0.001s	sys 0.000s
W20260709 13:09:21.033165 29572 log.cc:927] Time spent T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
I20260709 13:09:21.190371 25909 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:21.032661 29501 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:21.048910 29312 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:21.080027 29553 consensus_meta.cc:287] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.608s	user 0.003s	sys 0.011s
I20260709 13:09:21.109552 29500 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:09:21.049161 29577 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
W20260709 13:09:21.133634 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:21.189534 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:21.138248 29143 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:21.049667 29424 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:21.155239 29481 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:21.203307 25909 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:21.022539 29554 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:21.190729 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:21.079936 29555 consensus_meta.cc:287] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.891s	user 0.013s	sys 0.000s
I20260709 13:09:21.204879 25238 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:09:20.953163 29487 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:20.995296 29541 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:21.138513 29418 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:21.244222 29312 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:21.207156 25245 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:09:21.207237 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:21.249265 29511 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:20.960638 29364 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:21.208002 25391 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:09:21.155841 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:21.248813 29536 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:20.882104 29567 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:21.210790 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:21.317477 25241 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
W20260709 13:09:21.224108 25212 leader_election.cc:336] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.652s (SENT)
I20260709 13:09:21.318331 25212 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:21.110105 29544 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:21.226831 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:21.148442 29101 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:21.206887 29498 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:21.049718 29484 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.168s
I20260709 13:09:21.050691 29411 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:21.330067 25391 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:09:21.020395 29394 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:21.228767 29553 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:21.229589 29501 raft_consensus.cc:697] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:21.188640 29468 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:21.210405 29506 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:21.257448 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:21.257956 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:21.338346 25313 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:21.311898 29487 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:21.258589 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:21.259811 25312 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:09:21.260366 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:21.277115 29555 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:21.265236 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:21.266398 29383 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:21.267987 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:21.353996 25240 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:21.270674 29427 raft_consensus.cc:3055] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:21.208300 29415 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:21.231871 29524 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:21.230611 29143 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:21.284801 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:21.234097 29481 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:21.303963 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1777) took 1020 ms. Trace:
I20260709 13:09:20.947237 29485 consensus_queue.cc:237] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:21.232389 29513 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:20.974931 29499 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:21.293063 25285 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:21.385658 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 92717.9us
I20260709 13:09:21.387461 25285 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:20.975133 29542 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:21.289572 29577 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:09:21.314124 25242 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:21.192513 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:21.411935 25938 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:21.316166 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:20.984069 29508 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:21.332326 29364 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:21.251252 29491 consensus_queue.cc:1059] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.071s
I20260709 13:09:21.309954 29312 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:21.356510 29281 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 13:09:21.258255 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:21.327481 29390 consensus_meta.cc:287] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.551s	user 0.008s	sys 0.001s
I20260709 13:09:21.337641 25314 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
W20260709 13:09:21.326429 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 102370us
I20260709 13:09:21.371526 29481 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:21.343786 29501 consensus_queue.cc:237] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:21.344380 29482 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:21.344758 25317 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:09:21.262079 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:21.462037 25316 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:21.345173 29394 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:21.345649 25287 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:21.345988 29553 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:21.348037 29418 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:21.340098 29506 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:21.349354 29520 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:09:21.350652 29579 log.cc:927] Time spent T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
W20260709 13:09:21.355317 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 232631us
I20260709 13:09:21.360077 25313 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:21.324261 29580 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:09:21.369884 25284 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.052s	sys 0.053s
I20260709 13:09:21.366024 25386 rpcz_store.cc:276] 0709 13:09:20.283924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:20.284099 (+   175us) service_pool.cc:224] Handling call
0709 13:09:20.284428 (+   329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:20.284767 (+   339us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:20.284792 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:20.287818 (+  3026us) log.cc:844] Serialized 26 byte log entry
0709 13:09:20.394236 (+106418us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:20.394285 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:20.394366 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:21.303740 (+909374us) raft_consensus.cc:1685] finished
0709 13:09:21.303766 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:21.303904 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":89,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":2706,"thread_start_us":90616,"threads_started":1,"wal-append.queue_time_us":188037}
I20260709 13:09:21.442488 29546 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:21.326802 29101 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:21.330889 29498 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:21.417711 29513 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:21.325891 29536 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:21.342501 29518 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:09:21.249279 29554 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:21.438951 25212 leader_election.cc:336] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.654s (SENT)
I20260709 13:09:21.428042 25285 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:21.462949 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:21.463388 25938 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:21.494931 25212 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:21.440299 25910 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.000s	sys 0.015s
I20260709 13:09:21.415284 29525 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:21.423599 29312 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:21.477249 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 131658us
W20260709 13:09:21.485023 25284 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.053s	sys 0.054s
I20260709 13:09:21.356320 29427 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:21.362002 29415 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:21.485656 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 240366us
I20260709 13:09:21.425114 29502 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.060s
I20260709 13:09:21.416996 29383 consensus_queue.cc:1059] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.061s
W20260709 13:09:21.424621 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 310593us
I20260709 13:09:21.460302 29390 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:21.490141 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:21.509622 25358 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:21.365326 29524 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:21.481789 29367 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:21.421769 29499 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:21.526207 29499 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:21.527648 25387 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:21.529263 29513 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:21.529889 29508 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:21.530325 29508 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:21.530725 29508 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:21.531208 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:21.497663 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 69697.9us
I20260709 13:09:21.531860 25390 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
W20260709 13:09:21.498389 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 59471.7us
W20260709 13:09:21.537273 25212 leader_election.cc:336] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.686s (SENT)
I20260709 13:09:21.537693 25212 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:21.499892 29525 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:21.475729 29482 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:21.482048 29506 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:21.545570 29491 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:21.546198 29484 raft_consensus.cc:3055] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:21.546602 29484 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:21.547214 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:21.547811 25245 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:21.528954 29415 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:21.549930 29394 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:21.550196 25212 leader_election.cc:336] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.724s (SENT)
W20260709 13:09:21.551059 25212 leader_election.cc:336] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.800s (SENT)
I20260709 13:09:21.551394 25212 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:21.552120 29580 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:21.552812 29580 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:21.554870 29524 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:21.555608 25238 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
W20260709 13:09:21.555850 29589 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260709 13:09:21.505409 25943 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:09:21.506690 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:21.558887 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 52292.6us
I20260709 13:09:21.487805 29546 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:09:21.495762 25910 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.775s	user 0.000s	sys 0.016s
I20260709 13:09:21.511868 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:21.514168 25388 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
W20260709 13:09:21.513978 25239 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32994 (request call id 2069) took 2248 ms (client timeout 2106 ms). Trace:
I20260709 13:09:21.559609 25909 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:21.516218 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:21.561160 25908 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:21.519980 25320 raft_consensus.cc:2410] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 12 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 13:09:21.522003 25358 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:21.496474 29530 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:21.532612 29511 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:09:21.539876 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:21.540143 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:21.540345 29590 log.cc:927] Time spent T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
W20260709 13:09:21.541004 29587 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:09:21.541841 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:21.499086 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 712965us
I20260709 13:09:21.572753 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:21.561414 25392 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:21.575232 25284 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:21.538311 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:21.562503 29427 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:21.563025 29555 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:21.562520 29546 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:21.584849 25390 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 12: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:21.509618 29553 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:21.506697 29535 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:21.565862 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:21.566416 25394 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10. Ops: []
W20260709 13:09:21.566519 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 44605.5us
I20260709 13:09:21.560483 25393 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
W20260709 13:09:21.560601 25239 rpcz_store.cc:269] 0709 13:09:19.255399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:19.351974 (+ 96575us) service_pool.cc:224] Handling call
0709 13:09:21.503844 (+2151870us) spinlock_profiling.cc:243] Waited 887 ms on lock 0x7b5400192654. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:21.503994 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2039826304}
I20260709 13:09:21.572045 25242 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:21.535156 29367 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:21.579504 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:21.488348 29520 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:21.559603 29525 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:21.499311 29592 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.077s
I20260709 13:09:21.568205 29394 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:21.594090 25284 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:21.592160 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:21.602155 25390 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:21.595808 25909 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:21.595269 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:21.603605 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 47919.7us
I20260709 13:09:21.606192 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:21.606868 25390 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:21.610558 29337 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:21.611570 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:21.612661 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:21.612582 29486 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:21.613601 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:21.614672 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:21.614766 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:21.615540 25316 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
W20260709 13:09:21.615512 25212 leader_election.cc:336] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.933s (SENT)
I20260709 13:09:21.616142 25212 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:21.616977 29337 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:21.618551 29520 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:21.623805 29468 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:21.637703 25389 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:09:21.638397 29584 log.cc:927] Time spent T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.244s	user 0.001s	sys 0.000s
I20260709 13:09:21.639462 29364 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:21.639657 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:21.612865 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:21.642498 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:21.651099 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:21.587330 29553 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:09:21.657507 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 61762.4us
I20260709 13:09:21.664690 25909 leader_election.cc:400] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10.
I20260709 13:09:21.665019 25909 leader_election.cc:403] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 13:09:21.657951 25245 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:21.596781 29367 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:21.676600 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:21.643265 25390 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:21.675354 29519 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:21.681262 29582 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:21.681715 29582 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:21.682030 29582 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:21.683187 29312 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:21.685550 29491 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10.
I20260709 13:09:21.686264 29491 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:21.686600 29491 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:21.686967 29491 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:09:21.687785 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:21.687382 29367 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:21.690043 29390 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:21.690783 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:21.691567 29367 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:21.606410 29594 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:09:21.699867 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 103770us
I20260709 13:09:21.599292 29526 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:21.598008 29221 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:21.700840 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:21.603215 29427 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:09:21.704972 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:21.709841 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 148770us
I20260709 13:09:21.713567 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:21.676401 29553 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:21.714010 25908 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:21.718992 25390 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 11
W20260709 13:09:21.717491 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 50288.3us
I20260709 13:09:21.746631 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:21.699012 29591 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:09:21.747962 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 44024.9us
I20260709 13:09:21.749800 25909 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:21.714706 29605 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:21.753342 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:21.748716 25388 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:21.770542 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:21.758028 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:21.767465 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:21.772246 25938 raft_consensus.cc:2393] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:21.712311 29482 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:21.782061 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:21.783475 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:21.736835 29390 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:09:21.737977 29519 raft_consensus.cc:697] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:21.788280 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:21.786990 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:21.789994 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:21.791297 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:21.754848 29101 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:21.758814 29415 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:21.686619 29500 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:21.800153 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:21.800352 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:21.800662 25942 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:09:21.758143 29312 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:21.794787 25937 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:21.804231 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:21.813103 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:21.779251 29594 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:21.814622 25312 raft_consensus.cc:2410] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 13:09:21.801373 29390 raft_consensus.cc:697] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:21.816891 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:21.778930 29526 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.023s
I20260709 13:09:21.817792 29519 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:21.831377 25937 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:21.826637 29567 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:21.866120 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:21.866817 25245 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:21.867827 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:21.868592 25212 leader_election.cc:336] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.834s (SENT)
I20260709 13:09:21.835960 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:21.799108 29605 raft_consensus.cc:697] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:21.869827 25944 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12. Ops: [11.11-11.11]
I20260709 13:09:21.869961 25939 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 11: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:21.873834 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:21.869581 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:21.892120 25391 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:21.828226 29536 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:21.862661 29598 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.145s	user 0.000s	sys 0.000s
I20260709 13:09:21.776326 29591 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:21.899753 25319 raft_consensus.cc:2368] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11.
I20260709 13:09:21.870633 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:21.906272 25212 leader_election.cc:336] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.939s (SENT)
I20260709 13:09:21.861526 29468 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:21.904378 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:21.828083 29593 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:21.828649 29594 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:09:21.890908 29595 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.319s	user 0.001s	sys 0.000s
I20260709 13:09:21.913836 25911 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:21.870864 29498 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:21.904984 29544 consensus_meta.cc:287] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.570s	user 0.005s	sys 0.005s
I20260709 13:09:21.869526 29530 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 13:09:21.929538 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:21.929205 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:21.908002 29536 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:09:21.931919 29602 log.cc:927] Time spent T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.172s	user 0.000s	sys 0.001s
W20260709 13:09:21.934857 29143 consensus_meta.cc:287] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.553s	user 0.009s	sys 0.000s
I20260709 13:09:21.936138 29508 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:21.937484 29603 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 13:09:21.937245 29390 consensus_queue.cc:237] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:21.960242 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:21.943113 29599 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
I20260709 13:09:21.927671 29544 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:21.963338 29544 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:21.963541 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:21.960842 29508 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:21.967096 29518 consensus_queue.cc:1059] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:09:21.968274 29518 raft_consensus.cc:3055] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:21.968611 29518 raft_consensus.cc:740] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:21.968799 29286 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:21.953678 29143 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:21.983001 25212 leader_election.cc:336] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.142s (SENT)
I20260709 13:09:21.984094 25212 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:21.875239 29525 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:21.955974 25390 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:21.957008 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:21.950047 29500 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:21.970783 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.064868 25394 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 10: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:21.922215 29594 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:21.972668 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:21.973500 29614 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:09:22.083659 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 123476us
I20260709 13:09:22.021454 25314 raft_consensus.cc:2368] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12.
I20260709 13:09:22.062768 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:22.084569 25240 raft_consensus.cc:2393] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:22.070091 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:22.085431 25316 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:22.085747 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.086283 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:21.915434 29593 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:22.072314 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:22.074488 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:22.079214 29143 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:22.079628 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 96650.7us
I20260709 13:09:21.974396 29506 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:22.086725 25316 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:22.083350 29614 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:22.095366 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:09:22.097173 25212 leader_election.cc:336] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.167s (SENT)
I20260709 13:09:22.086925 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:22.082773 25937 consensus_queue.cc:237] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:22.102144 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:22.104822 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:22.106185 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.107342 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.107857 25394 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:22.109395 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:22.076833 29424 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.014s
I20260709 13:09:22.112483 25312 raft_consensus.cc:2368] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14.
I20260709 13:09:22.113250 29614 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:22.117908 29612 log.cc:927] Time spent T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.190s	user 0.000s	sys 0.000s
I20260709 13:09:22.119256 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:22.119607 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:22.094751 29140 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:22.100149 29536 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:22.124365 29536 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:22.124788 29536 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:22.070705 29491 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:22.096714 29605 consensus_queue.cc:237] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:22.131296 29491 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:22.131935 29518 consensus_queue.cc:260] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:22.133288 29518 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:21.959864 29481 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:22.134747 29553 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:22.136343 29481 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:22.136716 29553 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:22.137573 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.137921 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:22.138564 25314 raft_consensus.cc:2368] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14.
I20260709 13:09:22.141440 25240 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:09:22.141844 25212 leader_election.cc:336] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.218s (SENT)
I20260709 13:09:22.142243 25212 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:22.138769 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.144311 25395 raft_consensus.cc:2393] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:22.137573 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:22.145788 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.147567 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.149035 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.144600 25287 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:22.151319 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.152102 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.153189 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.154084 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.154346 25287 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:22.154685 25395 raft_consensus.cc:2368] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11.
I20260709 13:09:22.155814 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.072758 29394 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:22.157394 25320 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:22.163514 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:22.065629 29508 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:22.158586 25395 raft_consensus.cc:2393] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:09:22.158660 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:22.160429 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:22.160848 25287 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:22.161379 25388 raft_consensus.cc:2393] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 12: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:22.083047 29530 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:22.165299 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 80235.4us
I20260709 13:09:22.180851 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:22.182204 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:22.182684 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.183970 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:22.167558 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.097009 29582 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:22.157390 29143 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:22.210862 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.189373 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:22.218274 25388 raft_consensus.cc:2393] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:22.104854 29506 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:22.167985 29553 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:22.173063 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:22.220826 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:22.221866 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:22.222461 25317 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:22.223434 25314 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:22.188341 25394 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:22.172629 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.139132 29546 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:22.225073 25395 raft_consensus.cc:2376] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9. Re-sending same reply.
I20260709 13:09:22.171319 25240 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:22.189962 29580 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:22.189885 29481 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:22.190047 29535 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:22.239774 29546 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:22.177665 29508 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:22.243577 29508 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:22.178428 29140 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:09:22.245447 29567 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:22.247275 29591 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:22.248654 29593 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:22.171896 29555 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:22.252159 29525 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:22.172672 29424 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:22.257018 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:22.218865 25320 raft_consensus.cc:2393] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 9: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:22.219086 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.226805 25319 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:22.231281 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.261193 25316 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:09:22.232648 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.267014 25395 raft_consensus.cc:2393] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 10: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:09:22.234555 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 92733.4us
I20260709 13:09:22.267191 25312 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9. Ops: []
W20260709 13:09:22.268652 25212 leader_election.cc:336] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.140s (SENT)
I20260709 13:09:22.269245 25390 raft_consensus.cc:2376] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10. Re-sending same reply.
I20260709 13:09:22.178493 29394 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:22.236748 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.261545 25388 raft_consensus.cc:2393] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:22.269099 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:22.232683 29481 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:09:22.277421 25212 leader_election.cc:336] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.730s (SENT)
I20260709 13:09:22.270264 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.232577 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.223940 29553 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:22.283795 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.280807 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.279207 25319 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:22.197830 29530 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:22.267118 29500 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:22.282786 29143 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:22.289808 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.290342 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.312304 25390 raft_consensus.cc:2393] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:22.275766 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.310943 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.313576 25394 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:22.312883 25395 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:22.318009 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.284907 29553 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:22.320128 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.282639 29481 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:22.327497 25908 leader_election.cc:336] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.841s (SENT)
I20260709 13:09:22.327679 25242 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:22.330631 25908 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:22.328102 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.328837 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:22.331576 25390 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:22.331765 25316 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:22.332221 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.333247 25395 raft_consensus.cc:2393] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:22.336184 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.336913 25388 raft_consensus.cc:2393] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 10: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:22.337677 25908 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:22.341792 29498 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:22.342211 29620 log.cc:927] Time spent T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 13:09:22.342391 29498 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:22.343374 29567 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:09:22.344539 29498 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:22.347892 25392 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.011s	sys 0.002s
W20260709 13:09:22.349457 25392 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.012s	sys 0.002s
I20260709 13:09:22.350366 25394 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:22.356254 25390 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:22.360193 25908 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:22.354516 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:22.362545 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.362962 25319 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:22.355409 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.364434 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.365634 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.366220 25390 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:09:22.368705 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.369578 25388 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:22.376199 25390 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:22.377687 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.383950 25395 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:22.385846 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.386670 25392 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:09:22.393819 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 208740us
I20260709 13:09:22.391474 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:22.396056 25316 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:22.395098 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:22.404090 25394 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:09:22.404912 25209 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:22.367668 29567 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:22.406895 25209 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:22.414312 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:22.386917 29523 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:22.415215 25936 raft_consensus.cc:2393] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:22.416635 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:22.417398 25937 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:22.385617 29526 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:22.425355 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.406874 29583 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:22.410876 29555 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:22.410425 29580 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:22.429570 25394 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:09:22.431783 25388 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:22.440232 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:22.436146 25212 leader_election.cc:336] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.584s (SENT)
I20260709 13:09:22.441315 25212 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:22.449752 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:22.450593 25936 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:22.421774 29140 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:22.481460 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.437556 29555 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:22.475013 29140 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:22.487533 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:22.484467 25936 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: []
W20260709 13:09:22.484938 29600 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.819s	user 0.003s	sys 0.000s
I20260709 13:09:22.454864 29143 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:22.490509 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.498668 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:22.482896 29582 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:22.489318 29555 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:22.430593 29523 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:22.503087 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.509042 25209 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:22.501698 25212 leader_election.cc:336] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.242s (SENT)
I20260709 13:09:22.513513 25212 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:22.514483 29622 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:09:22.514865 25212 leader_election.cc:336] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.662s (SENT)
I20260709 13:09:22.514889 29622 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:22.515556 29622 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:22.517932 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.518885 29143 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:22.490468 29140 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:22.515522 25212 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:22.525399 25388 raft_consensus.cc:2393] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 10: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:22.521878 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:22.527845 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.528615 25394 raft_consensus.cc:2393] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:22.528431 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:22.546795 25285 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:22.455138 29594 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:22.554554 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:22.562803 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:22.563653 25941 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:22.564556 25212 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:22.577813 25285 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:22.586318 25212 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:22.583642 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:22.600128 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:22.601506 29527 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:22.602126 29527 raft_consensus.cc:697] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:22.601714 29484 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:22.603014 29527 consensus_queue.cc:237] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:22.607415 25212 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:22.608440 29535 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:22.609193 29535 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:22.609730 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2033) took 1291 ms. Trace:
W20260709 13:09:22.610867 25359 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.009s	sys 0.020s
I20260709 13:09:22.612205 25318 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:22.612018 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:22.612071 25359 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.009s	sys 0.021s
W20260709 13:09:22.613222 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 546763us
I20260709 13:09:22.538470 29143 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:22.615672 25943 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:22.616845 25938 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:22.619637 25945 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 10: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:22.609833 29304 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:22.624683 25392 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:09:22.510360 29536 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:22.627326 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:22.627933 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:22.631222 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.631810 25392 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:22.610229 25244 rpcz_store.cc:276] 0709 13:09:21.291201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:21.338246 (+ 47045us) service_pool.cc:224] Handling call
0709 13:09:21.357819 (+ 19573us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:21.358277 (+   458us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:21.358323 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:21.358896 (+   573us) log.cc:844] Serialized 26 byte log entry
0709 13:09:21.493490 (+134594us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:09:21.493535 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:21.493614 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:22.484610 (+990996us) spinlock_profiling.cc:243] Waited 3.72 ms on lock 0x7b2400120638. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973a4b9c4 00007f2973a49acb 00007f2973c04dc9 00007f2973c04a55 00007f2973c0409d 00007f297683610e 00007f297684d88c 00007f297684cbb6 00007f297684c7d8 00007f2976856221
0709 13:09:22.549952 (+ 65342us) raft_consensus.cc:1685] finished
0709 13:09:22.549973 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:22.582391 (+ 32418us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":118,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":5560,"spinlock_wait_cycles":8555392,"thread_start_us":78612,"threads_started":1,"wal-append.queue_time_us":199481}
I20260709 13:09:22.618580 29544 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:22.640733 29544 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:22.641708 29631 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
W20260709 13:09:22.642828 29623 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
W20260709 13:09:22.648131 29630 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
I20260709 13:09:22.649324 29304 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:22.650240 29619 log.cc:927] Time spent T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.175s	user 0.000s	sys 0.000s
W20260709 13:09:22.651711 29628 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.173s	user 0.000s	sys 0.000s
I20260709 13:09:22.651871 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.653074 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:22.653403 29624 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 13:09:22.653644 25394 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:22.654374 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 662) took 1398 ms. Trace:
I20260709 13:09:22.654945 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:22.654767 25321 rpcz_store.cc:276] 0709 13:09:21.256446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:21.261082 (+  4636us) service_pool.cc:224] Handling call
0709 13:09:21.261463 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:21.475632 (+214169us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:21.475661 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:21.475699 (+    38us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:21.475735 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:22.654150 (+1178415us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b5400045954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:22.654203 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:22.654317 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":492087168}
I20260709 13:09:22.657238 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:22.657856 25240 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:22.657948 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:22.597815 29525 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:22.664016 25939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 808) took 1539 ms. Trace:
I20260709 13:09:22.615191 29580 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:22.666771 25939 rpcz_store.cc:276] 0709 13:09:21.124419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:21.125471 (+  1052us) service_pool.cc:224] Handling call
0709 13:09:22.663811 (+1538340us) spinlock_profiling.cc:243] Waited 645 ms on lock 0x7b540062a2d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:22.663951 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1482858368}
W20260709 13:09:22.668710 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:22.678050 25243 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:09:22.679884 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:22.677980 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:22.685681 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:22.686108 25942 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 11. Ops: []
I20260709 13:09:22.689733 25238 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:22.694769 25394 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:22.706557 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:22.717317 25246 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:22.719612 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:22.715052 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.723953 25240 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:22.731159 29492 raft_consensus.cc:3055] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:22.737082 25392 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:22.747686 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.748070 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:22.748456 25392 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:22.748860 25394 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:09:22.705051 29614 consensus_meta.cc:287] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.591s	user 0.005s	sys 0.002s
I20260709 13:09:22.761807 25937 raft_consensus.cc:2410] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
W20260709 13:09:22.754678 25237 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 2010) took 2387 ms (client timeout 2098 ms). Trace:
W20260709 13:09:22.709800 29627 log.cc:927] Time spent T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.182s	user 0.000s	sys 0.000s
I20260709 13:09:22.710798 29544 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:22.759439 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:22.778124 25238 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:22.778612 25395 raft_consensus.cc:3055] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:22.779217 25395 raft_consensus.cc:740] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:22.757687 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:22.772758 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 165401us
I20260709 13:09:22.773974 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:22.706287 29594 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:22.707860 29304 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:22.768497 25237 rpcz_store.cc:269] 0709 13:09:20.366855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:20.381983 (+ 15128us) service_pool.cc:224] Handling call
0709 13:09:22.754229 (+2372246us) spinlock_profiling.cc:243] Waited 462 ms on lock 0x7b540028bed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:22.754551 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1061452032}
I20260709 13:09:22.784302 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:22.791275 25238 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:22.792228 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:22.792903 25238 raft_consensus.cc:2393] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 9: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:22.793238 25392 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:09:22.788196 25244 raft_consensus.cc:2393] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 10: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:22.744867 29492 raft_consensus.cc:740] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:22.794646 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:22.798357 25238 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:22.795162 25212 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:22.783895 25285 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:22.816715 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:22.817788 25237 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:22.770974 25394 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:22.824697 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:22.825744 25242 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:22.829322 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:22.827148 25285 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:22.775848 29614 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:22.800663 25395 consensus_queue.cc:260] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:22.832444 25244 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 12
W20260709 13:09:22.840605 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 57818.9us
I20260709 13:09:22.841703 25212 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:22.766811 29530 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:22.842505 25395 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:22.845639 25237 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:22.847014 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:22.847625 25237 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:22.708902 29523 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:22.847929 25285 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:22.830654 29544 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:22.827474 29535 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:22.866526 25237 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:09:22.899361 25285 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:22.881855 25242 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:09:22.841481 29580 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:22.904027 25395 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 7. (term mismatch)
I20260709 13:09:22.906884 25395 pending_rounds.cc:85] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 6
I20260709 13:09:22.907217 25395 pending_rounds.cc:107] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 9.7
I20260709 13:09:22.907442 25395 raft_consensus.cc:2889] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:22.908074 25244 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:22.858048 29364 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:22.915856 25242 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:09:22.917613 25242 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
W20260709 13:09:22.917816 29636 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 13:09:22.918767 29635 log.cc:927] Time spent T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 13:09:22.857461 29530 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:09:22.859602 29544 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:09:22.921998 29639 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 13:09:22.922861 25285 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:22.923627 29646 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:22.849115 29492 consensus_queue.cc:260] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:22.965993 29492 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:22.925623 29530 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:22.852558 29427 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:22.936201 25285 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:22.950800 25945 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 906) took 1167 ms. Trace:
I20260709 13:09:22.974975 25945 rpcz_store.cc:276] 0709 13:09:21.783423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:21.783537 (+   114us) service_pool.cc:224] Handling call
0709 13:09:22.950414 (+1166877us) spinlock_profiling.cc:243] Waited 819 ms on lock 0x7b5400675554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:22.950673 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1882560256}
I20260709 13:09:22.953343 29614 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:22.930382 29536 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:22.962427 29555 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:22.984445 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 142814us
I20260709 13:09:22.989734 25244 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:22.935134 29544 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:22.970100 29530 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:09:22.983438 29553 consensus_meta.cc:287] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.636s	user 0.006s	sys 0.006s
I20260709 13:09:22.969053 29542 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 13:09:22.986299 29481 consensus_meta.cc:287] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.629s	user 0.010s	sys 0.002s
I20260709 13:09:22.970820 29646 raft_consensus.cc:697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:22.976454 29427 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:22.998826 25212 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:22.932586 29364 raft_consensus.cc:697] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:22.984177 29545 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:22.920614 29580 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:23.021338 25212 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:22.995113 29644 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.030s
I20260709 13:09:23.009105 29536 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 13:09:23.006170 29647 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 13:09:23.066110 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 668) took 1807 ms. Trace:
W20260709 13:09:23.052465 25318 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 659) took 2906 ms (client timeout 1841 ms). Trace:
W20260709 13:09:23.081809 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:23.108518 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 87261.5us
W20260709 13:09:23.087767 25387 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41830 (request call id 810) took 2708 ms (client timeout 2598 ms). Trace:
W20260709 13:09:23.044417 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:23.109216 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:23.070930 25317 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 676) took 1491 ms (client timeout 1898 ms). Trace:
I20260709 13:09:23.110882 25908 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:23.112424 25908 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:23.057083 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:23.108955 25318 rpcz_store.cc:269] 0709 13:09:20.146427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:20.146546 (+   119us) service_pool.cc:224] Handling call
0709 13:09:23.052185 (+2905639us) spinlock_profiling.cc:243] Waited 2.34 s on lock 0x7b54000f9454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:23.052341 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5388284288}
W20260709 13:09:23.113711 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:23.110999 25317 rpcz_store.cc:269] 0709 13:09:21.579426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:21.579552 (+   126us) service_pool.cc:224] Handling call
0709 13:09:23.070710 (+1491158us) spinlock_profiling.cc:243] Waited 619 ms on lock 0x7b5400088ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:23.070828 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1423593344}
W20260709 13:09:23.109961 25387 rpcz_store.cc:269] 0709 13:09:20.379389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:20.383765 (+  4376us) service_pool.cc:224] Handling call
0709 13:09:23.087523 (+2703758us) spinlock_profiling.cc:243] Waited 562 ms on lock 0x7b540013d8d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:23.087650 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1292130048}
I20260709 13:09:23.108734 25315 rpcz_store.cc:276] 0709 13:09:21.258491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:21.262684 (+  4193us) service_pool.cc:224] Handling call
0709 13:09:21.263067 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:21.526869 (+263802us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:09:21.527179 (+   310us) log.cc:844] Serialized 14 byte log entry
0709 13:09:21.593873 (+ 66694us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:21.593940 (+    67us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:21.594024 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:23.065751 (+1471727us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b5400043dd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:23.065815 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:23.066021 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15592,"prepare.queue_time_us":6245,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":16004,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":1747,"spinlock_wait_cycles":606152064,"thread_start_us":51201,"threads_started":1,"wal-append.queue_time_us":180559,"wal-append.run_cpu_time_us":2436,"wal-append.run_wall_time_us":1246028}
I20260709 13:09:23.126375 25238 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:23.131469 25358 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:23.045087 29622 consensus_meta.cc:287] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.529s	user 0.008s	sys 0.004s
W20260709 13:09:23.049608 29140 consensus_meta.cc:287] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.530s	user 0.011s	sys 0.000s
W20260709 13:09:23.092911 29618 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.680s	user 0.000s	sys 0.003s
I20260709 13:09:23.034766 29143 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:09:23.053929 29629 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.537s	user 0.003s	sys 0.003s
I20260709 13:09:23.119949 25359 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:23.096940 29427 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:23.142508 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:23.158934 25945 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:23.142413 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:23.139976 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 185756us
I20260709 13:09:23.143144 25237 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:09:23.140637 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:23.081550 29519 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:23.082047 29304 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:23.023196 29643 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:23.011373 29553 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:23.164214 29659 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:23.175848 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:23.176278 29659 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:23.176623 29553 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:23.149708 25287 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:23.176895 29659 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:23.075889 29508 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:23.178120 29655 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:23.178205 29508 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:23.178756 29655 raft_consensus.cc:697] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:23.154012 29652 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:23.179314 29659 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:23.179710 29655 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:23.180404 29508 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:23.151976 29622 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:23.114612 29591 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:23.022111 29481 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:23.121359 29614 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:23.039386 29555 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:23.182575 29555 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:23.182821 29481 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:23.051813 29535 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:23.050429 29646 consensus_queue.cc:237] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:23.183753 29553 consensus_queue.cc:1059] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:09:23.183844 29535 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:23.185362 25287 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:23.186148 29535 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:23.125624 29418 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:23.186827 29304 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:23.187381 29418 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:23.151533 29140 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:23.190481 29418 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:23.191308 29418 raft_consensus.cc:3055] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:23.191509 29140 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:23.191865 29418 raft_consensus.cc:740] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:09:23.134009 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:23.193852 29541 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:23.194604 25387 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:23.194764 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:23.195343 29541 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:23.195698 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:23.125722 29523 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:23.174314 29519 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:23.170097 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:23.173610 29593 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:23.176685 25945 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 10: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:23.144896 29545 raft_consensus.cc:697] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:23.196329 25387 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:23.196655 29523 raft_consensus.cc:697] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:23.209267 29364 consensus_queue.cc:237] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:23.209514 29523 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:23.197109 29591 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:23.197875 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:23.216135 29418 consensus_queue.cc:260] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:23.217466 25394 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:23.217742 29418 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:23.220568 29586 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:23.221230 29586 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:23.223606 29586 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:23.225942 29591 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:23.198984 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:23.228602 25390 raft_consensus.cc:2376] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12. Re-sending same reply.
I20260709 13:09:23.202477 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:23.229976 25945 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:23.215261 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:23.230063 25243 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:23.230983 25394 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:23.199326 25940 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 11: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:23.230996 25387 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:23.233029 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:23.233574 25911 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:23.238665 25238 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:23.233996 25942 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:23.243046 25911 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:23.252108 25242 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:23.252382 25243 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:23.254462 25940 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:09:23.240497 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:23.203663 29580 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:23.199597 29519 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:23.200141 29593 raft_consensus.cc:697] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:23.171746 29585 consensus_queue.cc:1059] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.142s
I20260709 13:09:23.180516 29530 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:23.294710 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 174838us
W20260709 13:09:23.342891 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 202349us
I20260709 13:09:23.345693 25358 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:23.345422 25359 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:23.253775 29542 raft_consensus.cc:697] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:23.204452 29545 consensus_queue.cc:237] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:23.293036 29582 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:23.375890 25242 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:09:23.292567 29614 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
I20260709 13:09:23.354324 25285 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:23.292857 29304 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:23.309782 29594 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:23.361670 29652 raft_consensus.cc:697] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:23.343343 29492 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:23.314268 29527 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:23.398509 29527 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:23.401298 29665 consensus_queue.cc:1059] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:09:23.349241 29508 consensus_queue.cc:1059] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20260709 13:09:23.402570 29530 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:23.407181 25387 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 10. Current term is 11. Ops: []
W20260709 13:09:23.348290 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 238102us
I20260709 13:09:23.410203 25390 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:09:23.410719 25212 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:23.371455 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:09:23.382247 25389 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41830 (request call id 816) took 2254 ms (client timeout 2129 ms). Trace:
I20260709 13:09:23.392590 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:23.376586 29591 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:23.396662 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:23.397114 29594 raft_consensus.cc:697] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:23.418123 25212 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:23.413079 25389 rpcz_store.cc:269] 0709 13:09:21.127528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:21.127630 (+   102us) service_pool.cc:224] Handling call
0709 13:09:23.382006 (+2254376us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x7b54003237d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:23.382162 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1116091136}
I20260709 13:09:23.420225 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:23.422842 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:23.423774 25241 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:23.426126 29541 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:09:23.427459 25212 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:23.376323 29582 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:23.429893 29582 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:23.432080 29544 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:23.434037 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 88397.9us
I20260709 13:09:23.434634 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:23.435367 25242 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:23.434716 25212 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:23.435829 25358 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:23.400966 29527 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:23.437016 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:23.460144 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:23.477011 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 236576us
I20260709 13:09:23.423023 29585 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:23.464301 29394 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:23.498917 29500 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:23.419376 29523 consensus_queue.cc:1059] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 13:09:23.501519 29542 consensus_queue.cc:237] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:23.437011 29580 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:09:23.507943 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:23.504993 29582 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:23.422719 29513 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.070s
I20260709 13:09:23.436731 29668 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:23.433229 29593 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:23.493383 29622 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:23.516819 29652 consensus_queue.cc:237] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:23.523756 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 46949.8us
W20260709 13:09:23.533730 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 348431us
W20260709 13:09:23.539064 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 103933us
I20260709 13:09:23.577112 25359 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:23.586094 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:23.534078 25242 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:23.547080 29500 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:23.517894 29427 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:23.595382 25942 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 13:09:23.547313 29664 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:23.547850 29585 raft_consensus.cc:697] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 13 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:23.637660 25394 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 13:09:23.641080 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 55127.5us
W20260709 13:09:23.642606 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 207939us
I20260709 13:09:23.643587 25358 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:23.647326 29530 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:23.651249 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:23.653162 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:23.653602 29544 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:23.653517 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:23.577257 29508 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:23.727744 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:23.729951 25245 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:23.728542 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:23.665669 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 87819.5us
I20260709 13:09:23.666174 29683 raft_consensus.cc:3055] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:23.683506 29686 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:23.709842 29492 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:23.686120 25240 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:23.686546 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 748) took 1009 ms. Trace:
I20260709 13:09:23.747691 25243 rpcz_store.cc:276] 0709 13:09:22.677624 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:22.679534 (+  1910us) service_pool.cc:224] Handling call
0709 13:09:23.685784 (+1006250us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x7b5400456854. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:23.686476 (+   692us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1263379712}
I20260709 13:09:23.686916 29661 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:23.666589 29591 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:09:23.694950 29684 log.cc:927] Time spent T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 13:09:23.677714 29304 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:23.564586 29594 consensus_queue.cc:237] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:23.715719 29427 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:23.685700 29670 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
I20260709 13:09:23.737692 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2372) took 1308 ms. Trace:
I20260709 13:09:23.738826 25242 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:09:23.737211 29683 raft_consensus.cc:740] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:23.761272 25393 rpcz_store.cc:276] 0709 13:09:22.419161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:22.428355 (+  9194us) service_pool.cc:224] Handling call
0709 13:09:22.453911 (+ 25556us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:22.454198 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:22.454236 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:22.486874 (+ 32638us) log.cc:844] Serialized 26 byte log entry
0709 13:09:22.626949 (+140075us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:09:23.726806 (+1099857us) log.cc:844] Serialized 14 byte log entry
0709 13:09:23.727023 (+   217us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:23.727113 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:23.727154 (+    41us) raft_consensus.cc:1685] finished
0709 13:09:23.727169 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:23.727275 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1087299,"raft.queue_time_us":10001,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":119,"thread_start_us":105014,"threads_started":1,"wal-append.queue_time_us":225596}
I20260709 13:09:23.745857 29536 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:09:23.746594 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 103087us
I20260709 13:09:23.726259 29554 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:23.763953 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:23.778789 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 127688us
I20260709 13:09:23.767434 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:23.775945 25358 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:23.777237 25245 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:23.726883 29687 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:23.790436 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:23.803613 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:23.787550 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:23.806049 25321 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:23.803318 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:23.789521 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:23.807974 25317 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:23.809294 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:23.809912 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:23.810127 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:23.821925 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:23.754474 29661 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:23.797001 25315 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:23.901070 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:23.896692 25245 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 12: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:23.899885 25242 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:23.910044 25317 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
W20260709 13:09:23.916106 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 140270us
I20260709 13:09:23.917299 25358 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:23.922859 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:23.926538 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 349486us
I20260709 13:09:23.926819 25358 leader_election.cc:400] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11.
I20260709 13:09:23.927443 25358 leader_election.cc:403] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:09:23.929178 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:23.929649 25358 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:23.930126 25313 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:23.931378 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:23.932911 25358 leader_election.cc:400] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12.
I20260709 13:09:23.933519 25358 leader_election.cc:403] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 13:09:23.785373 29686 raft_consensus.cc:697] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:23.759080 29492 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:23.790058 29643 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:23.802517 29668 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:23.802729 29508 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:09:23.824960 29640 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 13:09:23.901551 29661 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:23.911365 29652 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:23.707321 29585 consensus_queue.cc:237] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:23.981225 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 48496.4us
I20260709 13:09:23.998677 25358 leader_election.cc:400] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14.
I20260709 13:09:23.999032 25358 leader_election.cc:403] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:09:23.802292 29580 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:23.785490 29677 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.206s
I20260709 13:09:23.760720 29304 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:23.790313 29536 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:24.007119 29668 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:23.945291 29688 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:09:23.947830 29649 log.cc:927] Time spent T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
I20260709 13:09:23.945031 29591 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11.
I20260709 13:09:23.987505 29687 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:23.938777 29593 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:20.453111 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:09:24.036391 29680 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.537s	user 0.000s	sys 0.010s
I20260709 13:09:24.038676 29101 raft_consensus.cc:3055] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 13:09:24.038815 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 300666us
I20260709 13:09:24.021394 29143 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:24.045140 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1798) took 2173 ms. Trace:
I20260709 13:09:23.978956 29683 consensus_queue.cc:260] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:23.938027 29691 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.106s
I20260709 13:09:24.062790 29536 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:24.042583 29582 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:24.074735 25391 rpcz_store.cc:276] 0709 13:09:21.871386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:21.921476 (+ 50090us) service_pool.cc:224] Handling call
0709 13:09:21.940307 (+ 18831us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:22.139723 (+199416us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:09:22.407061 (+267338us) log.cc:844] Serialized 14 byte log entry
0709 13:09:23.579858 (+1172797us) spinlock_profiling.cc:243] Waited 1.82 ms on lock 0x7b18004b71e0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b3fc0a 00007f297d27f219 00007f2973bc7a40 00007f2973bc24e4 00007f2973bc2455 00007f297a6aa3c7 00007f2976cbc9a7 00007f2976ccc0bf 00007f2976ccc038
0709 13:09:23.580143 (+   285us) log.cc:844] Serialized 14 byte log entry
0709 13:09:23.607386 (+ 27243us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:23.607467 (+    81us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:23.607546 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:24.044729 (+437183us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b5400282dd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:24.044796 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:24.044995 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1368706,"prepare.queue_time_us":1588530,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":7244,"raft.queue_time_us":98568,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":6368,"spinlock_wait_cycles":462209152,"thread_start_us":317934,"threads_started":2,"wal-append.queue_time_us":111273}
W20260709 13:09:24.091063 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:24.050679 29668 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:24.014221 29580 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:24.112540 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1803) took 1498 ms. Trace:
I20260709 13:09:24.010457 29652 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:24.069753 29101 raft_consensus.cc:740] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:24.113060 25388 rpcz_store.cc:276] 0709 13:09:22.616980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:22.618213 (+  1233us) service_pool.cc:224] Handling call
0709 13:09:22.657898 (+ 39685us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:22.658137 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:09:23.988867 (+1330730us) log.cc:844] Serialized 14 byte log entry
0709 13:09:24.112123 (+123256us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:24.112199 (+    76us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:24.112258 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:24.112335 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:24.112475 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1095888,"prepare.queue_time_us":1292897,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":6144,"thread_start_us":365022,"threads_started":2}
W20260709 13:09:24.122095 25209 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.640s (SENT) [suppressed 37 similar messages]
I20260709 13:09:24.081804 29687 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12.
I20260709 13:09:24.110239 29683 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:24.111172 29688 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14.
I20260709 13:09:24.059054 29690 consensus_queue.cc:1059] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
I20260709 13:09:24.118676 29659 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:09:24.144987 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 193056us
I20260709 13:09:24.084215 29143 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:24.148428 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:24.124943 29530 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:24.155045 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:24.158474 25390 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:09:24.162407 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:24.163334 25391 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:24.167616 29143 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:24.169188 29304 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:24.174157 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:24.174827 25388 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:24.080540 29686 consensus_queue.cc:237] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:24.189797 29692 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.186s	user 0.002s	sys 0.000s
W20260709 13:09:24.204843 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 206332us
I20260709 13:09:24.210554 25358 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:24.212846 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:24.208496 29500 consensus_meta.cc:287] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.578s	user 0.004s	sys 0.009s
I20260709 13:09:24.218501 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1812) took 1586 ms. Trace:
I20260709 13:09:24.218519 25358 leader_election.cc:400] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14.
I20260709 13:09:24.220546 25358 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:24.201478 29143 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:24.219380 25392 rpcz_store.cc:276] 0709 13:09:22.632412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:22.794262 (+161850us) service_pool.cc:224] Handling call
0709 13:09:22.794760 (+   498us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:22.796476 (+  1716us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:09:23.953187 (+1156711us) log.cc:844] Serialized 14 byte log entry
0709 13:09:24.217873 (+264686us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:24.217946 (+    73us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:09:24.218037 (+    91us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:24.218203 (+   166us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:24.218407 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":961208,"prepare.queue_time_us":1182363,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":5630,"raft.queue_time_us":1058,"thread_start_us":470476,"threads_started":2}
I20260709 13:09:24.225752 25388 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:24.209535 29492 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:09:24.163012 29101 consensus_queue.cc:260] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:24.217986 29690 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:24.218921 29591 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:24.218541 29500 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:24.229467 29143 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:24.244684 29500 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:24.244962 29143 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:24.233376 29704 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:24.233052 29593 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260709 13:09:24.246454 29101 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:24.264050 29693 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
W20260709 13:09:24.286366 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 164295us
W20260709 13:09:24.286788 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:24.314165 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 80418.2us
W20260709 13:09:24.281744 29697 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 13:09:24.316044 29593 consensus_queue.cc:1059] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 13:09:24.316463 29698 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260709 13:09:24.281940 29704 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:24.318670 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:24.288589 25284 connection.cc:256] RPC call timeout handler was delayed by -1.6379159739998386s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2068s for a response
I20260709 13:09:24.320978 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:24.321441 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:24.290818 25908 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.007s	sys 0.000s
W20260709 13:09:24.323866 25908 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.007s	sys 0.000s
I20260709 13:09:24.321444 25391 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
W20260709 13:09:24.324615 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:24.296216 25395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 840) took 1100 ms. Trace:
W20260709 13:09:24.309840 25909 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.000s	sys 0.008s
W20260709 13:09:24.325453 25909 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.001s	sys 0.008s
I20260709 13:09:24.328092 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:24.330307 25908 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:24.325343 25395 rpcz_store.cc:276] 0709 13:09:23.195646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:23.195815 (+   169us) service_pool.cc:224] Handling call
0709 13:09:23.196066 (+   251us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:23.196287 (+   221us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:09:24.292326 (+1096039us) log.cc:844] Serialized 14 byte log entry
0709 13:09:24.295830 (+  3504us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:24.295891 (+    61us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:24.295942 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:24.296007 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:24.296151 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":578832,"prepare.queue_time_us":823649,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":13296,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":85,"thread_start_us":491027,"threads_started":3,"wal-append.queue_time_us":1627}
I20260709 13:09:24.332502 29585 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:24.332680 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:24.332748 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:24.333268 29585 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:24.334551 29721 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:24.310935 29580 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:24.333011 25240 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:09:24.337733 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:24.338697 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:24.340227 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:24.344129 29542 consensus_queue.cc:1059] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:09:24.344324 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:24.344762 29731 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:24.345381 29500 raft_consensus.cc:3055] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:24.345742 29500 raft_consensus.cc:740] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:24.350317 25284 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:24.348120 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:24.382506 25358 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:24.382640 25284 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:24.347754 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:24.384347 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 54137.5us
I20260709 13:09:24.415915 25908 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:24.385448 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:24.417033 25938 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:24.335409 29585 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:24.335108 29721 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:24.328444 29722 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:24.408059 29694 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:09:24.431907 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 49474.7us
I20260709 13:09:24.389029 29591 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:24.425874 25908 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:24.439819 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:24.387403 29554 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:24.433161 29708 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.199s	user 0.001s	sys 0.000s
I20260709 13:09:24.340059 29304 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:24.423193 29677 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:24.387905 29523 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:09:24.414834 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 57419.9us
I20260709 13:09:24.423694 29580 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:24.336942 29506 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:24.433604 29694 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:24.435688 29731 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:24.523392 29683 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:24.529628 29506 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:24.436244 29337 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:24.421335 29500 consensus_queue.cc:260] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:24.532554 29716 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 13:09:24.532440 29721 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:24.533203 29706 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.248s	user 0.001s	sys 0.000s
I20260709 13:09:24.432180 29585 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.094s
I20260709 13:09:24.444594 29690 raft_consensus.cc:697] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:24.387557 29704 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:24.551731 29690 consensus_queue.cc:237] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:24.533914 29523 raft_consensus.cc:697] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:24.537272 29731 raft_consensus.cc:697] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:24.523099 29622 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:09:24.542953 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 127923us
I20260709 13:09:24.544930 29732 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:24.413645 29723 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.150s
I20260709 13:09:24.526104 29677 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:24.562358 29677 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:24.534982 29694 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:24.565976 29731 consensus_queue.cc:237] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:24.548432 29721 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:24.538885 29712 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:09:24.527693 29304 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:24.583896 25314 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:24.584672 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:24.530316 29580 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:24.555289 29500 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:24.569943 29717 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.227s	user 0.000s	sys 0.001s
I20260709 13:09:24.541378 29683 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:24.595261 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:24.597180 25320 raft_consensus.cc:2393] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:24.543473 29337 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:24.601914 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 176138us
I20260709 13:09:24.570256 29622 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:24.598357 29664 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:24.608482 29661 consensus_meta.cc:287] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.596s	user 0.011s	sys 0.003s
I20260709 13:09:24.614467 29580 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:24.615473 29484 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:24.627363 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:24.615778 29506 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:24.618330 25908 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:24.618367 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:24.622776 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:24.622973 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:24.628362 25395 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:24.630692 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:24.638266 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:24.639920 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:24.588701 29523 consensus_queue.cc:237] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:24.635005 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:24.641002 25312 raft_consensus.cc:2368] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12.
W20260709 13:09:24.645045 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:24.645749 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:24.646457 25395 raft_consensus.cc:2368] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12.
I20260709 13:09:24.647173 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:24.647187 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:24.648201 25319 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:24.637082 25394 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
W20260709 13:09:24.641724 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 116610us
I20260709 13:09:24.643332 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:24.643213 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:24.649408 25313 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:24.648264 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:24.632994 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:24.652388 25312 raft_consensus.cc:2368] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11.
I20260709 13:09:24.651369 25392 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 13:09:24.654664 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:24.655874 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:24.624012 29591 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:24.662151 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:24.662876 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:24.641924 29519 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:24.667481 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:24.660316 25911 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:24.666194 25940 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:09:24.671105 25945 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:24.676818 25943 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:09:24.677706 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:24.620568 29661 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:24.673007 25944 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 13:09:24.675396 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:24.679554 25312 raft_consensus.cc:2368] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11.
I20260709 13:09:24.636066 29664 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:24.663748 29506 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:24.697172 25938 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:09:24.682843 25940 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:24.740056 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:24.751127 25939 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:24.695641 29643 consensus_meta.cc:287] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.703s	user 0.012s	sys 0.001s
I20260709 13:09:24.671412 29734 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:24.771799 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:24.774039 29733 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:24.774130 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:24.774487 29732 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:24.774794 25313 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
W20260709 13:09:24.775698 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 132964us
I20260709 13:09:24.775889 25389 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 14: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:24.778095 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:24.726323 29519 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:24.786164 25940 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:24.679095 29591 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:24.802038 29733 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:24.805166 29694 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:24.806552 29733 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:24.688496 29580 consensus_queue.cc:1059] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.093s
W20260709 13:09:24.812584 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 141884us
I20260709 13:09:24.779603 29643 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:24.803155 29704 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:24.818320 29337 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:24.752636 29506 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:24.840798 25209 leader_election.cc:400] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12.
I20260709 13:09:24.841250 25209 leader_election.cc:403] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 13:09:24.834769 29304 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:24.853109 25209 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:24.838280 29683 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:24.817354 29731 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260709 13:09:24.786268 29734 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:24.845168 29642 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 1.996s	user 0.004s	sys 0.009s
I20260709 13:09:24.840474 29704 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:24.822448 29736 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.045s
I20260709 13:09:24.854182 29492 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:24.852479 29143 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12.
I20260709 13:09:24.867897 29506 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:24.869745 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:24.902027 29661 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:24.870663 29591 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:24.870988 25318 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:24.903146 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2262) took 1138 ms. Trace:
I20260709 13:09:24.871598 29664 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:24.876103 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1780) took 3365 ms. Trace:
I20260709 13:09:24.903781 29704 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:24.889917 29530 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:09:24.882794 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:24.904786 29530 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:24.886229 25209 leader_election.cc:400] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11.
I20260709 13:09:24.905578 25209 leader_election.cc:403] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:09:24.905593 25321 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:24.888543 25939 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
W20260709 13:09:24.853561 29738 log.cc:927] Time spent T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260709 13:09:24.807498 29732 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:24.903502 25318 rpcz_store.cc:276] 0709 13:09:23.764882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:23.765040 (+   158us) service_pool.cc:224] Handling call
0709 13:09:23.765429 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:24.870746 (+1105317us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b5400476e20. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976c29dc0 00007f2976cb41c0 00007f2976cb5231 00007f2976cb1d8f 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c
0709 13:09:24.902887 (+ 32141us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:24.903091 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":585001216}
I20260709 13:09:24.911564 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 908) took 1735 ms. Trace:
I20260709 13:09:24.869778 29622 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:24.871035 29688 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:24.904130 25386 rpcz_store.cc:276] 0709 13:09:21.510517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:21.510669 (+   152us) service_pool.cc:224] Handling call
0709 13:09:21.511009 (+   340us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:21.538484 (+ 27475us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:21.538517 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:22.670502 (+1131985us) spinlock_profiling.cc:243] Waited 21.2 ms on lock 0x7b2c000e1828. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f29755aba11 00007f29755bb372 00007f2976ceda01 00007f2976cb243a 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2
0709 13:09:22.687677 (+ 17175us) log.cc:844] Serialized 52 byte log entry
0709 13:09:22.849755 (+162078us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:09:24.863728 (+2013973us) spinlock_profiling.cc:243] Waited 5.2 ms on lock 0x7b180041df40. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f297685a4bc 00007f2973ba848b 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc2455 00007f297a6aa3c7 00007f2976cbc9a7 00007f2976ccc0bf
0709 13:09:24.863952 (+   224us) log.cc:844] Serialized 14 byte log entry
0709 13:09:24.864344 (+   392us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:24.872722 (+  8378us) spinlock_profiling.cc:243] Waited 4.61 ms on lock 0x7b180041df40. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b3fc0a 00007f297d27f219 00007f297ab8658f 00007f2973bc5571 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd
0709 13:09:24.875682 (+  2960us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:24.875784 (+   102us) raft_consensus.cc:1685] finished
0709 13:09:24.875804 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:24.876018 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":903840,"prepare.queue_time_us":1137404,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":3640,"raft.queue_time_us":1311,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":3640,"spinlock_wait_cycles":80552832,"thread_start_us":310972,"threads_started":2,"wal-append.queue_time_us":212223,"wal-append.run_wall_time_us":1106713}
I20260709 13:09:24.912002 25942 rpcz_store.cc:276] 0709 13:09:23.176266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:23.249267 (+ 73001us) service_pool.cc:224] Handling call
0709 13:09:23.312967 (+ 63700us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:23.620026 (+307059us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:24.911284 (+1291258us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b5400630454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:24.911490 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":648841600}
W20260709 13:09:24.919006 29658 log.cc:927] Time spent T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 1.668s	user 0.007s	sys 0.008s
I20260709 13:09:24.912359 29143 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 11
I20260709 13:09:24.922298 29688 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:24.909400 29492 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:24.922961 29675 log.cc:927] Time spent T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 1.425s	user 0.008s	sys 0.008s
I20260709 13:09:24.905339 29530 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:24.931736 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:24.934804 25386 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:24.936187 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:24.901198 29690 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:24.927167 29732 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:24.934005 29591 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:24.954362 29554 consensus_meta.cc:287] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.509s	user 0.004s	sys 0.006s
I20260709 13:09:24.939772 29614 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:24.966948 29659 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:24.967537 29659 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:24.972572 29143 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11.
I20260709 13:09:24.974361 29530 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:24.958091 29554 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:24.984257 29530 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:24.986214 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:24.991218 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:24.991465 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:24.991950 25394 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:24.995386 25209 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:24.957172 29591 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:24.997269 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:24.997985 25944 raft_consensus.cc:2368] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11.
I20260709 13:09:24.998600 29591 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:24.998870 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:24.977720 29734 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:24.958096 29614 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:25.002321 29614 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:25.011752 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:25.012571 25242 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:25.012811 25241 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:09:25.014092 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:25.014711 25245 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:25.016363 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:25.017707 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:25.017918 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:25.018702 25243 raft_consensus.cc:2410] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 12 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 13:09:24.957746 29690 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:25.023301 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:25.030289 25238 raft_consensus.cc:2410] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 13 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 13:09:25.027014 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:25.033366 25245 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:09:25.033679 25240 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:25.024128 29530 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:25.024595 29622 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:25.036535 25244 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:09:25.037770 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:25.073649 25246 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:09:25.026255 29643 consensus_queue.cc:1059] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.014s
I20260709 13:09:25.050544 25359 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:25.050750 29736 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:09:25.087899 25394 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:25.075965 25359 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:25.054194 29690 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:25.058284 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:25.058954 25243 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 6. Current term is 7. Ops: []
I20260709 13:09:25.060833 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:25.063222 25209 leader_election.cc:400] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11.
I20260709 13:09:25.090370 25209 leader_election.cc:403] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:09:25.065670 25909 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:25.090875 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:25.091593 25245 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:25.091629 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:25.092235 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:25.092832 25387 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:25.092911 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:25.093367 25394 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 13:09:25.097605 25386 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:25.052762 29694 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:25.078871 29665 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.586s	user 0.012s	sys 0.013s
I20260709 13:09:25.005376 29732 raft_consensus.cc:2955] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Committing config change with OpId 12.10: config changed from index 4 to 10, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:25.098683 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:25.103441 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:25.104719 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:25.107232 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:25.111104 25359 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:25.101457 29665 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.642s	user 0.013s	sys 0.013s
I20260709 13:09:25.079722 29622 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:25.099413 29735 raft_consensus.cc:2764] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 13:09:25.122712 25245 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:25.128302 25941 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:09:25.098167 29591 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:25.125674 25244 raft_consensus.cc:2955] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Committing config change with OpId 12.10: config changed from index 4 to 10, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:09:25.126813 29677 consensus_meta.cc:287] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.558s	user 0.009s	sys 0.003s
I20260709 13:09:25.135870 25212 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:25.116210 29742 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:25.125809 29686 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:25.140930 25212 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:25.135897 29591 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:25.178257 29591 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:25.145347 29709 consensus_queue.cc:1059] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:09:25.179503 29709 consensus_queue.cc:1243] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer 2329e428ebbf484fa3ea9452133df567 log is divergent from this leader: its last log entry 12.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:09:25.126613 29143 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 11
I20260709 13:09:25.182328 25212 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:25.184298 25212 leader_election.cc:400] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12.
I20260709 13:09:25.157790 29643 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:25.168696 29492 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:25.189560 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:25.134529 29690 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:25.159541 29677 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:25.133188 29622 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:25.124125 29652 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:25.176405 29686 raft_consensus.cc:697] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:25.177533 25245 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 13:09:25.204632 25284 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:25.195991 29703 log.cc:927] Time spent T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.933s	user 0.000s	sys 0.005s
I20260709 13:09:25.132272 29683 consensus_queue.cc:1059] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260709 13:09:25.206669 29530 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 13:09:25.208480 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:25.193292 29643 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:09:25.192967 29690 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:25.216082 29546 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:25.186098 29742 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:25.207420 29554 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.017s
I20260709 13:09:25.237394 29683 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:25.238008 29683 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:25.240285 29683 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:25.240065 29614 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:25.222460 29690 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:25.236598 29591 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:25.236371 29659 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:25.244189 29591 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:25.249511 29683 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:25.251375 29683 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:25.249883 29143 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11.
I20260709 13:09:25.245654 29614 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:25.254776 29683 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:25.255138 29659 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:25.257054 29686 consensus_queue.cc:237] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:25.246105 29580 raft_consensus.cc:3055] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:25.260944 29580 raft_consensus.cc:740] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:25.262959 29580 consensus_queue.cc:260] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:25.263979 29742 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:25.264197 29580 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:25.265378 29677 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:25.274739 29732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:38928 (request call id 0) took 3307 ms. Trace:
I20260709 13:09:25.270547 29622 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:25.278376 29483 raft_consensus.cc:3055] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:25.277802 29591 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:25.287240 29483 raft_consensus.cc:740] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:25.291895 29483 consensus_queue.cc:260] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:25.284279 29732 rpcz_store.cc:276] 0709 13:09:21.967361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:21.970047 (+  2686us) service_pool.cc:224] Handling call
0709 13:09:22.104154 (+134107us) log.cc:844] Serialized 501 byte log entry
0709 13:09:23.390702 (+1286548us) spinlock_profiling.cc:243] Waited 8.06 ms on lock 0x7b1800061720. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:09:24.276410 (+885708us) spinlock_profiling.cc:243] Waited 47.8 ms on lock 0x7b54004a27a0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976c2b142 00007f2976c0beaa 00007f2976c12d53 00007f2976c12cd9 00007f2976c12c69 00007f2976c12c31 00007f2976c11dfd 000000000038f354 000000000038f199 00007f2973bc5475
0709 13:09:25.274597 (+998187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42807,"raft.queue_time_us":18622,"raft.run_cpu_time_us":20902,"raft.run_wall_time_us":920117,"spinlock_wait_cycles":128367744,"thread_start_us":206036,"threads_started":1,"wal-append.queue_time_us":370153,"wal-append.run_cpu_time_us":2196,"wal-append.run_wall_time_us":1178454}
I20260709 13:09:25.293457 29483 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:25.295545 29690 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:25.298532 25128 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet eda233ace2a24b379fef320a5f610c7a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 13)
I20260709 13:09:25.299957 29732 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:25.302408 25941 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:09:25.311563 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:25.313560 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:25.314018 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:25.316280 29694 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:25.316880 29694 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:25.319000 29694 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:25.320261 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:25.322398 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:25.305327 29622 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:25.324352 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:25.291052 29591 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:25.326086 25940 raft_consensus.cc:2410] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 12 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 13:09:25.328315 25358 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:25.301633 29732 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:25.334256 29659 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:25.335219 29686 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:25.337036 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:25.337841 25939 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:25.327139 29690 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:25.418412 29762 log.cc:927] Time spent T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
I20260709 13:09:25.428982 29591 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:25.430409 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:25.433885 29732 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:25.435685 29747 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.538s	user 0.004s	sys 0.000s
W20260709 13:09:25.475899 25908 leader_election.cc:336] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.898s (SENT)
I20260709 13:09:25.476629 25908 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:25.532135 29731 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:25.594851 29731 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:25.568156 29594 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:09:25.662442 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:25.663864 29722 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:25.664592 29722 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:25.673286 29731 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:25.706866 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 823) took 1057 ms. Trace:
I20260709 13:09:25.707314 25938 rpcz_store.cc:276] 0709 13:09:24.649434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:24.649557 (+   123us) service_pool.cc:224] Handling call
0709 13:09:24.651873 (+  2316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:24.750387 (+ 98514us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:25.706637 (+956250us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x7b54006748d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:25.706808 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":67866368}
I20260709 13:09:25.709167 29690 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:25.724674 29690 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:25.733944 29764 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 13:09:25.738082 29722 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:25.771945 29622 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:25.772724 29622 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:25.773568 29742 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:25.774451 29742 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:25.775876 29622 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:25.777379 29690 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:25.777931 29744 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.888s	user 0.003s	sys 0.000s
I20260709 13:09:25.818747 25132 maintenance_manager.cc:643] P 4d591857b6ac45ad9cab82445b07ae03: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 7.348s	user 0.197s	sys 0.037s Metrics: {"bytes_written":30033,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":130290,"dirs.run_cpu_time_us":422,"dirs.run_wall_time_us":22131,"drs_written":1,"lbm_read_time_us":10459,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":2,"lbm_write_time_us":353337,"lbm_writes_10-100_ms":14,"lbm_writes_lt_1ms":20,"peak_mem_usage":0,"reinsert_count":0,"rows_written":63,"thread_start_us":55552,"threads_started":2,"update_count":5,"wal-append.queue_time_us":9541}
W20260709 13:09:25.821974 25209 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.718s	user 0.011s	sys 0.006s
W20260709 13:09:25.822476 25209 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.011s	sys 0.006s
I20260709 13:09:25.822777 25133 maintenance_manager.cc:419] P 4d591857b6ac45ad9cab82445b07ae03: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 10065 bytes on disk
W20260709 13:09:25.822772 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 759636us
I20260709 13:09:25.825515 25132 maintenance_manager.cc:643] P 4d591857b6ac45ad9cab82445b07ae03: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":328,"lbm_reads_lt_1ms":4}
I20260709 13:09:25.828325 29768 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:25.829253 29768 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:25.830991 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:25.831575 25246 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:25.832197 29768 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:25.832970 29768 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:25.833618 29768 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:25.836426 29768 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:25.837316 29768 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:25.838119 29768 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:25.841400 29768 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:25.842064 29768 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:25.842633 29768 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:25.845618 29768 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:25.860463 25246 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:25.186455 25212 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:25.861822 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 676694us
I20260709 13:09:25.862825 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:25.865618 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:25.867336 29769 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:25.867985 25246 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 13:09:25.869771 29769 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:25.869022 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:25.872529 25209 leader_election.cc:400] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11.
I20260709 13:09:25.870349 29769 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:25.876159 25246 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:09:25.877413 25212 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:25.878386 29768 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:25.879529 25246 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:25.880779 25246 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:25.876461 29769 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:25.873414 25209 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:25.881891 25246 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:25.883006 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:25.883742 25246 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:25.884822 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:25.885345 29769 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:25.885696 25246 raft_consensus.cc:2393] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 9: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:25.886137 29769 raft_consensus.cc:697] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:25.887069 29769 consensus_queue.cc:237] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:25.888043 25246 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:09:25.891608 25212 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:25.892829 29768 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:25.895228 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:25.896454 29768 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:25.896960 29768 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:25.897439 29768 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:25.910637 29768 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:26.019145 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 109499us
I20260709 13:09:26.021103 29772 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:26.021770 29772 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:26.027143 25212 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:26.177330 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 156415us
I20260709 13:09:26.179153 25212 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:26.229722 29772 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:26.230584 29772 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:26.231336 29772 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:26.200664 29774 consensus_queue.cc:1059] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.045s
I20260709 13:09:26.320446 29774 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:26.321362 29774 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:26.330374 29774 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:26.331055 29774 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:26.331636 29774 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:25.927887 29768 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:26.434602 29772 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:26.435359 29772 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:26.579106 29768 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:26.580111 29768 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:26.582996 29768 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:26.583781 29768 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:09:26.592592 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:26.782516 25342 debug-util.cc:398] Leaking SignalData structure 0x7b08000a8ce0 after lost signal to thread 24113
W20260709 13:09:26.783103 25342 debug-util.cc:398] Leaking SignalData structure 0x7b08001a4ee0 after lost signal to thread 24116
I20260709 13:09:26.795749 29768 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:26.799136 29768 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:26.802225 29777 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:26.888509 25939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 888) took 1505 ms. Trace:
I20260709 13:09:27.060021 25939 rpcz_store.cc:276] 0709 13:09:25.383473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:25.383703 (+   230us) service_pool.cc:224] Handling call
0709 13:09:25.417605 (+ 33902us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:25.422520 (+  4915us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:25.422556 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:25.880739 (+458183us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b2c000476c8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f29755aba11 00007f29755bb372 00007f2976ceda01 00007f2976cb243a 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2
0709 13:09:25.881161 (+   422us) log.cc:844] Serialized 26 byte log entry
0709 13:09:25.924111 (+ 42950us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:25.924161 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:25.924263 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:26.888118 (+963855us) raft_consensus.cc:1685] finished
0709 13:09:26.888147 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:26.888425 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4427,"raft.queue_time_us":5443,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":77,"spinlock_wait_cycles":3585280,"thread_start_us":33415,"threads_started":1,"wal-append.queue_time_us":996009}
I20260709 13:09:27.150717 29779 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:27.151566 29779 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:27.145764 29777 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:27.156109 29777 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:27.156922 29777 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:27.157605 29777 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:27.163537 29777 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:27.146046 29768 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:27.171454 29768 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:27.174343 29768 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:27.175112 29768 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:27.175751 29768 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:27.178318 29768 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:27.159791 29779 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:27.182497 29769 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.259s	user 0.006s	sys 0.017s
W20260709 13:09:27.183172 25285 connection.cc:256] RPC call timeout handler was delayed by -1.4171202300000232s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17830000000000001s for a response
W20260709 13:09:27.160742 25209 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.009s	sys 0.002s
W20260709 13:09:27.221597 25360 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.849s	user 0.006s	sys 0.005s
I20260709 13:09:27.215695 25394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 873) took 1196 ms. Trace:
W20260709 13:09:27.356295 25360 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.984s	user 0.012s	sys 0.006s
I20260709 13:09:27.356359 25394 rpcz_store.cc:276] 0709 13:09:26.019704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:27.214604 (+1194900us) service_pool.cc:224] Handling call
0709 13:09:27.215046 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:27.215270 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:27.215306 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:27.215360 (+    54us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:27.215411 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:27.215477 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:27.215623 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:27.357995 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:27.229205 25209 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.010s	sys 0.002s
I20260709 13:09:27.359087 29686 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:27.359894 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 880) took 2057 ms. Trace:
W20260709 13:09:27.229730 29769 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.306s	user 0.007s	sys 0.017s
W20260709 13:09:27.226516 25358 connection.cc:256] RPC call timeout handler was delayed by -1.2468207890000258s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1716s for a response
I20260709 13:09:27.360327 25942 rpcz_store.cc:276] 0709 13:09:25.302073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:25.302194 (+   121us) service_pool.cc:224] Handling call
0709 13:09:25.302521 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:25.302874 (+   353us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:25.302899 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:25.303395 (+   496us) log.cc:844] Serialized 26 byte log entry
0709 13:09:25.329660 (+ 26265us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:25.336877 (+  7217us) log.cc:844] Serialized 14 byte log entry
0709 13:09:25.337092 (+   215us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:25.337209 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:27.357931 (+2020722us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x7b18003e0b00. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:09:27.359533 (+  1602us) raft_consensus.cc:1685] finished
0709 13:09:27.359655 (+   122us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:27.359825 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":8402,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":1179,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":3517,"spinlock_wait_cycles":7819008,"thread_start_us":25865,"threads_started":1,"wal-append.queue_time_us":291556,"wal-append.run_cpu_time_us":4265,"wal-append.run_wall_time_us":1183085}
W20260709 13:09:27.361996 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:27.230060 25285 connection.cc:256] RPC call timeout handler was delayed by -1.1026846970000861s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2157s for a response
W20260709 13:09:27.362391 25285 connection.cc:256] RPC call timeout handler was delayed by -1.0422472660000039s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19490000000000002s for a response
I20260709 13:09:27.363444 25285 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:27.359324 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 703566us
I20260709 13:09:27.222765 29772 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:27.366582 29485 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:27.367616 29485 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:27.368466 29769 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:27.211092 29774 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:27.359123 25394 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
W20260709 13:09:27.340487 25212 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.160s	user 0.021s	sys 0.029s
W20260709 13:09:27.370973 25212 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.191s	user 0.021s	sys 0.029s
W20260709 13:09:27.371315 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 1.19216e+06us
I20260709 13:09:27.358026 29661 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:27.372540 29661 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:27.374464 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:27.316686 29722 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:27.375801 29782 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:27.376375 29722 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:27.376437 29782 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:27.378749 29782 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:27.378787 29722 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:27.344877 25237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2063) took 3989 ms. Trace:
I20260709 13:09:27.379494 29782 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:27.345791 25240 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:27.380209 29782 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:27.380688 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2109) took 1586 ms. Trace:
I20260709 13:09:27.381050 25240 rpcz_store.cc:276] 0709 13:09:25.793815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:27.309655 (+1515840us) service_pool.cc:224] Handling call
0709 13:09:27.345607 (+ 35952us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:27.380433 (+ 34826us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:27.380629 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:27.345939 25395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 867) took 1429 ms. Trace:
I20260709 13:09:27.344895 29779 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:27.382249 25395 rpcz_store.cc:276] 0709 13:09:25.916872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:27.270427 (+1353555us) service_pool.cc:224] Handling call
0709 13:09:27.345413 (+ 74986us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:27.345595 (+   182us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:27.345617 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:27.345669 (+    52us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:27.345721 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:27.345783 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:27.345888 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:27.382464 29779 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:27.379693 25237 rpcz_store.cc:276] 0709 13:09:23.355663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:23.502337 (+146674us) service_pool.cc:224] Handling call
0709 13:09:23.502679 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:23.729106 (+226427us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:23.729163 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:23.729791 (+   628us) log.cc:844] Serialized 26 byte log entry
0709 13:09:24.116547 (+386756us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:24.116625 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:24.116721 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:25.195422 (+1078701us) spinlock_profiling.cc:243] Waited 7.29 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:09:27.288978 (+2093556us) raft_consensus.cc:1685] finished
0709 13:09:27.289011 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:27.344798 (+ 55787us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":222712,"raft.queue_time_us":237171,"raft.run_cpu_time_us":47,"raft.run_wall_time_us":51,"spinlock_wait_cycles":16764800,"thread_start_us":330121,"threads_started":1,"wal-append.queue_time_us":475767,"wal-append.run_cpu_time_us":39841,"wal-append.run_wall_time_us":3107356}
I20260709 13:09:27.384887 29779 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:27.347931 29594 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:27.385711 29594 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:27.386391 29659 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:27.344456 25321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 696) took 3531 ms (client timeout 1507 ms). Trace:
I20260709 13:09:27.218765 29776 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:27.386904 29659 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:27.387266 29776 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:27.388273 29594 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:27.348824 25244 consensus_meta.cc:287] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 2.212s	user 0.029s	sys 0.015s
I20260709 13:09:27.359913 29686 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:27.389317 29776 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:27.390205 25394 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20260709 13:09:27.390551 25394 pending_rounds.cc:85] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 9
I20260709 13:09:27.390760 25394 pending_rounds.cc:107] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260709 13:09:27.390995 25394 raft_consensus.cc:2889] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:27.391268 29686 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:27.379029 29661 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:27.344587 29591 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:27.391960 29591 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:27.392982 29782 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:27.392421 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 773) took 2335 ms. Trace:
I20260709 13:09:27.393294 29772 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:27.393868 29772 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:27.394678 29591 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:27.393648 25244 rpcz_store.cc:276] 0709 13:09:25.057453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:25.101092 (+ 43639us) service_pool.cc:224] Handling call
0709 13:09:25.125336 (+ 24244us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:25.125564 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:09:27.388978 (+2263414us) log.cc:844] Serialized 14 byte log entry
0709 13:09:27.392044 (+  3066us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:27.392094 (+    50us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:27.392179 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:27.392252 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:27.392367 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":125,"raft.queue_time_us":220,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"thread_start_us":2670,"threads_started":1,"wal-append.queue_time_us":2411}
I20260709 13:09:27.395084 29780 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:27.395614 29780 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:27.396251 29772 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:27.396809 29772 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:27.369964 29774 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:27.397285 29775 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:27.397506 29774 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:27.397816 29775 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:27.399616 29780 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:27.399816 29775 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:27.385104 29722 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:27.401573 29722 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:27.402186 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 765) took 2388 ms. Trace:
I20260709 13:09:27.402527 25241 rpcz_store.cc:276] 0709 13:09:25.013846 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:25.014018 (+   172us) service_pool.cc:224] Handling call
0709 13:09:25.014325 (+   307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:25.022905 (+  8580us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:25.022938 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:25.026751 (+  3813us) log.cc:844] Serialized 26 byte log entry
0709 13:09:25.201529 (+174778us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:25.201585 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:25.201668 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:27.401976 (+2200308us) raft_consensus.cc:1685] finished
0709 13:09:27.401999 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:27.402133 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8182,"raft.queue_time_us":9928,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":1775,"thread_start_us":100029,"threads_started":1,"wal-append.queue_time_us":172132}
I20260709 13:09:27.408689 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 819) took 4210 ms. Trace:
I20260709 13:09:27.410324 25911 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:27.369004 29769 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:27.413147 29769 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:27.409305 25937 rpcz_store.cc:276] 0709 13:09:23.174959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:23.176999 (+  2040us) service_pool.cc:224] Handling call
0709 13:09:23.177479 (+   480us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:23.201613 (+ 24134us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:09:23.201884 (+   271us) log.cc:844] Serialized 14 byte log entry
0709 13:09:25.178756 (+1976872us) spinlock_profiling.cc:243] Waited 2.06 ms on lock 0x7b180036f160. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b3fc0a 00007f297d27f219 00007f2973bc7a40 00007f2973bc24e4 00007f2973bc2455 00007f297a6aa3c7 00007f2976cbc9a7 00007f2976ccc0bf 00007f2976ccc038
0709 13:09:25.222268 (+ 43512us) log.cc:844] Serialized 14 byte log entry
0709 13:09:25.258880 (+ 36612us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:25.258958 (+    78us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:25.259039 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:27.384935 (+2125896us) spinlock_profiling.cc:243] Waited 23.9 ms on lock 0x7b5400702bd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:27.384979 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:27.385095 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12256,"prepare.queue_time_us":1807363,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":77,"raft.queue_time_us":2905,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":6527,"spinlock_wait_cycles":59707008,"thread_start_us":241595,"threads_started":3,"wal-append.queue_time_us":261593,"wal-append.run_cpu_time_us":2393,"wal-append.run_wall_time_us":2013588}
W20260709 13:09:27.414340 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 48693.9us
I20260709 13:09:27.415407 25285 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:27.416496 29485 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:27.417260 29485 raft_consensus.cc:697] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:27.418421 29485 consensus_queue.cc:237] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:27.419158 25242 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.291s	user 0.015s	sys 0.023s
W20260709 13:09:27.419813 25242 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.291s	user 0.015s	sys 0.023s
I20260709 13:09:27.428072 25388 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:27.399618 29774 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:27.388899 29659 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:27.399816 25321 rpcz_store.cc:269] 0709 13:09:23.821824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:23.900718 (+ 78894us) service_pool.cc:224] Handling call
0709 13:09:27.312858 (+3412140us) spinlock_profiling.cc:243] Waited 998 ms on lock 0x7b5400044554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:27.344387 (+ 31529us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2294302592}
I20260709 13:09:27.397286 29772 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:27.403496 29722 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:27.451624 25936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 813) took 4323 ms. Trace:
I20260709 13:09:27.454396 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:27.435181 29337 raft_consensus.cc:3055] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:27.465988 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 870) took 1541 ms. Trace:
I20260709 13:09:27.490864 25389 rpcz_store.cc:276] 0709 13:09:25.924632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:27.443250 (+1518618us) service_pool.cc:224] Handling call
0709 13:09:27.443703 (+   453us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:27.465628 (+ 21925us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:27.465655 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:27.465696 (+    41us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:27.465736 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:27.465805 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:27.465934 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:27.492972 29788 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:27.493670 29788 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:27.466136 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 55901.1us
I20260709 13:09:27.471837 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 874) took 1451 ms. Trace:
I20260709 13:09:27.479192 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 868) took 1562 ms. Trace:
I20260709 13:09:27.488173 25391 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:27.490726 29337 raft_consensus.cc:740] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:09:27.496925 25391 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41830 (request call id 866) took 1580 ms (client timeout 1651 ms). Trace:
I20260709 13:09:27.463328 29742 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:27.497361 25391 rpcz_store.cc:269] 0709 13:09:25.916152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:27.454026 (+1537874us) service_pool.cc:224] Handling call
0709 13:09:27.496830 (+ 42804us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:27.496150 25387 rpcz_store.cc:276] 0709 13:09:26.076685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:27.443924 (+1367239us) service_pool.cc:224] Handling call
0709 13:09:27.471056 (+ 27132us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:27.471314 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:27.471353 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:27.471475 (+   122us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:27.471523 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:27.471597 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:27.471756 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:27.486603 25936 rpcz_store.cc:276] 0709 13:09:23.128308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:23.128475 (+   167us) service_pool.cc:224] Handling call
0709 13:09:23.156960 (+ 28485us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:24.942009 (+1785049us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:24.942061 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:24.942125 (+    64us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:24.942187 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:27.451271 (+2509084us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b54006d88d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:27.451340 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:27.451533 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4104519552}
I20260709 13:09:27.496206 25393 rpcz_store.cc:276] 0709 13:09:25.917169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:27.428733 (+1511564us) service_pool.cc:224] Handling call
0709 13:09:27.478471 (+ 49738us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:27.478726 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:27.478763 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:27.478822 (+    59us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:27.478875 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:27.478950 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:27.479105 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:27.477849 29483 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:09:27.465577 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 2.26081e+06us
I20260709 13:09:28.117640 29794 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:28.118290 25284 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:27.503894 29685 raft_consensus.cc:3055] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:27.540042 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 881) took 2235 ms. Trace:
I20260709 13:09:27.504249 29337 consensus_queue.cc:260] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:27.794575 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2108) took 2000 ms. Trace:
I20260709 13:09:28.138516 29766 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:27.503082 29772 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:27.496592 29784 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:27.542703 29788 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:27.659153 29792 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 13:09:27.961210 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 463765us
W20260709 13:09:27.946820 25908 connection.cc:256] RPC call timeout handler was delayed by -1.2153108630000133s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21760000000000002s for a response
W20260709 13:09:27.875957 25358 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.005s	sys 0.025s
I20260709 13:09:28.119673 29794 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.131392 29779 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:28.133107 29580 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:28.134157 29742 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:28.136384 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 766) took 3120 ms. Trace:
I20260709 13:09:28.140786 29337 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:28.147920 29580 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:28.148447 29784 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.141769 25941 rpcz_store.cc:276] 0709 13:09:25.304940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:25.305063 (+   123us) service_pool.cc:224] Handling call
0709 13:09:25.305422 (+   359us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:25.305739 (+   317us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:25.305756 (+    17us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:25.306123 (+   367us) log.cc:844] Serialized 52 byte log entry
0709 13:09:27.432605 (+2126482us) spinlock_profiling.cc:243] Waited 3.52 ms on lock 0x7b18002254c0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f29768520d1 00007f2976c6da0f
0709 13:09:27.460810 (+ 28205us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:27.460867 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:27.460958 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:27.539710 (+ 78752us) raft_consensus.cc:1685] finished
0709 13:09:27.539733 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:27.539972 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":66,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":3358,"spinlock_wait_cycles":8096000,"thread_start_us":27940,"threads_started":1,"wal-append.queue_time_us":2221539}
I20260709 13:09:28.151646 29784 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:28.152779 29580 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:28.152904 29779 raft_consensus.cc:697] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:28.144912 29685 raft_consensus.cc:740] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:28.155143 29779 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.155298 29685 consensus_queue.cc:260] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:28.146992 25358 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.784s	user 0.010s	sys 0.034s
W20260709 13:09:28.157676 25358 connection.cc:256] RPC call timeout handler was delayed by -1.0398207890000322s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1946s for a response
I20260709 13:09:28.156996 29685 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:28.163549 29788 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:28.164680 29788 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.163676 29792 raft_consensus.cc:697] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:28.146303 25242 rpcz_store.cc:276] 0709 13:09:25.014850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:25.015302 (+   452us) service_pool.cc:224] Handling call
0709 13:09:25.015622 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:25.015846 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:09:25.128316 (+112470us) log.cc:844] Serialized 14 byte log entry
0709 13:09:27.421158 (+2292842us) log.cc:844] Serialized 14 byte log entry
0709 13:09:28.134262 (+713104us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:28.134379 (+   117us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:28.134492 (+   113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:28.134617 (+   125us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:28.134894 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":122030,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":81,"raft.queue_time_us":698473,"raft.run_cpu_time_us":382,"raft.run_wall_time_us":382,"thread_start_us":2403685,"threads_started":3,"wal-append.queue_time_us":235598,"wal-append.run_cpu_time_us":1217,"wal-append.run_wall_time_us":1001248}
I20260709 13:09:28.172297 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:28.173779 25241 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:28.169183 29792 consensus_queue.cc:237] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.139861 25245 rpcz_store.cc:276] 0709 13:09:25.793454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:27.268927 (+1475473us) service_pool.cc:224] Handling call
0709 13:09:27.269463 (+   536us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:27.354613 (+ 85150us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:27.354672 (+    59us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:27.376390 (+ 21718us) log.cc:844] Serialized 52 byte log entry
0709 13:09:27.452013 (+ 75623us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:27.452055 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:27.452129 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:27.793658 (+341529us) raft_consensus.cc:1685] finished
0709 13:09:27.793749 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:27.794229 (+   480us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":167,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"thread_start_us":75226,"threads_started":1,"wal-append.queue_time_us":51749}
I20260709 13:09:28.177611 29742 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:28.182500 25240 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 13:09:28.183779 25908 leader_election.cc:336] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.507s (SENT)
I20260709 13:09:28.184410 25908 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:28.187350 29801 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:28.188441 29801 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.190644 29580 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:28.191437 29580 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:28.197820 29803 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:28.205168 29794 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:28.206489 25285 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.672s	user 0.000s	sys 0.025s
W20260709 13:09:28.206884 25285 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.001s	sys 0.025s
I20260709 13:09:28.209226 29799 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:28.210103 29799 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.212760 29799 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:28.213376 29483 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.215771 29483 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:28.216212 25909 connection.cc:256] RPC call timeout handler was delayed by -1.0043995579999319s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18180000000000002s for a response
I20260709 13:09:28.218397 29801 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:28.218680 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:28.219070 29801 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:28.219472 25937 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:28.219913 29801 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.222216 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:28.223333 25938 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 12: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:09:28.223657 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 103629us
I20260709 13:09:28.224700 25284 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:28.225811 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:28.226675 25939 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:28.196205 29742 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:28.232402 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 79781us
I20260709 13:09:28.198813 29803 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.243662 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:28.241864 25937 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:28.242401 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:28.245045 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:28.247462 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:28.247849 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:28.257581 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:28.257586 29772 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:28.263033 29772 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.241679 29799 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:28.257676 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 857) took 3620 ms. Trace:
I20260709 13:09:28.267984 25241 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:28.268450 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:28.270236 29798 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:28.270938 29798 raft_consensus.cc:697] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:28.271525 29772 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:28.272378 29798 consensus_queue.cc:237] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.274699 29580 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:28.275411 29788 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:28.249379 29483 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:28.276538 29483 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.279170 29483 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:28.280375 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:28.267985 25392 rpcz_store.cc:276] 0709 13:09:24.636597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:24.661090 (+ 24493us) service_pool.cc:224] Handling call
0709 13:09:24.662853 (+  1763us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:24.663441 (+   588us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:24.663482 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:24.695040 (+ 31558us) log.cc:844] Serialized 52 byte log entry
0709 13:09:24.896365 (+201325us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:24.923434 (+ 27069us) log.cc:844] Serialized 14 byte log entry
0709 13:09:24.923667 (+   233us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:24.923758 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:25.777315 (+853557us) spinlock_profiling.cc:243] Waited 5.21 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:09:28.248980 (+2471665us) raft_consensus.cc:1685] finished
0709 13:09:28.254028 (+  5048us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:28.257413 (+  3385us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17400,"prepare.queue_time_us":27045,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":1272,"raft.run_cpu_time_us":1175,"raft.run_wall_time_us":11366,"spinlock_wait_cycles":13885568,"thread_start_us":109915,"threads_started":1,"wal-append.queue_time_us":116840}
I20260709 13:09:28.294103 25936 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 13:09:28.294945 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:28.298664 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:28.299583 25243 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:28.300606 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:28.303272 25937 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:28.309001 25243 raft_consensus.cc:2368] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11.
I20260709 13:09:28.308904 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:28.289719 29784 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.076s
I20260709 13:09:28.310168 25911 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:28.349984 25242 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 13:09:28.400584 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 55936us
I20260709 13:09:28.312549 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:28.349408 25241 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:28.312474 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:28.377168 25386 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41830 (request call id 871) took 2450 ms (client timeout 1940 ms). Trace:
I20260709 13:09:28.313532 25942 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 8. (term mismatch)
W20260709 13:09:28.337620 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:28.361027 25240 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:28.358614 25394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 876) took 1986 ms. Trace:
I20260709 13:09:28.315752 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:28.349498 25938 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:28.404578 25245 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:28.378005 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:28.405738 25386 rpcz_store.cc:269] 0709 13:09:25.926280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:28.376663 (+2450383us) service_pool.cc:211] Skipping call since client already timed out
0709 13:09:28.376877 (+   214us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:09:28.413409 25942 pending_rounds.cc:85] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 7
I20260709 13:09:28.388196 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.418540 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:28.388455 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.377403 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.421514 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.375797 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:28.388798 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:28.423071 25317 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:28.355813 29803 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:28.419487 25937 raft_consensus.cc:2376] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10. Re-sending same reply.
I20260709 13:09:28.357668 29801 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:28.424652 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 169464us
I20260709 13:09:28.315017 29804 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:28.430958 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:28.431774 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:28.435021 29687 raft_consensus.cc:3055] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:28.312793 29688 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:28.418458 25942 pending_rounds.cc:107] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 10.8
I20260709 13:09:28.441982 25942 raft_consensus.cc:2889] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:28.413779 25394 rpcz_store.cc:276] 0709 13:09:26.371721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:27.371006 (+999285us) service_pool.cc:224] Handling call
0709 13:09:27.371504 (+   498us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:27.391421 (+ 19917us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:28.358077 (+966656us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x7b54001ebc54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:28.358417 (+   340us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":42165504}
I20260709 13:09:28.416841 29799 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:28.439152 29687 raft_consensus.cc:740] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:28.444198 29681 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:28.450994 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:28.453263 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:28.454905 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:28.448014 29804 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:28.453490 25940 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:28.448474 29681 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:28.459438 29681 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:28.459991 25911 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:28.460505 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:28.461563 25240 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:28.462754 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:28.462643 29687 consensus_queue.cc:260] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:28.463742 25240 raft_consensus.cc:2393] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 7: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:28.463865 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:28.464146 29687 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:28.465492 25320 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:28.460062 25242 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:28.466907 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:28.469373 25245 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:28.469601 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:28.464777 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:28.470752 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 49262.6us
I20260709 13:09:28.472690 29788 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:28.473327 29788 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:28.473881 29788 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:28.474036 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.445484 29801 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:28.470337 29792 consensus_queue.cc:1059] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20260709 13:09:28.475947 29801 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.476223 25240 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:09:28.473364 29688 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:28.477815 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:28.478161 29688 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.478989 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:28.479315 25392 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:28.480121 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:28.480823 25240 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:28.481791 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:28.482441 25240 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:28.476233 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:28.478734 29801 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:28.475281 25246 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 13:09:28.487236 29685 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:28.487574 29688 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:28.488042 25390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 872) took 2561 ms. Trace:
I20260709 13:09:28.487934 29685 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:28.488361 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:28.488440 25390 rpcz_store.cc:276] 0709 13:09:25.927042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:28.477084 (+2550042us) service_pool.cc:224] Handling call
0709 13:09:28.487390 (+ 10306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:28.487588 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:28.487614 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:28.487670 (+    56us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:28.487699 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:28.487751 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:28.487867 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:28.489570 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.489833 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:28.490108 29685 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:28.490352 25318 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:28.483685 25386 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
W20260709 13:09:28.490612 25358 leader_election.cc:336] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.605s (SENT)
I20260709 13:09:28.491035 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.491665 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.494073 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.495939 29799 raft_consensus.cc:3055] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:09:28.496416 29799 raft_consensus.cc:740] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:28.498606 29779 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:28.497452 29799 consensus_queue.cc:260] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.499529 29779 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:28.500737 29337 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:28.501659 29530 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:28.501533 29337 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.502316 29530 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:28.503899 29337 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:28.504091 29664 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:28.504490 29337 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:28.586090 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:28.593909 29337 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:28.507639 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.615450 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.598373 25240 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:28.600160 25241 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:28.564940 25317 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:28.596473 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.508502 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:28.519014 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:28.573091 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:28.622579 29704 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:28.622820 25243 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
I20260709 13:09:28.623313 29704 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.623898 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:28.624650 25243 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:28.575906 29685 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:28.569245 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:28.611521 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:28.504645 29664 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:28.609953 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 119383us
I20260709 13:09:28.596756 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.619135 25313 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:28.627993 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:28.628746 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.510711 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.628777 29812 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:28.596902 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.517699 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:28.639446 25315 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:28.630246 25237 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:09:28.574460 29688 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:28.644776 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.630935 29530 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:28.607878 29799 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:28.632912 25320 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
W20260709 13:09:28.633481 25239 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32994 (request call id 2110) took 2839 ms (client timeout 1687 ms). Trace:
I20260709 13:09:28.633726 29704 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:28.635079 29337 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:28.626999 25321 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:28.648167 25237 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:28.637331 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:28.639755 25359 connection.cc:256] RPC call timeout handler was delayed by -1.6345801109999911s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16140000000000002s for a response
W20260709 13:09:28.639461 25140 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39596 (request call id 285) took 25633 ms (client timeout 15000 ms). Trace:
W20260709 13:09:28.649434 25359 connection.cc:256] RPC call timeout handler was delayed by -1.6244819920000282s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16870000000000002s for a response
I20260709 13:09:28.640574 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:28.629700 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:28.647507 25239 rpcz_store.cc:269] 0709 13:09:25.794172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:28.633145 (+2838973us) service_pool.cc:211] Skipping call since client already timed out
0709 13:09:28.633288 (+   143us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:09:28.651331 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:28.652060 25239 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:09:28.506933 29779 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:28.646788 25315 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:09:28.649820 25359 connection.cc:256] RPC call timeout handler was delayed by -1.167073078000044s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19940000000000002s for a response
I20260709 13:09:28.652030 25940 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:28.576045 25909 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:28.587328 29794 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:28.504925 29580 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:28.648531 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:28.505118 29772 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:28.658420 25285 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:28.587585 29788 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.661711 25313 raft_consensus.cc:2376] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11. Re-sending same reply.
I20260709 13:09:28.660470 25319 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:28.636253 29664 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:28.668145 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.668459 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:28.649567 25140 rpcz_store.cc:269] 0709 13:09:02.978366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:02.978604 (+   238us) service_pool.cc:224] Handling call
0709 13:09:28.610032 (+25631428us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:09:18.967985 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:09:18.968620 (+   635us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:09:18.968673 (+    53us) write_op.cc:435] Acquired schema lock
0709 13:09:18.968694 (+    21us) tablet.cc:662] Decoding operations
0709 13:09:18.969485 (+   791us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:09:18.969593 (+   108us) write_op.cc:641] Partition lock acquired for write op
0709 13:09:18.969621 (+    28us) tablet.cc:685] Acquiring locks for 5 operations
0709 13:09:18.970299 (+   678us) tablet.cc:701] Row locks acquired
0709 13:09:18.972701 (+  2402us) write_op.cc:260] PREPARE: finished
0709 13:09:18.972929 (+   228us) write_op.cc:270] Start()
0709 13:09:18.976412 (+  3483us) write_op.cc:276] Timestamp: P: 1783602558972834 usec, L: 0
0709 13:09:18.976444 (+    32us) op_driver.cc:348] REPLICATION: starting
0709 13:09:19.025504 (+ 49060us) log.cc:844] Serialized 2070 byte log entry
0709 13:09:19.954659 (+929155us) op_driver.cc:464] REPLICATION: finished
0709 13:09:20.172459 (+217800us) write_op.cc:301] APPLY: starting
0709 13:09:20.172540 (+    81us) tablet.cc:1367] starting BulkCheckPresence
0709 13:09:20.173065 (+   525us) tablet.cc:1370] finished BulkCheckPresence
0709 13:09:20.173183 (+   118us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:09:20.173660 (+   477us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:09:20.173864 (+   204us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0709 13:09:20.173908 (+    44us) write_op.cc:312] APPLY: finished
0709 13:09:20.174144 (+   236us) log.cc:844] Serialized 47 byte log entry
0709 13:09:20.174312 (+   168us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:09:20.174419 (+   107us) write_op.cc:454] Released schema lock
0709 13:09:20.174463 (+    44us) write_op.cc:341] FINISH: Updating metrics
0709 13:09:20.174872 (+   409us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:09:20.174900 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2446661,"reported_tablets":42,"child_traces":[["op",{"apply.queue_time_us":204371,"apply.run_cpu_time_us":2396,"apply.run_wall_time_us":2395,"num_ops":5,"prepare.queue_time_us":164816,"prepare.run_cpu_time_us":41245,"prepare.run_wall_time_us":132734,"raft.queue_time_us":319195,"raft.run_cpu_time_us":10898,"raft.run_wall_time_us":185141,"replication_time_us":949511,"thread_start_us":534600,"threads_started":4,"wal-append.queue_time_us":212849,"wal-append.run_cpu_time_us":29647,"wal-append.run_wall_time_us":1959099}]]}
I20260709 13:09:28.677976 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:28.680801 25239 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:09:28.508014 29808 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.154s
I20260709 13:09:28.683991 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:28.684142 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 199846us
I20260709 13:09:28.690680 25939 raft_consensus.cc:2393] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:28.689496 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.690536 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:28.695284 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 54082.2us
I20260709 13:09:28.696312 25285 leader_election.cc:400] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11.
I20260709 13:09:28.696599 25285 leader_election.cc:403] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 13:09:28.659493 29688 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:28.704514 29801 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:28.689342 29792 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:28.715796 25359 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:28.717425 25359 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:28.720757 25246 raft_consensus.cc:2410] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 13:09:28.657431 29685 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:28.744288 29804 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:28.686692 29580 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.691469 29794 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.665714 29812 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:28.874693 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 170361us
I20260709 13:09:28.981945 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:28.957464 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:29.049275 25318 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
W20260709 13:09:28.916319 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 209828us
W20260709 13:09:28.898034 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 200655us
I20260709 13:09:28.700405 29483 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:29.006318 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.901494 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.957643 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.062008 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.697474 29813 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:28.948107 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 230802us
I20260709 13:09:28.998612 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:28.981297 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.064515 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.984440 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:28.989194 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.068545 25395 raft_consensus.cc:2410] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 13:09:28.986817 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:28.960654 29685 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:29.063678 29681 consensus_meta.cc:287] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.604s	user 0.008s	sys 0.007s
I20260709 13:09:28.901854 29687 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:29.081517 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:28.875941 29818 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:29.081173 25389 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:29.078213 25390 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:29.085549 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.064675 29483 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:29.090246 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:28.915905 29772 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:28.877081 29792 raft_consensus.cc:697] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:29.096328 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:29.097267 25395 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:29.091169 25285 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:29.086822 25359 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:29.103034 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:28.876471 29694 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.209s
I20260709 13:09:29.082943 29681 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:29.109436 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:29.109402 29794 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:29.110121 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:29.089393 29688 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:29.110711 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.094192 29483 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:29.107856 25391 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:29.110350 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.105906 25394 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:29.117029 25390 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:09:29.118615 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.119599 25390 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:29.109778 29681 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:29.103865 29788 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:29.111640 29687 raft_consensus.cc:697] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:29.114295 29784 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:29.125154 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.128248 25389 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:29.131922 25387 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:29.132619 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.132870 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.137648 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.139355 25389 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:29.139410 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:29.139963 25209 leader_election.cc:336] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.748s (SENT)
I20260709 13:09:29.140350 25387 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:29.140499 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.141227 25389 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
W20260709 13:09:29.141502 25209 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.68:39211 timed out after 1.750s (SENT) [suppressed 46 similar messages]
I20260709 13:09:29.141672 25387 raft_consensus.cc:3055] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:29.142097 25387 raft_consensus.cc:740] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:29.139384 25391 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:29.143052 25387 consensus_queue.cc:260] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:29.143637 29808 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:09:29.144140 25387 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:09:29.145362 25212 leader_election.cc:336] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.640s (SENT)
I20260709 13:09:29.145907 25212 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:29.146930 29622 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:29.149879 29784 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:29.151015 29784 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:29.151978 29622 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:29.152570 29622 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:29.154990 29622 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:29.158375 25342 debug-util.cc:398] Leaking SignalData structure 0x7b08000facc0 after lost signal to thread 25178
I20260709 13:09:29.159075 29812 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:29.160185 25342 debug-util.cc:398] Leaking SignalData structure 0x7b080010b720 after lost signal to thread 25270
W20260709 13:09:29.163650 29799 consensus_meta.cc:287] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.501s	user 0.008s	sys 0.000s
I20260709 13:09:29.164598 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:29.150853 29828 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.015s
I20260709 13:09:29.166925 29580 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:29.170133 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:29.171784 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:29.171818 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:29.172602 25313 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:29.159596 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:29.166836 25942 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:09:29.181654 25391 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:29.181957 25394 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 13:09:29.183553 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.184517 25314 raft_consensus.cc:1275] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:09:29.181743 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:29.184859 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:29.130880 29688 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:29.200075 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:29.198956 25393 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 6.
I20260709 13:09:29.210562 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:29.212304 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:29.216478 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:29.216735 25342 debug-util.cc:398] Leaking SignalData structure 0x7b08001d3620 after lost signal to thread 25345
I20260709 13:09:29.216701 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:29.218305 25342 debug-util.cc:398] Leaking SignalData structure 0x7b08002de280 after lost signal to thread 25346
I20260709 13:09:29.199579 29828 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:29.219040 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.223537 25390 raft_consensus.cc:2410] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 13:09:29.209587 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:29.225039 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:29.226775 25394 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:29.224901 25936 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:29.230996 25944 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:29.234709 25937 raft_consensus.cc:2410] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
W20260709 13:09:29.236594 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 163531us
W20260709 13:09:29.238140 25342 debug-util.cc:398] Leaking SignalData structure 0x7b0800378860 after lost signal to thread 25419
I20260709 13:09:29.242678 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.244074 25395 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:29.237679 25911 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:29.245590 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:29.246012 25389 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:09:29.246608 25321 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:29.235113 29483 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:29.250952 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:29.251288 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:29.252171 25945 raft_consensus.cc:2410] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 13:09:29.252048 25240 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:29.252429 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:29.252730 25945 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 925) took 1033 ms. Trace:
I20260709 13:09:29.253783 25945 rpcz_store.cc:276] 0709 13:09:28.219628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:29.224716 (+1005088us) service_pool.cc:224] Handling call
0709 13:09:29.252663 (+ 27947us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37120}
I20260709 13:09:29.256060 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:29.256235 25239 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:29.245354 29828 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:29.259406 29580 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:29.207039 29808 consensus_queue.cc:1059] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 13:09:29.259728 29828 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:29.260030 29580 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:29.260351 25342 debug-util.cc:398] Leaking SignalData structure 0x7b08003180a0 after lost signal to thread 25969
I20260709 13:09:29.260433 29527 raft_consensus.cc:3055] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:29.245397 29734 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:29.268179 29527 raft_consensus.cc:740] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:29.268234 25315 raft_consensus.cc:2393] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 10: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:29.262243 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:29.263520 29831 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:29.270658 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:29.264261 29813 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11.
I20260709 13:09:29.265612 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:29.267517 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2072) took 1084 ms. Trace:
W20260709 13:09:29.267871 25342 debug-util.cc:398] Leaking SignalData structure 0x7b080035adc0 after lost signal to thread 29546
I20260709 13:09:29.268683 29734 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:29.273617 29831 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:29.274201 29831 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:29.278412 25245 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:09:29.281598 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:29.282444 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:29.282841 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:29.272567 25944 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:29.286382 25240 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:29.269896 25945 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:29.272465 25244 rpcz_store.cc:276] 0709 13:09:28.140645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:28.143691 (+  3046us) service_pool.cc:224] Handling call
0709 13:09:29.197369 (+1053678us) spinlock_profiling.cc:243] Waited 3.67 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8de9c 00007f2973b8da5b 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:09:29.204885 (+  7516us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:29.210242 (+  5357us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:29.210293 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:29.217475 (+  7182us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:09:29.217546 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:29.217643 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:29.221841 (+  4198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8435584}
I20260709 13:09:29.294786 25390 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:09:29.301282 25944 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 13:09:29.304924 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:29.306915 29837 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:29.235929 29792 consensus_queue.cc:237] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:29.312985 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:29.318424 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:29.319370 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.426620 25393 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
W20260709 13:09:29.330374 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 65449.8us
I20260709 13:09:29.276603 29527 consensus_queue.cc:260] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:29.331983 29808 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:29.332206 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:29.332573 29794 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:29.318060 29837 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:29.333369 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:29.283537 29831 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:29.342360 25342 debug-util.cc:398] Leaking SignalData structure 0x7b080029cde0 after lost signal to thread 29709
I20260709 13:09:29.349503 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.462028 25390 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:29.256698 29687 consensus_queue.cc:237] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:29.351874 29580 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:29.352975 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:29.295212 29734 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:29.364387 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:29.319931 29483 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:29.334676 29772 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:29.373467 29777 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:29.375983 29798 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:29.376616 29779 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:29.362861 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:29.392102 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.397245 25387 raft_consensus.cc:1275] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20260709 13:09:29.504223 25387 pending_rounds.cc:85] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 5
I20260709 13:09:29.504109 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.504572 25387 pending_rounds.cc:107] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 13:09:29.405499 25394 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:29.504729 25944 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 12: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:29.504865 25387 raft_consensus.cc:2889] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:29.407867 25391 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:29.302165 25246 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:29.504261 25389 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:29.417919 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:29.504858 25390 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:29.421739 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:29.461089 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:29.508167 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.441205 25240 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:29.500155 25941 raft_consensus.cc:2410] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
W20260709 13:09:29.500705 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 147874us
I20260709 13:09:29.413684 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:29.510110 25316 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:29.510143 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:29.511209 29580 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:29.511572 29580 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:29.511898 29580 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:29.486092 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.505667 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:29.471900 29622 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:29.517634 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.518461 25391 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:29.519310 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.520325 25394 raft_consensus.cc:2410] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:09:29.513494 29772 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:29.537364 29622 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:29.538422 29772 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:29.539038 29772 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:29.541250 29777 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:29.541918 25386 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:29.543823 29777 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:29.534222 25209 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:29.546455 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:29.546953 29837 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:29.548033 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 51162.1us
I20260709 13:09:29.536464 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:29.549160 25392 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:09:29.536854 25388 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41830 (request call id 869) took 3568 ms (client timeout 3341 ms). Trace:
W20260709 13:09:29.549770 25388 rpcz_store.cc:269] 0709 13:09:25.917471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:27.374123 (+1456652us) service_pool.cc:224] Handling call
0709 13:09:29.443411 (+2069288us) spinlock_profiling.cc:243] Waited 52.9 ms on lock 0x7b54003d59d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:29.486095 (+ 42684us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":121645824}
W20260709 13:09:29.554960 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 49222.9us
W20260709 13:09:29.556931 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 177634us
W20260709 13:09:29.558524 25284 connection.cc:256] RPC call timeout handler was delayed by -2.4449291209999728s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 13:09:29.557709 25285 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:29.566821 25240 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:29.551178 29483 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:29.574401 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:29.517590 29704 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260709 13:09:29.576198 25212 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:29.628499 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:29.575135 29837 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:29.519001 29808 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:29.640184 25285 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:29.556273 29798 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:29.675838 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 51662.6us
W20260709 13:09:29.682201 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 134901us
I20260709 13:09:29.624002 29779 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:29.646150 29818 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:29.653981 29846 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:29.693569 29844 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:29.656697 29483 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:29.645143 29688 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.082s
I20260709 13:09:29.746663 29508 consensus_queue.cc:1059] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.067s
I20260709 13:09:29.747365 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:29.748927 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 189220us
I20260709 13:09:29.677634 29842 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.086s
I20260709 13:09:29.695542 29580 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:29.784505 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 104896us
I20260709 13:09:29.726531 29622 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:29.740973 29846 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:29.784834 25358 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:29.783861 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:29.747879 29779 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:29.790686 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 163372us
W20260709 13:09:29.799932 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 60268.1us
I20260709 13:09:29.787786 29831 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:29.791270 29813 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:29.815917 25212 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:29.790289 29846 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:29.753804 29818 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:29.824620 25943 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: []
W20260709 13:09:29.824002 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 80507.9us
I20260709 13:09:29.830103 29798 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:29.793071 29704 consensus_queue.cc:1059] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 13:09:29.827394 25212 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:29.818555 29813 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:29.793824 25143 catalog_manager.cc:5697] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:09:29.834079 29813 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:09:29.837805 25287 connection.cc:256] RPC call timeout handler was delayed by -3.8504311940000662s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17830000000000001s for a response
W20260709 13:09:29.838259 25287 connection.cc:256] RPC call timeout handler was delayed by -3.6938311940000403s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1957s for a response
W20260709 13:09:29.838546 25287 connection.cc:256] RPC call timeout handler was delayed by -3.5138311940000904s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2157s for a response
I20260709 13:09:29.839802 25316 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:29.840139 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:29.841508 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:29.863579 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:29.864468 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:29.866834 25316 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 10. Ops: [8.5-8.5]
I20260709 13:09:29.862594 29798 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:29.876322 25321 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
W20260709 13:09:29.876045 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 55611.7us
I20260709 13:09:29.876612 29862 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:29.874958 29861 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260709 13:09:29.881709 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:29.869268 29808 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:29.886545 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 74326.8us
I20260709 13:09:29.891722 25209 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:29.874927 29843 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
I20260709 13:09:29.874732 29858 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:29.898980 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:29.882294 29856 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 13:09:29.899856 25321 raft_consensus.cc:2376] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13. Re-sending same reply.
I20260709 13:09:29.863847 29779 consensus_queue.cc:1059] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 13:09:29.874773 29857 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:29.880424 29854 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:29.881949 29862 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:29.879479 29842 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:29.915303 25285 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:29.911612 25143 catalog_manager.cc:5697] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:29.918159 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:29.918895 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:29.920162 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 108365us
I20260709 13:09:29.920826 25359 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:29.921681 29868 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:29.922158 29868 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:29.922557 29868 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:29.926939 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 99627.1us
I20260709 13:09:29.927273 29867 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:09:29.927950 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:29.927407 29872 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:29.929095 29872 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:29.928606 25212 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:29.929606 29872 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:29.930931 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:29.931093 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:29.931434 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:29.931922 25212 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:29.932560 29867 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:29.932996 29867 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:29.933379 29867 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:29.933419 29873 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:29.932044 25240 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:29.935668 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:29.937383 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:29.938839 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:29.939039 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:29.939600 25243 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:29.941998 29872 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:29.945945 29873 raft_consensus.cc:697] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:29.946557 29875 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:29.947263 29873 consensus_queue.cc:237] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:29.909590 29857 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:29.909744 29854 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:29.911175 29862 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:29.902066 29837 raft_consensus.cc:697] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:29.912966 29842 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:29.967522 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:29.963439 25143 catalog_manager.cc:5697] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:09:29.976450 25359 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:29.979801 25321 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: []
W20260709 13:09:29.978163 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 74401.5us
I20260709 13:09:29.956704 29580 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:29.966365 29854 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:09:29.990741 29849 log.cc:927] Time spent T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.221s	user 0.001s	sys 0.001s
W20260709 13:09:29.992192 29836 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.217s	user 0.001s	sys 0.000s
I20260709 13:09:29.993448 29866 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:09:29.993944 29866 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:29.994349 29866 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:09:29.994467 25243 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20260709 13:09:29.997767 29855 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
I20260709 13:09:29.976729 29842 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:29.956501 29857 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:29.958248 29483 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:30.000365 29813 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:29.918286 29798 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:29.972486 29687 raft_consensus.cc:3055] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:29.987872 25143 catalog_manager.cc:5697] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 5 to 9, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 9 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:09:30.020982 25359 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:29.909899 29858 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:30.003618 29808 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:30.027910 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:30.029088 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:30.030738 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:30.030864 29837 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:30.031441 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:30.033738 29858 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:30.038210 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:30.036794 25143 catalog_manager.cc:5697] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 6 to 10. New cstate: current_term: 10 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:09:30.042019 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:30.037611 25386 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:30.038221 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:30.030884 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:30.038648 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.040373 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.037322 25388 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
W20260709 13:09:30.043955 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 112107us
I20260709 13:09:30.039697 25314 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:09:30.042634 25392 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:30.043381 25319 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:30.048830 25390 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:30.051471 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.054581 25317 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:30.054673 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.054988 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.055922 25317 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:09:30.056018 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.057163 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.057753 25315 raft_consensus.cc:1217] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Deduplicated request from leader. Original: 8.5->[10.6-13.7]   Dedup: 10.6->[13.7-13.7]
I20260709 13:09:30.058477 25315 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14. Ops: [10.6-13.7]
I20260709 13:09:30.060974 29865 consensus_queue.cc:1059] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:09:30.061364 29847 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.014s
I20260709 13:09:30.062886 29859 raft_consensus.cc:3055] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:30.052976 25316 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
I20260709 13:09:30.064185 25212 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:30.020210 29687 raft_consensus.cc:740] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:30.057791 25320 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:30.061014 25285 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:30.043104 29808 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:09:30.084285 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 271993us
I20260709 13:09:30.083222 25143 catalog_manager.cc:5697] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 6 to 7, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 7 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:09:30.090613 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 104661us
I20260709 13:09:30.092348 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.091531 25143 catalog_manager.cc:5697] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 10 to 12, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 12 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:09:30.075850 29580 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.016s
I20260709 13:09:30.063328 29859 raft_consensus.cc:740] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:30.047125 29818 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:30.099965 25285 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:30.099241 25314 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:30.101778 25359 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:30.091096 29808 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:30.103612 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:30.104859 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.106998 25284 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:30.053970 29854 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:30.099480 29861 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:30.118098 29874 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:30.118649 29874 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:30.119021 29874 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:30.122741 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.113862 25212 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:30.124102 25320 raft_consensus.cc:2410] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 13:09:30.125330 29875 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:30.125581 29622 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:09:30.125982 29875 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:30.126485 29875 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:30.113814 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:30.125413 25212 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:30.131788 25359 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:30.110394 29483 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:30.140919 25212 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:30.149364 29869 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:30.113596 29818 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:30.122963 29861 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:30.171425 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:30.170306 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 57004.8us
I20260709 13:09:30.166458 29818 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:30.173758 29877 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:30.177263 25359 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:30.176301 25212 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:30.184954 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 70703.9us
I20260709 13:09:30.174130 29818 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:30.167626 29622 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:30.153743 29883 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:30.169939 29580 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:30.133929 25143 catalog_manager.cc:5697] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 reported cstate change: config changed from index 2 to 8, term changed from 6 to 10, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 10 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:30.131966 29846 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:30.190687 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.162230 29869 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:30.196761 25359 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:30.178330 29877 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:30.154878 29483 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:30.175261 29861 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:30.188023 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:30.191138 29772 consensus_meta.cc:287] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.642s	user 0.011s	sys 0.002s
I20260709 13:09:30.204427 29868 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:30.221212 25320 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:30.186977 29688 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:09:30.230393 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 63625.9us
I20260709 13:09:30.238541 29772 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:30.228179 29837 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:30.241390 29837 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:30.241426 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.241899 29837 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:30.242183 25314 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:30.163137 29687 consensus_queue.cc:260] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:30.243548 29687 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:30.228533 29877 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:30.245822 25237 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:30.247212 29871 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:30.247699 29871 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:30.247923 29868 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:30.248035 29871 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:30.240836 29772 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:30.249521 29882 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:30.250022 29882 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:30.250427 29882 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:30.251866 29860 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.138s
I20260709 13:09:30.253539 29880 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.099s
I20260709 13:09:30.254523 29878 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:30.255018 29878 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:30.255461 29878 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:30.198261 29883 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:30.257454 29883 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:30.200429 29580 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:30.259153 29846 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:30.260684 29813 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:30.262372 29879 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
I20260709 13:09:30.262733 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.263536 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.263952 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.179226 29859 consensus_queue.cc:260] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:30.268682 29859 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:09:30.269409 29851 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.454s	user 0.000s	sys 0.004s
I20260709 13:09:30.228394 29869 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:30.247931 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:30.271561 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:30.274649 25911 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:30.259408 29580 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:30.245410 25143 catalog_manager.cc:5697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:09:30.293855 25212 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:30.296409 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 106146us
I20260709 13:09:30.297101 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:30.297946 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:30.299971 25911 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:30.230047 29483 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:09:30.294579 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 66976.5us
I20260709 13:09:30.306689 25939 raft_consensus.cc:2368] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:09:30.303889 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:30.310287 25285 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:30.313032 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:30.313944 25944 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:30.315310 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.235653 29704 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:30.237947 29688 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:30.318837 25212 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:30.321061 25942 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:30.324572 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.317924 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:30.324349 25943 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:30.329785 25320 raft_consensus.cc:2410] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 13:09:30.238240 29847 consensus_queue.cc:1059] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.088s
I20260709 13:09:30.343485 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:30.344218 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:30.350723 25317 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:30.351193 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.350813 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.353372 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.329835 29688 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:30.324508 29704 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:30.329375 29831 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 13:09:30.366119 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 55945.4us
I20260709 13:09:30.333760 29772 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:30.367221 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:30.371060 25386 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:30.367057 25143 catalog_manager.cc:5697] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 5 to 7, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 7 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:09:30.373795 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.368448 25312 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:30.374459 25319 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:30.344847 29798 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:30.380501 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.382493 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:30.383471 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:30.382740 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:30.397243 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:30.373889 29880 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:30.374810 29772 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:30.411583 25243 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 13:09:30.416718 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.414412 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:30.408437 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:30.422616 25239 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:30.416326 25241 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:09:30.429486 25386 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
W20260709 13:09:30.431862 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 132750us
W20260709 13:09:30.420842 25908 leader_election.cc:336] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.651s (SENT)
I20260709 13:09:30.396857 29704 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:30.439674 25911 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:30.411162 29772 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:30.443347 25143 catalog_manager.cc:5697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 3 to 8, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 8 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
W20260709 13:09:30.460026 25908 leader_election.cc:336] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.722s (SENT)
I20260709 13:09:30.462280 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:30.405646 29880 raft_consensus.cc:697] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:30.413036 29865 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:30.477656 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:30.480130 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:30.399995 29894 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:30.416436 29891 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:30.493532 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:30.494400 25387 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:30.441808 29854 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:30.498239 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:30.490281 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:30.407608 29837 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:30.410421 29867 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:30.502784 25386 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:09:30.504854 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 65655.2us
I20260709 13:09:30.526903 25911 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:30.523706 25143 catalog_manager.cc:5697] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 2 to 9, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 9 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:09:30.528010 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 155540us
I20260709 13:09:30.534984 29895 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:30.535425 29895 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:30.535765 29895 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:30.537496 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:30.587092 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:30.533142 29844 consensus_meta.cc:287] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.749s	user 0.011s	sys 0.003s
I20260709 13:09:30.532763 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:30.534173 29845 log.cc:927] Time spent T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.832s	user 0.001s	sys 0.003s
W20260709 13:09:30.557437 29862 consensus_meta.cc:287] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.578s	user 0.000s	sys 0.012s
I20260709 13:09:30.527623 29831 raft_consensus.cc:697] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:30.533597 29527 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:30.614053 25143 catalog_manager.cc:5697] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:09:30.637583 25388 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:30.530798 29837 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:30.647562 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 187584us
I20260709 13:09:30.646579 25143 catalog_manager.cc:5697] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 9 to 10, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 10 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:09:30.502887 29874 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:30.652806 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:30.646174 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 163151us
W20260709 13:09:30.648411 25908 leader_election.cc:336] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.818s (SENT)
I20260709 13:09:30.653653 25317 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:30.629684 29844 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:30.534020 29890 consensus_queue.cc:1059] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.176s
W20260709 13:09:30.649796 29808 consensus_meta.cc:287] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.527s	user 0.011s	sys 0.007s
I20260709 13:09:30.538116 29865 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:30.662869 29893 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.175s
I20260709 13:09:30.634936 29862 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:30.665414 29898 consensus_queue.cc:1059] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 13:09:30.669250 25245 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
W20260709 13:09:30.674242 29886 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.294s	user 0.001s	sys 0.000s
I20260709 13:09:30.680966 25242 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:30.648602 29894 raft_consensus.cc:697] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:30.682299 25395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 890) took 1484 ms. Trace:
I20260709 13:09:30.692010 25386 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:09:30.693358 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.703727 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:30.696110 25395 rpcz_store.cc:276] 0709 13:09:29.184011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:29.250462 (+ 66451us) service_pool.cc:224] Handling call
0709 13:09:29.254476 (+  4014us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:29.315914 (+ 61438us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:29.315943 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:29.333548 (+ 17605us) log.cc:844] Serialized 26 byte log entry
0709 13:09:29.500987 (+167439us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:29.501035 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:29.501102 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:29.652706 (+151604us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b480018c258. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297bbdbaa1 00007f297bbd2b37 00007f297bbd29ca 00007f297bcca26f 00007f297bcc9f37 00007f297bcde55c 00007f297bcde508 00007f297bcde488 00007f297bcde443 00007f297bcdd6ec 00007f297ac07a96
0709 13:09:30.499378 (+846672us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:09:30.667919 (+168541us) raft_consensus.cc:1685] finished
0709 13:09:30.667948 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:30.668134 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9591,"raft.run_cpu_time_us":4018,"raft.run_wall_time_us":339605,"spinlock_wait_cycles":61101568,"thread_start_us":163612,"threads_started":1,"wal-append.queue_time_us":239752}
I20260709 13:09:30.714777 25244 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: [12.6-12.6]
I20260709 13:09:30.718281 29901 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:30.718930 29901 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:30.719275 29901 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:09:30.729293 25284 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.007s	sys 0.012s
W20260709 13:09:30.729900 25284 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.007s	sys 0.012s
I20260709 13:09:30.728817 29831 consensus_queue.cc:237] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:30.735152 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:30.736406 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 151068us
I20260709 13:09:30.679569 29808 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:30.748831 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.749900 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:30.754216 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:30.754346 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.755751 25938 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:30.762423 25321 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:30.657343 29880 consensus_queue.cc:237] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:30.764038 25317 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:30.766736 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:30.765829 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 97928.5us
I20260709 13:09:30.744166 29867 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:30.769683 25908 leader_election.cc:336] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.940s (SENT)
I20260709 13:09:30.770843 25908 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:30.775933 25941 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:09:30.788914 25317 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 13:09:30.788021 29875 consensus_meta.cc:287] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.661s	user 0.016s	sys 0.003s
I20260709 13:09:30.743496 29877 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:30.809324 29903 consensus_queue.cc:1059] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.101s
W20260709 13:09:30.811295 25908 leader_election.cc:336] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.955s (SENT)
I20260709 13:09:30.814921 25284 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:30.817011 25284 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:30.810353 29896 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:30.887629 29906 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:30.823359 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:30.948608 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:30.986164 25395 raft_consensus.cc:2410] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 13:09:30.991789 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:30.992725 25317 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
W20260709 13:09:30.994829 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 183552us
W20260709 13:09:30.940672 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 123756us
W20260709 13:09:30.999984 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 253603us
I20260709 13:09:31.002916 25284 leader_election.cc:400] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:09:31.011013 25284 leader_election.cc:403] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:09:31.001009 25911 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:30.808764 29867 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:30.984097 29896 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:31.027354 25320 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:31.027467 29867 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:30.874272 29848 raft_consensus.cc:3055] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:09:31.029273 29848 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:31.029469 29865 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:31.030135 29865 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:31.030205 29848 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:31.030751 29867 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:31.031337 29848 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:30.994807 29906 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:31.036365 29878 consensus_meta.cc:287] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.780s	user 0.038s	sys 0.009s
W20260709 13:09:31.042624 29871 consensus_meta.cc:287] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.794s	user 0.008s	sys 0.010s
I20260709 13:09:31.049331 25388 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:09:31.037686 29912 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:31.053751 25938 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
W20260709 13:09:31.053824 25908 leader_election.cc:336] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.044s (SENT)
I20260709 13:09:31.055294 25908 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:31.056407 25908 leader_election.cc:336] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.107s (SENT)
W20260709 13:09:31.057340 25908 leader_election.cc:336] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.176s (SENT)
W20260709 13:09:31.061816 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:31.027446 29857 consensus_meta.cc:287] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.992s	user 0.007s	sys 0.012s
W20260709 13:09:31.066380 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 412463us
I20260709 13:09:30.987324 29874 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:31.030855 29865 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:31.036862 29878 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:30.807917 29875 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:30.789516 29894 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:31.057735 25908 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:31.076367 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:30.822074 29580 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:31.106492 29580 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:31.109222 29808 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:31.100893 25315 raft_consensus.cc:2410] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 13:09:31.114326 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:31.083814 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:31.088214 29921 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 14
I20260709 13:09:31.115088 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 791) took 1163 ms. Trace:
I20260709 13:09:31.115535 25242 rpcz_store.cc:276] 0709 13:09:29.951365 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:29.994386 (+ 43021us) service_pool.cc:224] Handling call
0709 13:09:29.994678 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:30.692887 (+698209us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:31.114748 (+421861us) spinlock_profiling.cc:243] Waited 686 ms on lock 0x7b54003a8ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:31.115003 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1577873792}
I20260709 13:09:31.115971 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:31.117425 29622 consensus_meta.cc:287] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.919s	user 0.010s	sys 0.012s
I20260709 13:09:31.118276 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:31.119163 25244 raft_consensus.cc:2410] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 13:09:31.119405 29862 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:31.120114 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:31.120185 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:31.090777 29842 consensus_meta.cc:287] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.041s	user 0.016s	sys 0.011s
I20260709 13:09:31.121498 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:31.121536 29842 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:31.122138 25392 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:31.123828 29842 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:31.089907 29772 consensus_meta.cc:287] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.602s	user 0.009s	sys 0.000s
I20260709 13:09:31.124763 29772 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:31.126823 29772 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:31.127151 29866 consensus_meta.cc:287] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.132s	user 0.013s	sys 0.000s
I20260709 13:09:31.118261 29622 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:09:31.102298 29687 consensus_meta.cc:287] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.858s	user 0.007s	sys 0.005s
I20260709 13:09:31.130223 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:31.131057 25212 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:31.075562 29877 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:31.131495 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:31.118947 29875 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:31.093709 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.132710 25244 raft_consensus.cc:2393] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 10: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:31.133803 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.135164 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.136194 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:31.137835 25313 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:09:31.128233 29921 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:09:31.116833 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:31.139024 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:31.104275 29483 consensus_meta.cc:287] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.771s	user 0.009s	sys 0.013s
I20260709 13:09:31.149847 29483 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:31.152369 29483 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:31.137676 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.119014 29794 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:31.153784 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:31.154501 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.154991 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.155110 25319 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:31.155961 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:31.092794 25239 consensus_meta.cc:287] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.659s	user 0.004s	sys 0.012s
I20260709 13:09:31.156610 25319 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:31.156785 25239 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:09:31.157719 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:31.158070 29896 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:31.158835 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:31.159932 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:31.160926 29901 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:31.161068 25239 raft_consensus.cc:2410] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:09:31.163097 25239 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:31.153831 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.164296 25312 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
W20260709 13:09:31.164969 29895 consensus_meta.cc:287] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.629s	user 0.000s	sys 0.026s
I20260709 13:09:31.165475 29895 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:31.167629 25239 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:31.163590 25314 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:31.168952 29925 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:31.168164 29794 raft_consensus.cc:697] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:31.170994 29794 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:31.168504 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 111204us
W20260709 13:09:31.174610 25908 leader_election.cc:336] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.230s (SENT)
I20260709 13:09:31.175240 25908 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:31.148910 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:31.176234 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 45286.5us
I20260709 13:09:31.176879 25313 raft_consensus.cc:2368] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13.
I20260709 13:09:31.156018 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:31.177634 29882 consensus_meta.cc:287] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.927s	user 0.013s	sys 0.000s
I20260709 13:09:31.140959 25244 raft_consensus.cc:2393] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 15: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:31.178092 29882 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:31.178609 29877 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:31.179189 29877 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:31.179526 25244 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 13:09:31.179605 29877 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:31.181914 25239 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:31.070101 29857 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:31.177816 25212 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:31.127671 29866 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:31.187343 25392 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:31.190181 25242 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:31.196240 25394 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 10: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:31.200023 29594 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:31.203804 25317 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:31.073695 29865 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:31.194847 25212 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:31.147084 29688 consensus_meta.cc:287] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.730s	user 0.010s	sys 0.006s
W20260709 13:09:31.214200 29818 consensus_meta.cc:287] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.984s	user 0.014s	sys 0.008s
I20260709 13:09:31.200773 29348 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:31.224206 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 782) took 1278 ms. Trace:
I20260709 13:09:31.224234 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:31.235329 25392 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:09:31.227869 25358 leader_election.cc:336] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.575s (SENT)
I20260709 13:09:31.238545 25395 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:09:31.210407 29861 consensus_meta.cc:287] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.965s	user 0.006s	sys 0.012s
I20260709 13:09:31.225858 25246 rpcz_store.cc:276] 0709 13:09:29.936993 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:29.937117 (+   124us) service_pool.cc:224] Handling call
0709 13:09:31.201707 (+1264590us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b5400059454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:31.215405 (+ 13698us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":602191616}
I20260709 13:09:31.254254 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.255357 25317 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:31.194854 29772 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:31.062258 29871 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:31.236780 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:31.249202 25212 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:31.223949 29594 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:31.220559 29865 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:09:31.250948 29831 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.022s	sys 0.036s
W20260709 13:09:31.295647 29831 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.023s	sys 0.036s
W20260709 13:09:31.269933 25358 leader_election.cc:336] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.614s (SENT)
W20260709 13:09:31.298400 25358 leader_election.cc:336] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.716s (SENT)
I20260709 13:09:31.298841 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:31.300050 25358 leader_election.cc:336] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.738s (SENT)
I20260709 13:09:31.300446 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:31.300596 29896 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:31.270361 29893 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:31.272200 29871 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:31.301620 29896 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:31.306618 29857 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:31.308094 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2277) took 1382 ms. Trace:
I20260709 13:09:31.308494 25316 rpcz_store.cc:276] 0709 13:09:29.925936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:29.926113 (+   177us) service_pool.cc:224] Handling call
0709 13:09:31.307915 (+1381802us) spinlock_profiling.cc:243] Waited 79.2 ms on lock 0x7b54000f9e54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:31.308014 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":182113152}
I20260709 13:09:31.264855 29772 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:31.264276 29861 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:31.309607 29772 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:31.310014 25358 leader_election.cc:336] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.778s (SENT)
W20260709 13:09:31.310966 25358 leader_election.cc:336] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.839s (SENT)
I20260709 13:09:31.311219 29861 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:31.267598 29901 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:31.313158 29878 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:31.320253 29882 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:31.311803 25358 leader_election.cc:336] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.946s (SENT)
W20260709 13:09:31.323117 25939 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.004s	sys 0.012s
W20260709 13:09:31.323715 25939 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.005s	sys 0.012s
I20260709 13:09:31.291529 25395 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:31.224591 29844 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:09:31.224334 29688 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:31.264515 29704 consensus_meta.cc:287] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.777s	user 0.005s	sys 0.006s
W20260709 13:09:31.361213 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:31.303951 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 54903.8us
I20260709 13:09:31.304155 25246 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14. Ops: []
W20260709 13:09:31.328191 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:31.303530 29594 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:31.333696 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.321655 29862 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:31.334455 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.335305 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.444303 25313 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:31.335693 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:31.445572 25314 raft_consensus.cc:2376] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11. Re-sending same reply.
I20260709 13:09:31.335978 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:31.394438 29704 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:31.338204 29844 raft_consensus.cc:697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:31.338932 29863 raft_consensus.cc:3055] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:31.340503 29688 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:31.344291 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:31.344859 29877 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:31.345569 25386 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:31.320878 29872 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:31.347710 29772 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:31.349458 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:31.462446 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.463774 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.464458 25316 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
W20260709 13:09:31.353603 29883 consensus_meta.cc:287] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.095s	user 0.013s	sys 0.010s
I20260709 13:09:31.353574 25392 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:31.354110 29906 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:31.391155 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.392509 29866 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:31.478515 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:31.308127 29869 consensus_meta.cc:287] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.998s	user 0.015s	sys 0.000s
I20260709 13:09:31.255263 29818 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:31.406862 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:31.410349 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 98565.4us
W20260709 13:09:31.413544 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 410798us
W20260709 13:09:31.483058 25358 leader_election.cc:336] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.994s (SENT)
I20260709 13:09:31.399714 29924 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:31.334988 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:31.445890 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 271284us
I20260709 13:09:31.503675 25320 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
W20260709 13:09:31.504021 25908 leader_election.cc:336] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.257s (SENT)
I20260709 13:09:31.460343 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:31.460331 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:31.505777 25246 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:31.473810 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.459817 29862 raft_consensus.cc:697] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:09:31.452293 29880 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.632s	user 0.057s	sys 0.098s
I20260709 13:09:31.391155 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:31.451630 29863 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:31.474802 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.560894 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:31.605671 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.606369 25313 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:31.607311 25387 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:31.588599 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:31.595364 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:31.610929 25240 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:31.507794 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:31.589298 29918 log.cc:927] Time spent T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.191s	user 0.000s	sys 0.001s
I20260709 13:09:31.589334 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:31.560654 29924 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:31.504452 29906 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:31.568786 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:31.607842 29880 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.058s	sys 0.099s
I20260709 13:09:31.504873 29878 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:31.628229 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:31.630832 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 147819us
I20260709 13:09:31.628660 25937 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:31.628134 29925 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:31.633671 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:31.634145 25944 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 13:09:31.596338 29772 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:31.634738 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.688133 25320 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:31.635380 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:31.689060 25313 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:31.689329 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2118) took 1030 ms. Trace:
I20260709 13:09:31.596516 29704 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:31.643545 29877 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:31.650559 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 146577us
I20260709 13:09:31.696416 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:31.653314 25314 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:31.636673 25391 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:31.660190 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:31.664678 25212 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:31.665016 25358 leader_election.cc:336] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.011s (SENT)
I20260709 13:09:31.698617 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:31.699087 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:31.658684 29906 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:31.700089 25944 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:31.668067 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:31.669533 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:31.671834 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:31.678402 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2113) took 1271 ms. Trace:
I20260709 13:09:31.703671 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:31.704496 25314 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:31.705648 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:31.679517 29818 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:31.682317 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:31.681362 25937 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:31.683509 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:31.710630 25938 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:31.710922 25239 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:31.684141 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:31.478571 29883 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:31.613153 29931 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:31.635995 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:31.484714 29869 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:31.712951 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:31.671933 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:31.727506 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:31.728350 25391 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:31.702828 25243 rpcz_store.cc:276] 0709 13:09:30.406813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:30.440011 (+ 33198us) service_pool.cc:224] Handling call
0709 13:09:30.473429 (+ 33418us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:30.706827 (+233398us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:09:30.706965 (+   138us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:30.735634 (+ 28669us) log.cc:844] Serialized 26 byte log entry
0709 13:09:30.819038 (+ 83404us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:30.819084 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:30.819156 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:31.678075 (+858919us) raft_consensus.cc:1685] finished
0709 13:09:31.678110 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:31.678314 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":58119,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":47459,"thread_start_us":275583,"threads_started":2,"wal-append.queue_time_us":327725}
I20260709 13:09:31.731271 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:31.708940 29772 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:31.732764 29931 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:31.733253 29772 raft_consensus.cc:697] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:31.733948 29906 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:31.734292 29772 consensus_queue.cc:237] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:31.735157 29931 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:31.736025 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 71047.2us
I20260709 13:09:31.737566 29594 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:31.737617 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:31.738446 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:31.739173 29869 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:31.739601 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:31.723105 29866 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:31.732765 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:31.741185 25393 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 10: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:31.741433 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:31.739202 25388 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:31.741760 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:31.742005 29862 consensus_queue.cc:237] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:31.745043 25394 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:31.750542 25941 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:09:31.689410 29878 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:31.760476 25243 raft_consensus.cc:1240] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:31.738878 29883 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:31.723170 29863 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:31.766038 29863 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:31.767753 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:31.661854 29844 consensus_queue.cc:237] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:31.770339 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:31.771153 25390 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:31.743947 29848 consensus_meta.cc:287] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.711s	user 0.010s	sys 0.002s
I20260709 13:09:31.762796 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:31.780400 25941 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:09:31.859637 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:31.860320 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:31.882889 25319 raft_consensus.cc:2410] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 13:09:31.892596 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:31.900780 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:31.922307 29932 log.cc:927] Time spent T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 13:09:31.937826 25243 raft_consensus.cc:2410] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 13:09:31.952040 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2143) took 1232 ms. Trace:
W20260709 13:09:31.956050 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 330746us
I20260709 13:09:31.958004 25239 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:31.960613 25239 raft_consensus.cc:1275] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
W20260709 13:09:31.958288 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:31.963472 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:31.964350 25941 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:31.965698 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:31.966593 25937 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:31.970431 25938 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
W20260709 13:09:31.987614 29865 consensus_meta.cc:287] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.674s	user 0.006s	sys 0.003s
I20260709 13:09:31.988299 29865 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:31.990264 29912 raft_consensus.cc:3055] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:31.993368 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:31.996299 25393 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:31.999027 25287 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:32.010998 25390 raft_consensus.cc:2410] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 13:09:32.029933 25390 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:32.021202 29912 raft_consensus.cc:740] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:32.052567 25393 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 13:09:32.036849 25394 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:32.039422 25391 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
W20260709 13:09:32.040820 29940 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 13:09:32.042093 29927 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 13:09:32.043840 29941 log.cc:927] Time spent T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 13:09:32.044142 25287 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:32.037709 29779 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:32.057067 25941 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:32.061507 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:32.064029 25314 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
W20260709 13:09:32.069865 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 333099us
I20260709 13:09:32.068944 25209 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:32.139626 25393 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:32.146556 29813 raft_consensus.cc:2764] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election decision vote started in defunct term 11: won
W20260709 13:09:32.148039 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 77370.8us
I20260709 13:09:32.149868 25391 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:32.057065 29878 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:32.035964 29925 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.112s
I20260709 13:09:32.068370 29924 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:32.171494 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 112372us
I20260709 13:09:32.169636 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:32.168299 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:32.170605 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 187397us
W20260709 13:09:32.169740 25358 leader_election.cc:336] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.427s (SENT)
I20260709 13:09:32.061146 29910 consensus_queue.cc:1059] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.106s
I20260709 13:09:32.178241 25358 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:32.154255 29837 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:32.178259 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:32.178264 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:32.181514 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 137551us
I20260709 13:09:32.181654 29875 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:32.182868 29865 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:32.183277 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:32.193243 25391 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:32.183981 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:32.196242 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:32.199028 25318 raft_consensus.cc:3055] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:32.199509 25318 raft_consensus.cc:740] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:32.201658 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:32.202541 25394 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:32.213336 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:32.205561 25908 leader_election.cc:336] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.899s (SENT)
I20260709 13:09:32.215250 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:32.222635 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:32.222872 25359 leader_election.cc:336] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 1.687s (SENT)
I20260709 13:09:32.223496 25314 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:32.223382 25359 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:32.224429 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:32.224599 29878 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:32.225543 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:32.225808 25940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 901) took 2737 ms. Trace:
I20260709 13:09:32.235435 25940 rpcz_store.cc:276] 0709 13:09:29.488810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:29.498857 (+ 10047us) service_pool.cc:224] Handling call
0709 13:09:29.505341 (+  6484us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:29.554419 (+ 49078us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:09:29.623391 (+ 68972us) log.cc:844] Serialized 14 byte log entry
0709 13:09:32.181401 (+2558010us) spinlock_profiling.cc:243] Waited 7.6 ms on lock 0x7b1800429a00. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f297685223a 00007f2976cbca2a
0709 13:09:32.225423 (+ 44022us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:32.225476 (+    53us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:32.225527 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:32.225610 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:32.225727 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16070,"prepare.queue_time_us":83975,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":14201,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":72,"spinlock_wait_cycles":17481984,"thread_start_us":29614,"threads_started":1}
I20260709 13:09:32.191113 29837 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:32.243706 29837 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:32.212673 29875 raft_consensus.cc:697] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:32.247300 29875 consensus_queue.cc:237] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:32.240705 25939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 850) took 1467 ms. Trace:
I20260709 13:09:32.231083 29900 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:09:33.452243 29910 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.228s	user 0.003s	sys 0.017s
W20260709 13:09:33.642370 29910 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.418s	user 0.003s	sys 0.018s
W20260709 13:09:33.829169 25175 debug-util.cc:398] Leaking SignalData structure 0x7b0800098a60 after lost signal to thread 24113
W20260709 13:09:33.829777 25175 debug-util.cc:398] Leaking SignalData structure 0x7b08002fce60 after lost signal to thread 24116
W20260709 13:09:34.562887 25313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2334) took 2947 ms (client timeout 2111 ms). Trace:
W20260709 13:09:34.563670 25313 rpcz_store.cc:269] 0709 13:09:31.615244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:32.191909 (+576665us) service_pool.cc:224] Handling call
0709 13:09:34.562746 (+2370837us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:34.565830 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:34.567483 25313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2338) took 2949 ms (client timeout 2994 ms). Trace:
W20260709 13:09:34.571137 25909 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 3.572s (SENT) [suppressed 46 similar messages]
W20260709 13:09:34.568195 25313 rpcz_store.cc:269] 0709 13:09:31.618579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:34.565305 (+2946726us) service_pool.cc:224] Handling call
0709 13:09:34.567396 (+  2091us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:34.584787 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:34.586045 25313 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:09:34.586843 25313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2347) took 2934 ms (client timeout 2974 ms). Trace:
W20260709 13:09:34.587298 25313 rpcz_store.cc:269] 0709 13:09:31.652023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:34.584414 (+2932391us) service_pool.cc:224] Handling call
0709 13:09:34.586768 (+  2354us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:34.588502 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:34.589660 25313 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 13: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:09:34.590201 25313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 735) took 2840 ms (client timeout 2890 ms). Trace:
W20260709 13:09:34.591255 25313 rpcz_store.cc:269] 0709 13:09:31.749595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:34.588180 (+2838585us) service_pool.cc:224] Handling call
0709 13:09:34.590137 (+  1957us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:34.592463 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:34.593693 25313 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:34.648996 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:35.224962 29928 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:35.227376 29910 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:35.228061 29910 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:35.231132 29910 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:35.231794 29910 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:35.232472 29910 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:35.238662 29910 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
W20260709 13:09:35.337525 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 3.13175e+06us
W20260709 13:09:35.343603 25313 consensus_meta.cc:287] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.749s	user 0.006s	sys 0.002s
W20260709 13:09:35.344385 25908 connection.cc:256] RPC call timeout handler was delayed by -3.1108033540000406s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.42460000000000009s for a response
W20260709 13:09:35.345407 25908 connection.cc:256] RPC call timeout handler was delayed by -3.0269908660000056s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33410000000000006s for a response
W20260709 13:09:35.347183 25908 connection.cc:256] RPC call timeout handler was delayed by -2.6320033540000622s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.4778s for a response
I20260709 13:09:35.344966 25313 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
W20260709 13:09:35.348685 25313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2333) took 3733 ms (client timeout 3197 ms). Trace:
W20260709 13:09:35.349251 25313 rpcz_store.cc:269] 0709 13:09:31.614955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:34.592122 (+2977167us) service_pool.cc:224] Handling call
0709 13:09:35.348607 (+756485us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:09:35.350365 25313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 734) took 3650 ms (client timeout 3341 ms). Trace:
W20260709 13:09:35.350806 25313 rpcz_store.cc:269] 0709 13:09:31.700378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:35.350122 (+3649744us) service_pool.cc:211] Skipping call since client already timed out
0709 13:09:35.350273 (+   151us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:09:35.352360 25313 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:09:35.353114 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 737) took 3424 ms. Trace:
I20260709 13:09:35.353539 25313 rpcz_store.cc:276] 0709 13:09:31.928740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:35.351576 (+3422836us) service_pool.cc:224] Handling call
0709 13:09:35.351932 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:35.352822 (+   890us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:35.353058 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:35.355535 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 738) took 3405 ms. Trace:
I20260709 13:09:35.357515 25313 rpcz_store.cc:276] 0709 13:09:31.950023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:35.354583 (+3404560us) service_pool.cc:224] Handling call
0709 13:09:35.354907 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:35.355135 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:35.355167 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:35.355252 (+    85us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:35.355297 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:35.355359 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:35.355480 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:09:35.359930 25908 connection.cc:256] RPC call timeout handler was delayed by -1.7636568430000352s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16920000000000002s for a response
W20260709 13:09:35.361369 25908 connection.cc:256] RPC call timeout handler was delayed by -1.4720568430000185s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2016s for a response
W20260709 13:09:35.361840 25908 connection.cc:256] RPC call timeout handler was delayed by -1.4639568430000054s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2025s for a response
W20260709 13:09:35.362262 25908 connection.cc:256] RPC call timeout handler was delayed by -1.4486568429999807s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2042s for a response
W20260709 13:09:35.362668 25908 connection.cc:256] RPC call timeout handler was delayed by -1.337956843000029s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21650000000000003s for a response
I20260709 13:09:35.364369 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 739) took 3366 ms. Trace:
I20260709 13:09:35.364930 25313 rpcz_store.cc:276] 0709 13:09:31.997595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:35.360613 (+3363018us) service_pool.cc:224] Handling call
0709 13:09:35.360983 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:35.363887 (+  2904us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:35.363917 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:35.363991 (+    74us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:35.364046 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:35.364124 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:35.364279 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:09:35.386592 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:35.399713 29863 consensus_meta.cc:287] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 3.633s	user 0.010s	sys 0.028s
W20260709 13:09:35.452929 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 3.25897e+06us
I20260709 13:09:35.455752 25209 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:35.460646 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:35.467955 25911 connection.cc:256] RPC call timeout handler was delayed by -2.6070170980000285s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17720000000000002s for a response
W20260709 13:09:35.468528 25911 connection.cc:256] RPC call timeout handler was delayed by -2.4238109729999451s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2025s for a response
I20260709 13:09:35.470690 29900 consensus_queue.cc:1059] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:35.472457 29900 raft_consensus.cc:3055] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:35.472891 29900 raft_consensus.cc:740] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:35.474515 29900 consensus_queue.cc:260] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:35.475800 29900 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:35.476033 25240 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 13:09:35.482106 25911 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:32.270944 25939 rpcz_store.cc:276] 0709 13:09:30.747492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:30.747645 (+   153us) service_pool.cc:224] Handling call
0709 13:09:30.748140 (+   495us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:31.367155 (+619015us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:31.417809 (+ 50654us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:31.422249 (+  4440us) log.cc:844] Serialized 26 byte log entry
0709 13:09:31.624971 (+202722us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:31.625015 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:31.625086 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:32.214297 (+589211us) raft_consensus.cc:1685] finished
0709 13:09:32.214326 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:32.214516 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":266813,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"spinlock_wait_cycles":375680,"thread_start_us":718179,"threads_started":2,"wal-append.queue_time_us":345158}
W20260709 13:09:35.500545 25246 consensus_meta.cc:287] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 3.992s	user 0.033s	sys 0.020s
I20260709 13:09:35.501008 25246 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
W20260709 13:09:35.508834 25908 leader_election.cc:336] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.692s (SENT)
W20260709 13:09:35.510890 25246 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 2123) took 4160 ms (client timeout 1855 ms). Trace:
W20260709 13:09:35.511484 25246 rpcz_store.cc:269] 0709 13:09:31.344550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:31.445529 (+100979us) service_pool.cc:224] Handling call
0709 13:09:35.504455 (+4058926us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:09:35.541260 25908 leader_election.cc:336] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.016s (SENT)
W20260709 13:09:35.547808 25392 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2461) took 3387 ms (client timeout 1730 ms). Trace:
W20260709 13:09:35.549513 25908 leader_election.cc:336] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.042s (SENT)
W20260709 13:09:35.571429 25908 leader_election.cc:336] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.165s (SENT)
W20260709 13:09:35.570696 25392 rpcz_store.cc:269] 0709 13:09:32.160858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:35.547558 (+3386700us) service_pool.cc:211] Skipping call since client already timed out
0709 13:09:35.547726 (+   168us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:09:35.582755 25938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 949) took 3947 ms (client timeout 3197 ms). Trace:
W20260709 13:09:35.583341 25938 rpcz_store.cc:269] 0709 13:09:31.635034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:31.662747 (+ 27713us) service_pool.cc:224] Handling call
0709 13:09:35.582453 (+3919706us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b5400460fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:35.582630 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26095488}
W20260709 13:09:35.598197 29837 consensus_meta.cc:287] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 3.354s	user 0.009s	sys 0.018s
W20260709 13:09:35.600337 25908 leader_election.cc:336] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.579s (SENT)
I20260709 13:09:35.601275 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:35.602173 25241 raft_consensus.cc:2410] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 13:09:35.603083 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:09:35.600251 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 144596us
I20260709 13:09:35.603794 25241 raft_consensus.cc:2368] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8.
W20260709 13:09:35.603255 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:35.607214 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:35.607856 25241 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:35.610956 29875 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:35.612259 25238 raft_consensus.cc:1275] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:09:35.614248 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:35.614799 25209 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:35.615093 29875 raft_consensus.cc:697] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:35.616075 29875 consensus_queue.cc:237] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:35.614869 25238 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:35.622216 29772 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:09:35.629961 25393 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 3.447s	user 0.017s	sys 0.029s
W20260709 13:09:35.630394 25393 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 3.447s	user 0.018s	sys 0.030s
I20260709 13:09:35.616967 25209 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:35.633745 25241 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:35.637768 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:35.638909 25241 raft_consensus.cc:2393] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:31.689812 25244 rpcz_store.cc:276] 0709 13:09:30.659348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:31.180510 (+521162us) service_pool.cc:224] Handling call
0709 13:09:31.184307 (+  3797us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:31.184499 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:09:31.604577 (+420078us) log.cc:844] Serialized 14 byte log entry
0709 13:09:31.688826 (+ 84249us) log.cc:844] Serialized 14 byte log entry
0709 13:09:31.688983 (+   157us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:31.689025 (+    42us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:31.689079 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:31.689166 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:31.689277 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12178,"prepare.queue_time_us":452740,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":71,"raft.queue_time_us":207,"raft.run_wall_time_us":4669,"spinlock_wait_cycles":17426944,"thread_start_us":503223,"threads_started":2}
I20260709 13:09:35.599351 29837 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:35.671815 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 189694us
W20260709 13:09:35.673424 25908 leader_election.cc:336] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 3.341s (SENT)
I20260709 13:09:35.674006 25908 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:35.665925 29872 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:35.684607 29872 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:35.685153 29872 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:35.700001 29527 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:35.700667 29527 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:35.701783 25243 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 12. Ops: [10.7-10.7]
I20260709 13:09:35.702822 29837 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:35.702939 29527 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:35.704128 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 49108.2us
I20260709 13:09:35.706028 25238 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:35.703620 29527 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:35.759958 25209 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:35.762132 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:35.772277 29875 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:09:35.776298 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:35.740113 29527 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:35.776212 29900 consensus_queue.cc:1059] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:35.779328 29527 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:35.779619 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:35.780577 29875 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:35.781028 29875 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:35.781711 25209 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:35.783696 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 110321us
I20260709 13:09:35.784323 25209 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:35.787082 29772 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:35.786957 29837 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:09:35.787844 29772 raft_consensus.cc:2764] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 13:09:35.788766 29900 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:35.789323 29900 raft_consensus.cc:697] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:35.790246 29900 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:35.796141 29875 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:35.797804 25287 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:35.798626 29594 raft_consensus.cc:3055] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:35.799057 29594 raft_consensus.cc:740] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:35.800359 29907 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:09:35.801388 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2115) took 5393 ms. Trace:
I20260709 13:09:35.801755 25245 rpcz_store.cc:276] 0709 13:09:30.407641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:30.555605 (+147964us) service_pool.cc:224] Handling call
0709 13:09:30.613237 (+ 57632us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:30.682331 (+ 69094us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:35.801141 (+5118810us) spinlock_profiling.cc:243] Waited 55.7 ms on lock 0x7b54000a00d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:35.801325 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":128066432}
I20260709 13:09:35.803817 29594 consensus_queue.cc:260] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:35.805846 29594 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:09:35.807909 25237 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 779) took 5877 ms (client timeout 3572 ms). Trace:
W20260709 13:09:35.808333 25237 rpcz_store.cc:269] 0709 13:09:29.930691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:29.930831 (+   140us) service_pool.cc:224] Handling call
0709 13:09:35.807630 (+5876799us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b54002d0ad4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:35.807781 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":721817344}
I20260709 13:09:32.216583 25318 consensus_queue.cc:260] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:35.812088 25318 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:35.814374 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:35.815323 25239 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 10: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:09:35.818154 25395 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2447) took 4746 ms (client timeout 1523 ms). Trace:
W20260709 13:09:35.818655 25395 rpcz_store.cc:269] 0709 13:09:31.071178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:31.114024 (+ 42846us) service_pool.cc:224] Handling call
0709 13:09:35.817886 (+4703862us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54002832d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:35.818029 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":283938688}
I20260709 13:09:35.826663 25209 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:35.827764 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:35.831911 25318 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:09:35.832757 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2302) took 5511 ms. Trace:
I20260709 13:09:35.833169 25318 rpcz_store.cc:276] 0709 13:09:30.321378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:30.325246 (+  3868us) service_pool.cc:224] Handling call
0709 13:09:30.330497 (+  5251us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:32.198891 (+1868394us) spinlock_profiling.cc:243] Waited 34.8 ms on lock 0x7b5400321f20. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976c29dc0 00007f2976cb41c0 00007f2976cb5231 00007f2976cb1d8f 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c
0709 13:09:35.832417 (+3633526us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:35.832662 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":80061824}
W20260709 13:09:35.865024 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:35.870038 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:35.866256 29873 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:35.878163 29846 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:35.834626 29872 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:35.881217 29872 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:31.953208 25242 rpcz_store.cc:276] 0709 13:09:30.719865 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:31.204722 (+484857us) service_pool.cc:224] Handling call
0709 13:09:31.205297 (+   575us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:31.234030 (+ 28733us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:09:31.606884 (+372854us) log.cc:844] Serialized 14 byte log entry
0709 13:09:31.950660 (+343776us) log.cc:844] Serialized 14 byte log entry
0709 13:09:31.950848 (+   188us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:31.950890 (+    42us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:09:31.950938 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:31.951001 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:31.951974 (+   973us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":400390,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":175,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":4091,"spinlock_wait_cycles":135322112,"thread_start_us":600232,"threads_started":2,"wal-append.queue_time_us":337281}
W20260709 13:09:35.894322 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 89488us
W20260709 13:09:35.896596 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 3.72652e+06us
I20260709 13:09:35.904023 29872 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:35.920171 29875 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:35.938149 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1829) took 5116 ms. Trace:
I20260709 13:09:35.938663 25386 rpcz_store.cc:276] 0709 13:09:30.822084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:30.822238 (+   154us) service_pool.cc:224] Handling call
0709 13:09:30.822618 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:31.458578 (+635960us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:35.937773 (+4479195us) spinlock_profiling.cc:243] Waited 523 ms on lock 0x7b5400481554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:35.938039 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1202063616}
W20260709 13:09:35.940124 25386 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2460) took 3779 ms (client timeout 2018 ms). Trace:
W20260709 13:09:35.940515 25386 rpcz_store.cc:269] 0709 13:09:32.160558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:35.939918 (+3779360us) service_pool.cc:211] Skipping call since client already timed out
0709 13:09:35.940042 (+   124us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:09:35.949903 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:35.950583 29508 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:35.951253 29508 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:35.959965 25320 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
W20260709 13:09:35.968294 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 73145.8us
W20260709 13:09:35.970654 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 62412.9us
I20260709 13:09:35.971664 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:35.961302 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 287828us
I20260709 13:09:35.989012 25209 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:35.991178 25209 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:36.006238 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2457) took 3846 ms. Trace:
I20260709 13:09:36.010515 25209 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:36.006865 25393 rpcz_store.cc:276] 0709 13:09:32.159522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:32.161225 (+  1703us) service_pool.cc:224] Handling call
0709 13:09:32.176771 (+ 15546us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:32.176944 (+   173us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:09:35.630753 (+3453809us) log.cc:844] Serialized 14 byte log entry
0709 13:09:36.005769 (+375016us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:36.005866 (+    97us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:36.005928 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:36.006013 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:36.006166 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5707,"prepare.queue_time_us":1121311,"prepare.run_cpu_time_us":172,"prepare.run_wall_time_us":12380,"thread_start_us":3818787,"threads_started":3}
I20260709 13:09:35.992453 29844 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:09:36.027617 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 129797us
I20260709 13:09:36.016502 29873 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:36.011694 29872 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:09:36.064400 25209 connection.cc:256] RPC call timeout handler was delayed by -4.7056979619999311s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1523s for a response
I20260709 13:09:35.992115 29875 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:36.065212 29873 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:36.038019 29837 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 13:09:36.095789 25209 connection.cc:256] RPC call timeout handler was delayed by -4.62649796200003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16110000000000002s for a response
W20260709 13:09:36.132148 25209 connection.cc:256] RPC call timeout handler was delayed by -4.4860979620000307s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1767s for a response
I20260709 13:09:36.132302 29873 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:36.132565 25209 connection.cc:256] RPC call timeout handler was delayed by -4.3987979620000033s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1864s for a response
I20260709 13:09:36.104833 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:36.133227 25940 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:36.037904 29844 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:36.134142 29844 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:09:36.132937 25209 connection.cc:256] RPC call timeout handler was delayed by -4.3564979619999349s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19110000000000002s for a response
I20260709 13:09:36.119225 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:36.137107 25209 connection.cc:256] RPC call timeout handler was delayed by -4.3564979619999349s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19110000000000002s for a response
W20260709 13:09:36.137476 25209 connection.cc:256] RPC call timeout handler was delayed by -4.34479796200003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19240000000000002s for a response
W20260709 13:09:36.122771 29847 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 14 is higher.
W20260709 13:09:36.137648 25359 connection.cc:256] RPC call timeout handler was delayed by -3.7200259410000172s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20920000000000002s for a response
W20260709 13:09:36.137781 25209 connection.cc:256] RPC call timeout handler was delayed by -4.29529796199995s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19790000000000002s for a response
W20260709 13:09:36.138127 25359 connection.cc:256] RPC call timeout handler was delayed by -3.713725941000007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20990000000000003s for a response
W20260709 13:09:36.138223 25209 connection.cc:256] RPC call timeout handler was delayed by -4.29529796199995s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19790000000000002s for a response
W20260709 13:09:36.138800 25209 connection.cc:256] RPC call timeout handler was delayed by -4.1764979619999849s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21110000000000004s for a response
W20260709 13:09:36.140844 25209 connection.cc:256] RPC call timeout handler was delayed by -4.1764979619999849s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21110000000000004s for a response
W20260709 13:09:36.148461 25319 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2337) took 4530 ms (client timeout 1611 ms). Trace:
W20260709 13:09:36.146075 25209 connection.cc:256] RPC call timeout handler was delayed by -4.1314979620000258s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21610000000000001s for a response
W20260709 13:09:36.150627 25209 connection.cc:256] RPC call timeout handler was delayed by -4.0972979619999705s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21989999999999998s for a response
I20260709 13:09:36.103232 29900 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:09:36.177790 25209 connection.cc:256] RPC call timeout handler was delayed by -4.0972979619999705s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21989999999999998s for a response
W20260709 13:09:36.205945 25209 connection.cc:256] RPC call timeout handler was delayed by -3.9775979620000044s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2332s for a response
W20260709 13:09:36.219921 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:36.225811 25209 connection.cc:256] RPC call timeout handler was delayed by -3.851597962000028s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2472s for a response
I20260709 13:09:36.233566 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:36.234575 25391 raft_consensus.cc:2393] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:36.235662 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:36.236369 25391 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:36.144804 29844 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:36.239780 29844 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:36.226016 25390 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 4.015s	user 0.014s	sys 0.024s
W20260709 13:09:36.243171 25390 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 4.032s	user 0.014s	sys 0.025s
I20260709 13:09:32.235792 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:36.244149 25314 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2344) took 4623 ms (client timeout 2472 ms). Trace:
W20260709 13:09:36.244580 25314 rpcz_store.cc:269] 0709 13:09:31.620689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:32.235514 (+614825us) service_pool.cc:224] Handling call
0709 13:09:36.244046 (+4008532us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:09:36.245505 25285 connection.cc:256] RPC call timeout handler was delayed by -3.9978980890000457s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20779999999999998s for a response
I20260709 13:09:36.246646 29860 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:36.246762 25285 connection.cc:256] RPC call timeout handler was delayed by -3.4783063170000332s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1582s for a response
W20260709 13:09:36.247170 25285 connection.cc:256] RPC call timeout handler was delayed by -3.2326063170000907s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1855s for a response
W20260709 13:09:36.247588 25285 connection.cc:256] RPC call timeout handler was delayed by -3.1237063170000283s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1976s for a response
W20260709 13:09:36.247865 25285 connection.cc:256] RPC call timeout handler was delayed by -2.9248063170000478s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2197s for a response
W20260709 13:09:36.249059 25285 connection.cc:256] RPC call timeout handler was delayed by -2.8339063170000145s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2298s for a response
W20260709 13:09:36.249368 25285 connection.cc:256] RPC call timeout handler was delayed by -2.626197585s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1593s for a response
W20260709 13:09:36.249665 25285 connection.cc:256] RPC call timeout handler was delayed by -2.5325448130000723s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2189s for a response
I20260709 13:09:36.247170 29860 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:36.252830 29860 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:36.148976 25319 rpcz_store.cc:269] 0709 13:09:31.617550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:31.618134 (+   584us) service_pool.cc:224] Handling call
0709 13:09:36.148144 (+4530010us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b54000f8f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:36.148329 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":452172544}
I20260709 13:09:36.260299 29895 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:36.230619 25209 connection.cc:256] RPC call timeout handler was delayed by -3.3997979619999796s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29740000000000005s for a response
I20260709 13:09:36.263145 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:36.264878 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:36.265697 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:36.268961 25209 connection.cc:256] RPC call timeout handler was delayed by -3.3997979619999796s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29740000000000005s for a response
W20260709 13:09:36.270053 25209 connection.cc:256] RPC call timeout handler was delayed by -3.3817979619999505s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29940000000000005s for a response
I20260709 13:09:36.267452 25936 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
W20260709 13:09:36.270347 25209 connection.cc:256] RPC call timeout handler was delayed by -3.3493979620000118s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.303s for a response
W20260709 13:09:36.268337 25284 connection.cc:256] RPC call timeout handler was delayed by -4.072279177999917s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1593s for a response
W20260709 13:09:36.271483 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:36.270723 25209 connection.cc:256] RPC call timeout handler was delayed by -3.1990979619999962s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31970000000000004s for a response
W20260709 13:09:36.272337 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:36.277416 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:36.272279 25209 connection.cc:256] RPC call timeout handler was delayed by -3.1990979619999962s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31970000000000004s for a response
I20260709 13:09:36.283319 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:36.283515 25388 consensus_meta.cc:287] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 4.541s	user 0.032s	sys 0.036s
I20260709 13:09:36.284039 25388 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
W20260709 13:09:36.284601 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:36.284566 25388 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2453) took 5146 ms (client timeout 2472 ms). Trace:
W20260709 13:09:36.285418 25388 rpcz_store.cc:269] 0709 13:09:31.138592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:31.738214 (+599622us) service_pool.cc:224] Handling call
0709 13:09:36.284504 (+4546290us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:36.286201 25242 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:36.287382 29924 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:09:36.318934 25394 consensus_meta.cc:287] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 4.116s	user 0.010s	sys 0.019s
I20260709 13:09:36.319571 25394 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
W20260709 13:09:36.320412 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:36.323894 25394 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2459) took 4160 ms (client timeout 1631 ms). Trace:
W20260709 13:09:36.324504 25394 rpcz_store.cc:269] 0709 13:09:32.160210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:32.169194 (+  8984us) service_pool.cc:224] Handling call
0709 13:09:36.320212 (+4151018us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:09:36.337857 25284 connection.cc:256] RPC call timeout handler was delayed by -3.7905791779999163s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1906s for a response
I20260709 13:09:36.342999 25942 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:36.343092 25391 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:09:36.358922 29894 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:09:36.362511 25359 connection.cc:256] RPC call timeout handler was delayed by -3.6912259409999706s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21240000000000003s for a response
W20260709 13:09:36.353634 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 304382us
W20260709 13:09:36.372984 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:36.454061 25359 connection.cc:256] RPC call timeout handler was delayed by -3.6840259409999589s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21320000000000003s for a response
W20260709 13:09:36.455173 25359 connection.cc:256] RPC call timeout handler was delayed by -3.6399259410000013s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21810000000000002s for a response
W20260709 13:09:36.455541 25359 connection.cc:256] RPC call timeout handler was delayed by -3.6192259409999679s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22040000000000004s for a response
W20260709 13:09:36.374125 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 381098us
W20260709 13:09:36.388082 25944 consensus_meta.cc:287] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 4.678s	user 0.031s	sys 0.032s
W20260709 13:09:36.390331 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:36.399544 29837 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:36.456296 25944 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7.
W20260709 13:09:36.455905 25359 connection.cc:256] RPC call timeout handler was delayed by -3.5517259409999724s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2279s for a response
W20260709 13:09:36.392316 25212 connection.cc:256] RPC call timeout handler was delayed by -3.3259174360000543s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16310000000000002s for a response
W20260709 13:09:36.456811 25359 connection.cc:256] RPC call timeout handler was delayed by -3.4283848779999744s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15700000000000003s for a response
W20260709 13:09:36.456933 25212 connection.cc:256] RPC call timeout handler was delayed by -3.1889867969999841s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19790000000000002s for a response
W20260709 13:09:36.457355 25359 connection.cc:256] RPC call timeout handler was delayed by -3.0445969549999745s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.197s for a response
W20260709 13:09:36.457378 25212 connection.cc:256] RPC call timeout handler was delayed by -3.092296258000033s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17300000000000001s for a response
W20260709 13:09:36.457940 25212 connection.cc:256] RPC call timeout handler was delayed by -3.0353552200000422s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2472s for a response
W20260709 13:09:36.457928 25359 connection.cc:256] RPC call timeout handler was delayed by -2.96087645099999s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1955s for a response
W20260709 13:09:36.458561 25212 connection.cc:256] RPC call timeout handler was delayed by -2.92812442200011s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20179999999999998s for a response
W20260709 13:09:36.457470 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 176177us
W20260709 13:09:36.458722 25359 connection.cc:256] RPC call timeout handler was delayed by -2.9030380140000034s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2134s for a response
W20260709 13:09:36.459259 25359 connection.cc:256] RPC call timeout handler was delayed by -2.5941259410000157s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33430000000000004s for a response
W20260709 13:09:36.459638 25359 connection.cc:256] RPC call timeout handler was delayed by -2.5906365700000151s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21660000000000001s for a response
W20260709 13:09:36.459965 25359 connection.cc:256] RPC call timeout handler was delayed by -2.0946513389999382s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2978s for a response
W20260709 13:09:36.462284 25284 connection.cc:256] RPC call timeout handler was delayed by -3.3434838179999815s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1582s for a response
W20260709 13:09:36.462734 25284 connection.cc:256] RPC call timeout handler was delayed by -2.9888838179999766s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1976s for a response
W20260709 13:09:36.463076 25284 connection.cc:256] RPC call timeout handler was delayed by -2.6990838179999628s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2298s for a response
I20260709 13:09:36.464519 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:36.466053 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:36.467489 25212 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:36.468858 25212 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:36.471304 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:36.472404 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:36.473045 25318 raft_consensus.cc:2393] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:36.477464 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:36.477757 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:36.478408 25242 raft_consensus.cc:2393] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:09:36.478722 25944 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 907) took 4753 ms (client timeout 1955 ms). Trace:
I20260709 13:09:36.477514 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:36.480485 25317 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:36.480782 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:36.486251 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:36.487113 25394 raft_consensus.cc:2368] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:09:36.477344 29895 consensus_queue.cc:1059] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:36.488600 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:36.489228 25388 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
W20260709 13:09:36.479130 25944 rpcz_store.cc:269] 0709 13:09:31.703401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:31.705250 (+  1849us) service_pool.cc:224] Handling call
0709 13:09:36.456752 (+4751502us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:36.408861 29875 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:36.503928 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:36.505298 25313 raft_consensus.cc:2376] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10. Re-sending same reply.
I20260709 13:09:36.508224 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:36.511211 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:36.512015 25391 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:36.512969 25237 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:36.519119 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:36.520087 25245 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 9: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:36.528127 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:36.529033 25242 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 10: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:36.535411 25209 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:36.478585 29865 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:36.478129 29860 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:36.547662 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:36.496516 29844 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:36.511582 29794 consensus_queue.cc:1059] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.043s
I20260709 13:09:36.554451 29594 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:36.554903 29772 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:36.527391 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:36.555505 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:36.572238 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2122) took 5228 ms. Trace:
I20260709 13:09:36.571499 29594 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:36.573236 29794 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:09:36.572232 25317 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:36.576813 29731 raft_consensus.cc:3055] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:36.577349 29731 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:36.578495 29731 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:36.495744 29872 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:36.502857 29895 raft_consensus.cc:3055] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:36.572660 25240 rpcz_store.cc:276] 0709 13:09:31.344247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:31.721790 (+377543us) service_pool.cc:224] Handling call
0709 13:09:31.756332 (+ 34542us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:35.476621 (+3720289us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:36.571890 (+1095269us) spinlock_profiling.cc:243] Waited 3.72 s on lock 0x7b5400363b54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:36.572169 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8553688320}
W20260709 13:09:36.562265 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 157857us
W20260709 13:09:36.601822 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 60895.3us
W20260709 13:09:36.606221 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 70895.6us
I20260709 13:09:36.607443 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:36.608613 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:36.599983 25285 leader_election.cc:336] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 1.593s (SENT)
I20260709 13:09:36.615644 25359 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:36.489303 29837 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:36.484105 29894 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:36.592180 29865 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:36.566892 29860 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:36.688047 29865 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:36.592372 29594 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:36.689338 25318 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:36.519759 29875 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:36.696580 25317 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:36.657773 29731 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:36.610179 29895 raft_consensus.cc:740] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:36.747009 25245 raft_consensus.cc:1813] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Handling vote request from an unknown peer a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:36.753978 25245 raft_consensus.cc:2410] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 13:09:36.753197 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:36.773329 25318 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:36.783854 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:36.810864 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 202282us
I20260709 13:09:36.657825 29772 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:36.814353 25390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2458) took 4654 ms. Trace:
I20260709 13:09:36.822427 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:36.829803 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:36.825503 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:36.828241 25390 rpcz_store.cc:276] 0709 13:09:32.159864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:32.161306 (+  1442us) service_pool.cc:224] Handling call
0709 13:09:32.161719 (+   413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:32.162013 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:09:32.210708 (+ 48695us) log.cc:844] Serialized 14 byte log entry
0709 13:09:36.524967 (+4314259us) log.cc:844] Serialized 14 byte log entry
0709 13:09:36.813438 (+288471us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:36.813573 (+   135us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:36.813798 (+   225us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:36.813955 (+   157us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:36.814281 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":579120,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":66,"thread_start_us":4608706,"threads_started":4}
W20260709 13:09:36.848299 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:36.849282 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:36.849895 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:36.801888 29873 consensus_meta.cc:287] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.652s	user 0.006s	sys 0.005s
I20260709 13:09:36.873680 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:36.879495 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:36.899959 25358 connection.cc:256] RPC call timeout handler was delayed by -3.936734834000049s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1553s for a response
W20260709 13:09:36.901698 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:36.902046 25358 connection.cc:256] RPC call timeout handler was delayed by -3.8825898399999232s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15700000000000003s for a response
W20260709 13:09:36.902427 25358 connection.cc:256] RPC call timeout handler was delayed by -3.7090348340000219s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1806s for a response
W20260709 13:09:36.902765 25358 connection.cc:256] RPC call timeout handler was delayed by -3.6811348340000904s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1837s for a response
I20260709 13:09:36.904228 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:36.904896 25394 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 7.
I20260709 13:09:36.918545 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:36.919492 25390 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:09:36.926589 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:36.933266 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 110888us
I20260709 13:09:36.936661 29837 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:36.936837 29772 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:36.937379 29837 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:09:36.937985 29837 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:36.938418 29837 raft_consensus.cc:697] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:36.858419 29794 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.066s
I20260709 13:09:36.941174 29794 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:09:36.943748 29794 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:09:36.945521 29860 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:36.946323 29894 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:36.924428 29895 consensus_queue.cc:260] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:36.947501 29895 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:36.947624 29772 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:36.949082 29772 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:36.955049 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:36.962644 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:36.947022 29594 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:36.973227 29772 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:36.973532 29878 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:36.973894 29772 raft_consensus.cc:697] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:36.974256 29878 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:37.021733 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 406105us
I20260709 13:09:37.025492 25390 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:36.924712 29900 raft_consensus.cc:3055] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:37.038249 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:36.882303 29873 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:37.052448 25394 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:37.058283 25944 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:37.063014 25390 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:37.077453 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:37.078087 25944 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:37.084036 29731 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:37.085959 25394 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20260709 13:09:37.095988 25394 pending_rounds.cc:85] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 6
I20260709 13:09:37.096267 25394 pending_rounds.cc:107] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260709 13:09:37.096493 25394 raft_consensus.cc:2889] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:09:37.099155 25911 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.015s	sys 0.023s
W20260709 13:09:37.099577 25911 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.016s	sys 0.023s
I20260709 13:09:37.103185 29875 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:37.104861 29508 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:37.037698 29844 raft_consensus.cc:697] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:37.053505 29900 raft_consensus.cc:740] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:37.136771 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:36.939249 29837 consensus_queue.cc:237] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:36.974982 29772 consensus_queue.cc:237] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:37.154580 29998 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:37.123579 29912 consensus_queue.cc:260] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:37.156163 29912 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:37.155930 29998 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:37.158226 29998 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:37.158998 29974 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:37.159590 29974 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:37.161440 29873 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:37.163832 29844 consensus_queue.cc:237] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:37.037460 29794 consensus_queue.cc:1059] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.088s
I20260709 13:09:37.167842 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:37.172683 25317 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:37.171756 29878 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:37.207732 29508 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:37.208175 29878 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:37.208261 29508 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:37.208930 29878 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:37.121264 29731 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:37.211051 29508 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:37.211310 29900 consensus_queue.cc:260] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:37.211853 29508 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:37.206734 29794 raft_consensus.cc:3055] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:37.212563 29900 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:09:37.230145 25358 connection.cc:256] RPC call timeout handler was delayed by -3.6352348340000162s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1888s for a response
W20260709 13:09:37.252022 25358 connection.cc:256] RPC call timeout handler was delayed by -3.5659348340000179s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1965s for a response
W20260709 13:09:37.252409 25358 connection.cc:256] RPC call timeout handler was delayed by -3.4993348340000239s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20390000000000003s for a response
W20260709 13:09:37.252694 25358 connection.cc:256] RPC call timeout handler was delayed by -3.4435348340000473s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2101s for a response
W20260709 13:09:37.253003 25358 connection.cc:256] RPC call timeout handler was delayed by -3.4309080279999762s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.4171s for a response
W20260709 13:09:37.253324 25358 connection.cc:256] RPC call timeout handler was delayed by -3.3184348340000724s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22400000000000003s for a response
W20260709 13:09:37.253624 25358 connection.cc:256] RPC call timeout handler was delayed by -3.0324237380000341s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1955s for a response
W20260709 13:09:37.253931 25358 connection.cc:256] RPC call timeout handler was delayed by -2.871323738000001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2134s for a response
W20260709 13:09:37.254202 25358 connection.cc:256] RPC call timeout handler was delayed by -2.8272440020000431s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16290000000000002s for a response
W20260709 13:09:37.254457 25358 connection.cc:256] RPC call timeout handler was delayed by -2.7048440020000726s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1765s for a response
W20260709 13:09:37.254716 25358 connection.cc:256] RPC call timeout handler was delayed by -2.668634834000045s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2962s for a response
W20260709 13:09:37.254956 25358 connection.cc:256] RPC call timeout handler was delayed by -2.6571440019999955s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18180000000000002s for a response
W20260709 13:09:37.255192 25358 connection.cc:256] RPC call timeout handler was delayed by -2.2908440019999716s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22250000000000003s for a response
W20260709 13:09:37.255437 25358 connection.cc:256] RPC call timeout handler was delayed by -2.1117237380000233s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2978s for a response
W20260709 13:09:37.255687 25358 connection.cc:256] RPC call timeout handler was delayed by -1.3944440020000002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32210000000000005s for a response
W20260709 13:09:37.300071 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:37.308967 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:37.265028 25317 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
W20260709 13:09:37.426589 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:37.426720 25390 raft_consensus.cc:2376] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8. Re-sending same reply.
I20260709 13:09:37.293421 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:37.432160 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:37.332967 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 51474.2us
W20260709 13:09:37.489411 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:37.468422 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:37.469558 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:37.470418 25287 connection.cc:256] RPC call timeout handler was delayed by -4.8992877599999929s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1918s for a response
W20260709 13:09:37.468860 25314 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.018s	sys 0.046s
W20260709 13:09:37.274293 29872 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.020s	sys 0.053s
I20260709 13:09:37.394479 29878 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:37.369323 29974 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:37.298559 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:37.493235 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:37.493778 25287 connection.cc:256] RPC call timeout handler was delayed by -4.7930877599999349s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2036s for a response
W20260709 13:09:37.578084 25287 connection.cc:256] RPC call timeout handler was delayed by -4.2314877599999363s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.266s for a response
I20260709 13:09:37.494763 25317 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:37.234158 29794 raft_consensus.cc:740] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:37.589354 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:37.590044 25317 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:37.375130 29731 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:37.493571 25209 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:37.246078 29508 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:09:37.488708 29895 consensus_meta.cc:287] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.541s	user 0.012s	sys 0.001s
I20260709 13:09:37.477689 29860 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:37.327821 29982 consensus_queue.cc:1059] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.199s
W20260709 13:09:37.550614 29985 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.202s	user 0.001s	sys 0.000s
I20260709 13:09:37.556974 29847 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:37.561947 29975 log.cc:927] Time spent T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.216s	user 0.000s	sys 0.001s
W20260709 13:09:37.562983 29980 log.cc:927] Time spent T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 13:09:37.636278 29837 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:09:37.652832 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 124841us
W20260709 13:09:37.656565 25908 leader_election.cc:336] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.025s (SENT)
I20260709 13:09:37.359021 29861 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:37.497618 29872 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.824s	user 0.024s	sys 0.067s
W20260709 13:09:37.617924 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 124441us
I20260709 13:09:37.635453 25317 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:09:37.658782 25944 raft_consensus.cc:2410] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 12 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 13:09:37.657855 25314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2355) took 1175 ms. Trace:
W20260709 13:09:37.672621 25359 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.013s	sys 0.015s
W20260709 13:09:37.684381 25359 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.015s	sys 0.021s
I20260709 13:09:37.634227 29508 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:37.644997 29847 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:37.684732 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 660377us
I20260709 13:09:37.673842 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:37.687578 25314 rpcz_store.cc:276] 0709 13:09:36.471229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:36.473291 (+  2062us) service_pool.cc:224] Handling call
0709 13:09:36.473645 (+   354us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:09:36.484366 (+ 10721us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:09:36.677857 (+193491us) log.cc:844] Serialized 14 byte log entry
0709 13:09:37.493732 (+815875us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:09:37.539836 (+ 46104us) log.cc:844] Serialized 78 byte log entry
0709 13:09:37.540033 (+   197us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:37.544719 (+  4686us) log.cc:844] Serialized 14 byte log entry
0709 13:09:37.545095 (+   376us) log.cc:844] Serialized 14 byte log entry
0709 13:09:37.643988 (+ 98893us) log.cc:844] Serialized 14 byte log entry
0709 13:09:37.645255 (+  1267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:37.645550 (+   295us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:37.645803 (+   253us) raft_consensus.cc:1685] finished
0709 13:09:37.645865 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:37.646475 (+   610us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49453,"prepare.queue_time_us":1311214,"prepare.run_cpu_time_us":126,"prepare.run_wall_time_us":3394,"raft.queue_time_us":63610,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":14010,"thread_start_us":881075,"threads_started":4,"wal-append.queue_time_us":302153}
I20260709 13:09:37.674240 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:37.724401 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 50240.5us
I20260709 13:09:37.725898 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:37.641721 29860 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:37.673377 29910 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:37.757956 29844 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:37.758296 29910 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:37.758567 29844 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:37.671723 29794 consensus_queue.cc:260] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:37.760861 29847 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:37.760968 29844 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:37.765549 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:37.766021 25243 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:37.772675 29844 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:09:37.727376 29995 log.cc:927] Time spent T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 13:09:37.705207 29861 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:37.747741 29997 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 13:09:37.774670 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:37.775152 29861 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:37.757675 29860 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:37.756569 29992 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.106s
I20260709 13:09:37.761725 29862 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:37.776062 29862 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:37.776582 29862 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:09:37.768821 25320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 733) took 6133 ms (client timeout 1692 ms). Trace:
W20260709 13:09:37.777671 25320 rpcz_store.cc:269] 0709 13:09:31.635171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:31.741493 (+106322us) service_pool.cc:224] Handling call
0709 13:09:37.768584 (+6027091us) spinlock_profiling.cc:243] Waited 4.18 s on lock 0x7b54000631d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:37.768732 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9616014080}
I20260709 13:09:37.766937 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:37.766140 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:37.771821 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 855) took 6267 ms. Trace:
I20260709 13:09:37.778779 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:37.779016 25240 raft_consensus.cc:2393] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:37.780099 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2279) took 7774 ms. Trace:
W20260709 13:09:37.775465 25312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 728) took 6438 ms (client timeout 2230 ms). Trace:
W20260709 13:09:37.715364 29980 log.cc:927] Time spent T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 13:09:37.681823 29837 raft_consensus.cc:697] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:37.789216 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:37.747596 29982 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 13:09:37.685782 29872 raft_consensus.cc:697] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:37.754048 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:37.771792 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:37.765940 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:37.778909 25942 rpcz_store.cc:276] 0709 13:09:31.504092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:31.504427 (+   335us) service_pool.cc:224] Handling call
0709 13:09:31.561321 (+ 56894us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:36.444481 (+4883160us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:37.771512 (+1327031us) spinlock_profiling.cc:243] Waited 4.78 s on lock 0x7b5400014d54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:37.771740 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10996300672}
W20260709 13:09:37.781299 25312 rpcz_store.cc:269] 0709 13:09:31.336988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:31.476730 (+139742us) service_pool.cc:224] Handling call
0709 13:09:37.774589 (+6297859us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b58000e3498. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8df8c 00007f2973b8def6 0000000000375939 00007f297bc011e0 00007f297bcca0c4 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:09:37.775399 (+   810us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":280187520}
I20260709 13:09:37.780468 25321 rpcz_store.cc:276] 0709 13:09:30.005621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:30.005753 (+   132us) service_pool.cc:224] Handling call
0709 13:09:30.027219 (+ 21466us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:30.765299 (+738080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:37.765099 (+6999800us) spinlock_profiling.cc:243] Waited 735 ms on lock 0x7b540002c6d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:37.780044 (+ 14945us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1690376064}
W20260709 13:09:37.808570 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 54647.7us
I20260709 13:09:37.806743 25317 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
W20260709 13:09:37.765708 29977 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.221s	user 0.001s	sys 0.001s
I20260709 13:09:37.728399 30001 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:37.845722 29875 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:37.846145 30001 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:37.849424 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:37.849603 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 896) took 7512 ms. Trace:
I20260709 13:09:37.789279 29844 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:37.843127 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:37.846385 29875 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:37.851948 25321 raft_consensus.cc:2393] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:37.842415 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:37.863548 25312 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:37.850857 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:37.822947 29992 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:37.864585 25320 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:37.866416 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:37.851052 29844 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:37.883351 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:37.885560 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:37.892056 29508 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:37.907845 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:37.908223 29837 consensus_queue.cc:237] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:37.909346 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:37.911028 25321 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
W20260709 13:09:37.913372 25212 leader_election.cc:336] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.523s (SENT)
I20260709 13:09:37.850140 25389 rpcz_store.cc:276] 0709 13:09:30.337060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:30.337266 (+   206us) service_pool.cc:224] Handling call
0709 13:09:30.337693 (+   427us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:30.417744 (+ 80051us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:09:35.956521 (+5538777us) spinlock_profiling.cc:243] Waited 47.1 ms on lock 0x7b18000136c0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f297685a4bc 00007f2973ba848b 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc2455 00007f297a6aa3c7 00007f2976cbc9a7 00007f2976ccc0bf
0709 13:09:35.956799 (+   278us) log.cc:844] Serialized 14 byte log entry
0709 13:09:36.495469 (+538670us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:36.517896 (+ 22427us) log.cc:844] Serialized 26 byte log entry
0709 13:09:36.518069 (+   173us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:36.518112 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:36.518186 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:37.849211 (+1331025us) raft_consensus.cc:1685] finished
0709 13:09:37.849340 (+   129us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:37.849524 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":250877,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":1021922,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":33699,"spinlock_wait_cycles":108250496,"thread_start_us":663828,"threads_started":4,"wal-append.queue_time_us":1395449}
I20260709 13:09:37.917850 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:37.919077 25389 raft_consensus.cc:2410] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 13:09:37.921692 25317 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:37.928627 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:37.929793 25389 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:09:37.931558 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:37.932839 25389 raft_consensus.cc:2410] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 13:09:37.937374 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:37.848999 30001 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:37.944021 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:37.946579 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:37.850754 29862 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:37.950507 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:37.951323 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:37.952546 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 140808us
I20260709 13:09:37.961496 25390 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:37.962925 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:37.972334 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:37.973089 25389 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:37.974005 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:37.975821 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:37.977550 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:37.978902 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:37.979727 25321 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:37.980412 29875 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:37.982267 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:37.982519 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:38.006037 25942 raft_consensus.cc:1813] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Handling vote request from an unknown peer a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:38.006685 25942 raft_consensus.cc:2410] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 13:09:37.984215 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:37.980672 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:37.984622 29992 consensus_queue.cc:1059] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260709 13:09:37.982280 25312 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "292e92613a634963a5de1f016eb951ea"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:09:38.012028 25312 raft_consensus.cc:606] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:37.983405 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:37.996124 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:37.987502 29979 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.012s
I20260709 13:09:37.958405 29508 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:37.993379 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:38.008081 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:38.013657 25938 raft_consensus.cc:2393] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 12: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:38.018793 25940 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:09:38.019505 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 52326us
W20260709 13:09:38.021881 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:38.022931 25316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 729) took 6626 ms (client timeout 2257 ms). Trace:
W20260709 13:09:38.023379 25316 rpcz_store.cc:269] 0709 13:09:31.396677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:31.476918 (+ 80241us) service_pool.cc:224] Handling call
0709 13:09:37.995936 (+6519018us) spinlock_profiling.cc:243] Waited 6.41 ms on lock 0x7b58000e3498. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:09:38.022846 (+ 26910us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14736256}
I20260709 13:09:38.029354 25321 raft_consensus.cc:3055] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:38.029765 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:38.010694 25358 leader_election.cc:336] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.553s (SENT)
I20260709 13:09:38.032482 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:37.874550 30000 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.165s
I20260709 13:09:38.034376 25945 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:38.029836 25321 raft_consensus.cc:740] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:38.041322 25316 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:38.039604 25320 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:09:38.049858 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:38.061951 25320 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:09:38.062538 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:38.036913 25389 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
W20260709 13:09:38.062849 25358 leader_election.cc:336] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.806s (SENT)
I20260709 13:09:38.063444 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:38.061249 25321 consensus_queue.cc:260] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:38.068109 25321 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:38.069470 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:38.070539 25320 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 13: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:09:38.071525 25358 leader_election.cc:336] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.837s (SENT)
I20260709 13:09:38.072172 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:38.074111 25316 raft_consensus.cc:3055] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:38.074539 25316 raft_consensus.cc:740] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:38.075595 25389 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:09:38.076725 25389 raft_consensus.cc:3055] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:38.077045 25389 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:38.058172 29992 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:38.034905 30003 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:38.080518 25316 consensus_queue.cc:260] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:38.082108 25320 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 10. Current term is 11. Ops: []
W20260709 13:09:38.083169 25358 leader_election.cc:336] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.888s (SENT)
I20260709 13:09:38.086215 25316 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:38.025857 29508 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:38.087571 25358 leader_election.cc:336] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.965s (SENT)
I20260709 13:09:38.088347 25358 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:38.089419 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:38.090052 25312 raft_consensus.cc:2376] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13. Re-sending same reply.
I20260709 13:09:38.091674 25389 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.096015 25321 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 6. (term mismatch)
I20260709 13:09:38.096583 25321 pending_rounds.cc:85] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 5
I20260709 13:09:38.097103 25321 pending_rounds.cc:107] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 12.6
I20260709 13:09:38.097594 25321 raft_consensus.cc:2889] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:09:38.098811 25358 leader_election.cc:336] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.039s (SENT)
W20260709 13:09:38.099501 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:38.099421 25240 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
W20260709 13:09:38.099740 25358 leader_election.cc:336] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.101s (SENT)
I20260709 13:09:38.100198 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:38.086683 29979 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:38.084180 29966 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:38.024151 29872 consensus_queue.cc:237] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:38.102445 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2365) took 1015 ms. Trace:
I20260709 13:09:38.102926 25321 rpcz_store.cc:276] 0709 13:09:37.086975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:38.024345 (+937370us) service_pool.cc:224] Handling call
0709 13:09:38.024704 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:38.102082 (+ 77378us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:38.102385 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:38.101665 29992 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:38.108582 25389 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:38.109287 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:38.110193 25358 leader_election.cc:336] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.240s (SENT)
I20260709 13:09:38.110842 25312 raft_consensus.cc:2376] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14. Re-sending same reply.
I20260709 13:09:38.111056 25316 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20260709 13:09:38.111416 25316 pending_rounds.cc:85] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 6
I20260709 13:09:38.111060 25358 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:38.111701 25316 pending_rounds.cc:107] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 13:09:38.111991 25316 raft_consensus.cc:2889] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:38.112679 29982 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:38.114508 29963 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:38.115218 29963 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.088529 30003 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:38.122584 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:38.125979 25358 leader_election.cc:336] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.962s (SENT)
I20260709 13:09:38.129354 29812 consensus_queue.cc:1059] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:09:38.130541 29812 raft_consensus.cc:3055] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:38.130892 29812 raft_consensus.cc:740] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:38.131726 29812 consensus_queue.cc:260] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:38.122046 29978 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:38.133612 29978 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.132910 29812 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:38.135532 29992 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:09:38.137727 25358 leader_election.cc:336] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 5.008s (SENT)
I20260709 13:09:38.138204 25358 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:38.141253 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:38.145695 29862 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:38.145992 29963 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:38.146577 29992 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:38.145738 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:38.149757 25392 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:09:38.151182 25392 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:09:38.152140 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:38.151815 29812 consensus_queue.cc:1059] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:09:38.152774 25392 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:38.152945 29812 consensus_queue.cc:1243] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:09:38.163161 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:38.163424 29992 raft_consensus.cc:3055] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:38.163861 25237 raft_consensus.cc:2376] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10. Re-sending same reply.
I20260709 13:09:38.164826 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:38.164889 25911 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:38.165711 25237 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 10: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:38.166254 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:38.166996 25238 raft_consensus.cc:2368] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8.
I20260709 13:09:38.167791 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:38.168320 25238 raft_consensus.cc:2376] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12. Re-sending same reply.
I20260709 13:09:38.169235 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.170162 25241 raft_consensus.cc:2410] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 13:09:38.165500 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.171375 25240 raft_consensus.cc:2410] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 13:09:38.171710 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.171959 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.172972 25243 raft_consensus.cc:2393] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 10: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:38.173046 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.173338 29974 raft_consensus.cc:3055] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:38.173753 25244 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:38.173760 29974 raft_consensus.cc:740] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:38.174976 29974 consensus_queue.cc:260] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:38.175967 25359 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:38.176347 29974 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:38.158720 25212 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:38.177680 25212 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:38.178872 25318 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2357) took 1707 ms (client timeout 1681 ms). Trace:
W20260709 13:09:38.179379 25318 rpcz_store.cc:269] 0709 13:09:36.471811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:36.480338 (+  8527us) service_pool.cc:224] Handling call
0709 13:09:38.178653 (+1698315us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b5400276d54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:38.178791 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":475377920}
I20260709 13:09:38.181895 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.182252 25359 leader_election.cc:400] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8.
I20260709 13:09:38.182684 25359 leader_election.cc:403] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 13:09:38.183861 25359 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:38.189561 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:38.195135 25359 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:38.202842 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:38.203851 25392 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
W20260709 13:09:38.168848 29900 consensus_meta.cc:287] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.902s	user 0.006s	sys 0.008s
I20260709 13:09:38.163885 29992 raft_consensus.cc:740] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:38.207697 29924 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:38.202253 29854 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:38.205686 25359 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:38.169499 29963 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:38.215646 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:38.217222 25936 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:38.216429 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:38.219399 25312 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:38.202253 29812 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:38.215199 29854 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:38.205092 29837 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:38.218710 25212 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:38.157104 29844 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.220551 29924 raft_consensus.cc:697] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:38.226161 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.225962 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.205369 30002 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 7. Current term is 8.
I20260709 13:09:38.227677 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:38.232264 25241 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:38.221741 29854 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:38.226902 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:38.236075 25392 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:38.236361 25390 raft_consensus.cc:2410] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 11 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 13:09:38.220479 29994 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20260709 13:09:38.227161 29988 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:38.237720 29988 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:38.219141 29993 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
W20260709 13:09:38.238215 29962 log.cc:927] Time spent T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.751s	user 0.004s	sys 0.000s
I20260709 13:09:38.213495 29862 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.019s
I20260709 13:09:38.215364 30000 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.019s
I20260709 13:09:38.221817 29979 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:38.233381 29924 consensus_queue.cc:237] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.217677 29963 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:38.213789 29978 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:38.228801 29812 raft_consensus.cc:697] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:38.238102 29988 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:38.221064 30006 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:38.246834 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:38.265439 25395 raft_consensus.cc:1240] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:38.265439 25389 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:09:38.266577 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2468) took 1008 ms. Trace:
I20260709 13:09:38.266955 25389 rpcz_store.cc:276] 0709 13:09:37.258090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:38.076355 (+818265us) service_pool.cc:224] Handling call
0709 13:09:38.076616 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:38.266317 (+189701us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:38.266522 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:38.245105 29963 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:38.267208 25392 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 13:09:38.270130 29910 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:38.269666 29812 consensus_queue.cc:237] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.249217 30006 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:38.291646 25386 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:09:38.293982 25386 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:09:38.278214 30006 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:38.295250 25386 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 13:09:38.303634 25386 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: []
W20260709 13:09:38.423013 25175 debug-util.cc:398] Leaking SignalData structure 0x7b080032eae0 after lost signal to thread 25178
I20260709 13:09:38.346192 25394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1848) took 1745 ms. Trace:
I20260709 13:09:38.455621 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:38.487215 25394 rpcz_store.cc:276] 0709 13:09:36.600941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:37.068208 (+467267us) service_pool.cc:224] Handling call
0709 13:09:37.068536 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:37.117642 (+ 49106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:38.345885 (+1228243us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b54000dded4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:38.346110 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":39106304}
I20260709 13:09:38.488579 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:38.489100 25175 debug-util.cc:398] Leaking SignalData structure 0x7b0800328160 after lost signal to thread 25270
I20260709 13:09:38.458148 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:38.458977 25320 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:09:38.489925 25239 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20260709 13:09:38.490355 25239 pending_rounds.cc:85] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 9
I20260709 13:09:38.490705 25239 pending_rounds.cc:107] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 13:09:38.460451 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:38.461098 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:38.490937 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:38.491958 25318 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 13: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:38.492517 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:38.492657 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:38.521898 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:38.474858 25315 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2301) took 8211 ms (client timeout 2181 ms). Trace:
I20260709 13:09:38.477480 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32994 (request call id 2159) took 1950 ms. Trace:
W20260709 13:09:38.478458 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:38.381968 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 860) took 6228 ms. Trace:
I20260709 13:09:38.523555 25245 rpcz_store.cc:276] 0709 13:09:36.526977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:36.530031 (+  3054us) service_pool.cc:224] Handling call
0709 13:09:38.476476 (+1946445us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b54003a8ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:38.476843 (+   367us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":401302912}
I20260709 13:09:38.318210 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1830) took 7361 ms. Trace:
W20260709 13:09:38.359462 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:38.523872 25941 rpcz_store.cc:276] 0709 13:09:32.153242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:32.153378 (+   136us) service_pool.cc:224] Handling call
0709 13:09:32.153716 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:32.153921 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:09:32.170998 (+ 17077us) log.cc:844] Serialized 14 byte log entry
0709 13:09:38.280925 (+6109927us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b180042c7c0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f297685223a 00007f2976cbca2a
0709 13:09:38.381208 (+100283us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:38.381280 (+    72us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:09:38.381342 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:38.381643 (+   301us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:38.381789 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":16965,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":2466,"raft.queue_time_us":137,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":747,"spinlock_wait_cycles":12064000,"thread_start_us":99987,"threads_started":1,"wal-append.queue_time_us":6217485}
I20260709 13:09:38.525575 25387 rpcz_store.cc:276] 0709 13:09:30.957336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:31.014519 (+ 57183us) service_pool.cc:224] Handling call
0709 13:09:31.039053 (+ 24534us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:31.607776 (+568723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:38.317861 (+6710085us) spinlock_profiling.cc:243] Waited 568 ms on lock 0x7b54000dd9d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:38.318122 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1306402688}
I20260709 13:09:38.523201 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:38.528056 25240 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
W20260709 13:09:38.527956 25908 leader_election.cc:336] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.890s (SENT)
I20260709 13:09:38.442466 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:38.528939 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:38.530082 25241 raft_consensus.cc:2393] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:38.530583 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:38.471835 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:38.531533 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.532019 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:38.532230 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:38.533509 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:38.533752 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:38.534768 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.502261 25242 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:38.496556 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:38.536307 25240 raft_consensus.cc:2393] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 12: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:38.489646 25316 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:38.536744 25243 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 9: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:09:38.500434 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:38.498164 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.491164 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:38.537920 25244 raft_consensus.cc:2393] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 12: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:38.492974 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:38.498232 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:38.538864 25321 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:09:38.522701 25315 rpcz_store.cc:269] 0709 13:09:30.263081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:30.263652 (+   571us) service_pool.cc:224] Handling call
0709 13:09:38.473969 (+8210317us) spinlock_profiling.cc:243] Waited 836 ms on lock 0x7b540005e454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:38.474397 (+   428us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1922029824}
I20260709 13:09:38.538765 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:38.501060 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:38.538305 25359 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:38.489626 25175 debug-util.cc:398] Leaking SignalData structure 0x7b08002fcd80 after lost signal to thread 25271
I20260709 13:09:38.460455 25317 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 11. Current term is 12. Ops: [9.9-11.10]
I20260709 13:09:38.543591 25317 raft_consensus.cc:3055] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Stepping down as leader of term 11
W20260709 13:09:38.543808 25175 debug-util.cc:398] Leaking SignalData structure 0x7b080032bc20 after lost signal to thread 25345
I20260709 13:09:38.544082 25317 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:38.538800 25318 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
W20260709 13:09:38.545982 25175 debug-util.cc:398] Leaking SignalData structure 0x7b08003fc2c0 after lost signal to thread 25419
I20260709 13:09:38.547226 25318 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 13:09:38.510442 25313 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260709 13:09:38.491109 25239 raft_consensus.cc:2889] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:38.549305 25313 pending_rounds.cc:85] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 7
W20260709 13:09:38.549412 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:38.549804 25313 pending_rounds.cc:107] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 13:09:38.550132 25313 raft_consensus.cc:2889] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:38.547009 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:38.550987 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 794) took 2271 ms. Trace:
I20260709 13:09:38.551388 25941 raft_consensus.cc:2376] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 7. Re-sending same reply.
I20260709 13:09:38.304651 29508 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:38.557267 25175 debug-util.cc:398] Leaking SignalData structure 0x7b080029ce80 after lost signal to thread 25969
I20260709 13:09:38.571209 25240 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:38.568365 29772 consensus_queue.cc:1059] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.004s
I20260709 13:09:38.328119 29594 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:38.311813 29794 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:38.610878 29996 raft_consensus.cc:3055] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:38.619112 29996 raft_consensus.cc:740] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:38.309684 29844 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:38.306422 29854 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.624579 29854 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:38.308104 29956 log.cc:927] Time spent T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
W20260709 13:09:38.308712 29957 log.cc:927] Time spent T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 13:09:38.645170 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:38.377516 29963 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.647110 29865 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:09:38.649924 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.651149 29963 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:38.651547 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:38.652554 25237 raft_consensus.cc:2376] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14. Re-sending same reply.
I20260709 13:09:38.366328 29988 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:38.333103 29992 consensus_queue.cc:260] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:38.340193 30003 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:38.384887 30006 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:38.659538 29988 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:38.661885 30006 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:38.335693 30005 log.cc:927] Time spent T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 13:09:38.535991 25209 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:38.534859 25314 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:38.558255 25239 rpcz_store.cc:276] 0709 13:09:36.279981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:38.488423 (+2208442us) service_pool.cc:224] Handling call
0709 13:09:38.489058 (+   635us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:38.550588 (+ 61530us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:38.550925 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:38.534847 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:38.569424 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:38.569813 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:38.570113 25244 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:38.569631 29837 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 13:09:38.571278 25175 debug-util.cc:398] Leaking SignalData structure 0x7b0800399980 after lost signal to thread 25970
I20260709 13:09:38.553579 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:38.585578 25316 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:38.586377 25321 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:09:38.585867 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.596565 25320 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:38.603143 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:38.606567 29910 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:38.609694 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.610087 29966 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:09:38.676954 25175 debug-util.cc:398] Leaking SignalData structure 0x7b080038a560 after lost signal to thread 29857
I20260709 13:09:38.601580 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.611421 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2361) took 1635 ms. Trace:
I20260709 13:09:38.616662 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:38.616333 29894 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:38.617604 25243 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:09:38.617698 25319 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
I20260709 13:09:38.610880 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:38.381735 29928 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:38.698417 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:38.699481 25943 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:38.699551 29928 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.620434 29996 consensus_queue.cc:260] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.604279 29974 consensus_queue.cc:1059] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 13:09:38.644657 29964 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:38.703361 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:38.662024 29919 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:09:38.667817 29903 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:38.668941 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 148145us
I20260709 13:09:38.669070 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.669395 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:38.670054 29854 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:38.670241 25241 raft_consensus.cc:2376] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9. Re-sending same reply.
I20260709 13:09:38.671658 25391 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:38.672915 25390 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:38.673600 25394 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:38.675410 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.676105 29772 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:38.602263 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:38.669266 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:09:38.688761 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:38.638427 29890 consensus_queue.cc:1059] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.025s
I20260709 13:09:38.679981 25319 pending_rounds.cc:85] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 6
I20260709 13:09:38.714463 25319 pending_rounds.cc:107] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 13:09:38.678282 25313 rpcz_store.cc:276] 0709 13:09:36.964873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:36.964990 (+   117us) service_pool.cc:224] Handling call
0709 13:09:38.468806 (+1503816us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x7b58000e3498. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8deee 0000000000375939 00007f297bc011e0 00007f297bcca0c4 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:09:38.509241 (+ 40435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:38.594081 (+ 84840us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:38.600618 (+  6537us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5292928}
I20260709 13:09:38.638238 29827 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.062s
I20260709 13:09:38.717247 25387 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:38.641634 25317 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.694422 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.698197 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:38.334939 29968 log.cc:927] Time spent T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:09:38.729203 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:38.705454 25941 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:38.677008 29966 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:38.709606 29992 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:38.710098 25242 raft_consensus.cc:2368] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12.
I20260709 13:09:38.710412 29854 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:38.714851 25319 raft_consensus.cc:2889] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:38.722589 29903 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.724298 25238 raft_consensus.cc:2393] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:38.724061 25911 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:38.725000 25240 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:38.724483 29964 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.725698 30006 raft_consensus.cc:3055] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:38.729478 29880 consensus_queue.cc:1059] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:38.732239 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:38.729979 29865 raft_consensus.cc:3055] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:38.743019 29854 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:38.753154 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:38.754376 25945 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
W20260709 13:09:38.755232 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:38.755792 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:38.746224 30006 raft_consensus.cc:740] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:38.758030 30006 consensus_queue.cc:260] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:38.759725 30006 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:38.749599 29865 raft_consensus.cc:740] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:38.755522 25358 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:38.762434 29865 consensus_queue.cc:260] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.767362 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:38.768513 25241 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:38.774910 29903 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:38.775051 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 51286.7us
I20260709 13:09:38.775810 29903 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:38.776226 29903 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:38.776587 29903 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:38.779261 29900 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:38.779834 25244 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:38.780366 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.780759 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.781215 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.785626 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.785349 29900 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.737092 29966 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:38.768450 25942 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:38.790441 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:38.791205 25386 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:38.767942 29865 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:38.788625 25312 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:38.794348 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:38.792905 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:38.735764 29910 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.815187 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:38.816130 25942 raft_consensus.cc:2376] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11. Re-sending same reply.
I20260709 13:09:38.819150 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.820221 25239 raft_consensus.cc:2393] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 8: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:38.824574 29910 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:38.833007 25943 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
W20260709 13:09:38.838589 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 83265.8us
I20260709 13:09:38.834985 25936 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:38.838608 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:38.796243 29964 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:38.876199 25941 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 8.
I20260709 13:09:38.874687 25390 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:38.878995 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.881842 25391 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:38.886229 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:38.885260 25245 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:38.898092 25240 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 10. Current term is 12. Ops: [10.9-10.9]
I20260709 13:09:38.899315 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.900543 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.901852 25243 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:38.903069 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.903789 25243 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 12: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:38.904623 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.905359 25243 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:38.905517 25943 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:38.906332 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:38.907249 25243 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:09:38.909566 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 87619.2us
I20260709 13:09:38.912207 29919 consensus_queue.cc:1059] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:38.913266 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2146) took 3234 ms. Trace:
I20260709 13:09:38.913488 29919 raft_consensus.cc:3055] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:38.914014 29919 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:38.914497 29854 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.915208 29919 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.916839 29919 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:38.917529 25358 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:38.918521 25245 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 10. Current term is 12. Ops: []
I20260709 13:09:38.917570 25240 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:38.920318 25240 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:38.922776 25319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2358) took 2450 ms. Trace:
I20260709 13:09:38.923278 25319 rpcz_store.cc:276] 0709 13:09:36.472094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:36.510951 (+ 38857us) service_pool.cc:224] Handling call
0709 13:09:38.478324 (+1967373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:38.766741 (+288417us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:38.921770 (+155029us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b540005e454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:38.922526 (+   756us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b540005e450. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:38.922697 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4662175872}
I20260709 13:09:38.926590 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:38.927340 25239 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:38.927847 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:38.929185 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:38.929986 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:38.931303 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:38.932708 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:38.933399 25237 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:38.960561 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:38.962150 25391 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 14. Ops: []
I20260709 13:09:38.934340 25245 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 13:09:38.934571 29928 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:38.938556 25394 raft_consensus.cc:2410] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
W20260709 13:09:38.964263 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:38.963796 29903 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.939674 29900 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:38.968531 25394 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:38.939884 29854 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:38.942898 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:38.969699 25945 raft_consensus.cc:2368] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11.
I20260709 13:09:38.943926 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:38.970433 25386 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:38.944298 25944 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:09:38.946049 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:38.944314 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:38.950774 25317 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:38.971745 25386 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:38.913753 25246 rpcz_store.cc:276] 0709 13:09:35.597837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:35.600890 (+  3053us) service_pool.cc:224] Handling call
0709 13:09:35.601264 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:35.601646 (+   382us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:35.601676 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:35.602199 (+   523us) log.cc:844] Serialized 26 byte log entry
0709 13:09:35.696230 (+ 94031us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:35.696545 (+   315us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:35.696883 (+   338us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:38.786558 (+3089675us) raft_consensus.cc:1685] finished
0709 13:09:38.788774 (+  2216us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:38.832282 (+ 43508us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3140,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":6228,"thread_start_us":92752,"threads_started":1,"wal-append.queue_time_us":2627968}
I20260709 13:09:38.943926 29996 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:38.956179 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:38.973680 25943 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:38.971705 25941 raft_consensus.cc:2376] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10. Re-sending same reply.
W20260709 13:09:38.951548 25908 leader_election.cc:336] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 3.572s (SENT)
I20260709 13:09:38.978585 29966 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.979920 25395 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:38.980499 25390 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:38.983245 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:38.983481 29862 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:38.984239 29862 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:38.986795 25940 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 12: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:38.986718 29862 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:38.989928 29903 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:38.991683 25317 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 9. (term mismatch)
I20260709 13:09:38.986713 25391 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 11. Ops: []
I20260709 13:09:39.010093 25317 pending_rounds.cc:85] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 8
I20260709 13:09:38.999898 25241 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:38.994297 29966 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:39.010560 25317 pending_rounds.cc:107] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 11.9
W20260709 13:09:39.022818 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:39.023967 25317 raft_consensus.cc:2889] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:39.036310 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:39.037269 25241 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:39.038962 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:39.042225 25238 raft_consensus.cc:2410] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 13:09:39.039039 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:39.049067 25244 raft_consensus.cc:2410] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 13:09:39.049305 25237 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:39.041410 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:39.050467 25245 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
W20260709 13:09:39.050846 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 160358us
I20260709 13:09:39.052376 29992 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:39.056634 25209 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:39.057900 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:39.065342 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 103295us
W20260709 13:09:39.068136 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:39.070003 25209 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:39.059782 29854 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:39.077677 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:39.071977 25246 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:39.078926 25388 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 12: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:39.080579 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:39.082451 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:39.073868 29854 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:39.088798 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:39.093827 25315 raft_consensus.cc:1275] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:09:39.090655 29854 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:39.099006 29973 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:39.100612 29812 consensus_queue.cc:1059] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 13:09:39.093885 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:39.102908 29872 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:39.103338 29872 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:39.103704 29872 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:39.090883 29508 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:39.109414 29854 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:39.111162 25909 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:39.116291 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:39.117868 25909 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:39.119608 25909 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:39.121435 29903 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:39.122373 29903 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:39.123055 29903 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:39.123759 29903 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:09:39.124589 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:39.126408 25909 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:39.126430 29794 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:39.130412 29928 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:39.130841 29928 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:39.131243 29928 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:39.134912 29854 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:39.113461 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:39.101711 29900 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:39.127564 25358 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:39.140996 29812 raft_consensus.cc:3055] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:39.141515 25909 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:39.141520 29812 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:09:39.141804 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 54488.8us
I20260709 13:09:39.144958 25909 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:39.145982 29894 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:39.146538 29894 raft_consensus.cc:697] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:39.146706 29903 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:39.147289 29903 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:39.147748 29903 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:39.149561 29973 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:39.147743 29894 consensus_queue.cc:237] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:39.150511 29900 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:39.155195 25393 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:09:39.156502 30025 consensus_queue.cc:1059] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:09:39.156612 29903 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:39.157848 30025 raft_consensus.cc:3055] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:39.158243 30025 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:39.141520 25939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 965) took 1452 ms. Trace:
I20260709 13:09:39.159735 29903 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:39.159202 30025 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:39.160722 29928 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:39.163002 29928 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:39.164093 29812 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:39.182142 29812 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:39.182324 25942 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 13:09:39.164041 25241 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:39.165870 30025 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:39.167789 29910 consensus_queue.cc:1059] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:09:39.169803 29961 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:39.170535 29903 consensus_queue.cc:1059] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 13:09:39.174602 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:39.159073 25939 rpcz_store.cc:276] 0709 13:09:37.688983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:37.689175 (+   192us) service_pool.cc:224] Handling call
0709 13:09:37.751400 (+ 62225us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:39.096585 (+1345185us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:09:39.096638 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:39.096704 (+    66us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:39.096740 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:39.141065 (+ 44325us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b5400671e54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:39.141110 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:39.141319 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":134,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":116,"spinlock_wait_cycles":3078807296}
I20260709 13:09:39.179777 25359 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:39.216580 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:39.218288 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:39.222901 25241 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:09:39.217715 25942 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
W20260709 13:09:39.187773 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:39.193197 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 107628us
W20260709 13:09:39.228219 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:39.209868 29961 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:39.232368 29961 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:39.233291 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:39.234091 25389 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:39.237485 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:39.237849 25244 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:39.242601 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:39.215067 29862 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:39.244887 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:39.245150 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:39.253515 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:39.252388 25936 raft_consensus.cc:2376] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11. Re-sending same reply.
I20260709 13:09:39.227738 29966 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:39.210208 29794 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.045s
W20260709 13:09:39.274253 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:39.257530 29996 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:39.255832 29900 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:39.274158 25944 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:39.280676 25939 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:09:39.244817 30006 consensus_queue.cc:1059] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
W20260709 13:09:39.274354 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 47658.6us
W20260709 13:09:39.289189 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:39.285699 25936 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:39.267366 29862 raft_consensus.cc:697] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:39.269675 29988 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:39.293735 25212 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:39.274142 29924 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.050s
I20260709 13:09:39.272045 29966 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:09:39.302248 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:39.303488 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:39.317070 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 438060us
W20260709 13:09:39.325282 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:39.297899 29996 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:39.304849 29992 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:09:39.335757 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 95917.2us
I20260709 13:09:39.340452 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:39.337710 25209 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:39.339957 25389 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:39.330754 29854 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 13:09:39.342563 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 157767us
I20260709 13:09:39.316250 29988 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:39.353902 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:39.338680 29994 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.065s	sys 0.059s
I20260709 13:09:39.356451 29992 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:39.368743 29994 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.065s	sys 0.060s
I20260709 13:09:39.379093 29844 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:09:39.388965 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 95355us
I20260709 13:09:39.373755 29966 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:39.396826 25212 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:39.398705 29988 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:39.382334 29992 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:39.401937 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 64301.1us
I20260709 13:09:39.405522 25393 raft_consensus.cc:1275] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 13:09:39.414053 29996 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:39.414211 25392 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:09:39.414728 25388 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 13 index: 6. (index mismatch)
I20260709 13:09:39.415304 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:39.432399 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:39.375561 29862 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:39.426893 30006 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:39.427606 29964 consensus_queue.cc:1059] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:39.416647 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:39.432155 25393 pending_rounds.cc:85] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 5
I20260709 13:09:39.441695 25393 pending_rounds.cc:107] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 13:09:39.442101 25393 raft_consensus.cc:2889] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:09:39.446265 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 84564.9us
I20260709 13:09:39.455979 25386 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:09:39.428364 29924 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.017s
I20260709 13:09:39.484045 25209 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:39.441115 29827 consensus_queue.cc:1059] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 13:09:39.477560 29994 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:39.482246 29988 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:39.507889 25908 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:39.472930 30006 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:39.510465 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 50462.3us
W20260709 13:09:39.510597 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 99996.7us
I20260709 13:09:39.482769 29903 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.025s
I20260709 13:09:39.492568 29527 consensus_queue.cc:1059] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.072s
I20260709 13:09:39.492766 29854 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:39.518918 25209 leader_election.cc:400] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11.
I20260709 13:09:39.519286 25209 leader_election.cc:403] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:09:39.529961 30003 consensus_queue.cc:1059] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260709 13:09:39.530897 30038 consensus_queue.cc:1059] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20260709 13:09:39.509351 29988 raft_consensus.cc:697] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:39.533857 29988 consensus_queue.cc:237] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:39.537118 29994 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11.
I20260709 13:09:39.538295 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:39.538844 29992 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:39.540270 25209 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:39.541290 29996 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:39.542984 29996 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:39.543848 29996 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:39.523548 29910 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:09:39.545289 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 56086.2us
I20260709 13:09:39.547554 30006 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:39.548266 30006 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:39.548439 25908 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:39.548779 30006 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:39.549775 29527 raft_consensus.cc:3055] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:39.550158 29527 raft_consensus.cc:740] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:39.551005 29527 consensus_queue.cc:260] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:39.551779 25209 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:39.552635 29527 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:39.553663 29996 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:09:39.554257 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 159238us
I20260709 13:09:39.568960 25387 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:09:39.571158 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:39.572361 25358 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.550s (SENT) [suppressed 70 similar messages]
I20260709 13:09:39.580519 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:39.508980 29872 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:39.551025 30006 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:39.581622 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:39.587560 25944 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:39.582563 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:39.590688 25945 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: []
W20260709 13:09:39.593755 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 45374.2us
I20260709 13:09:39.570526 29844 raft_consensus.cc:697] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:39.592432 25940 raft_consensus.cc:2376] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13. Re-sending same reply.
I20260709 13:09:39.585602 29903 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:39.605439 25284 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:39.577878 29994 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
W20260709 13:09:39.642104 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 86126.8us
W20260709 13:09:39.648015 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 75689.7us
I20260709 13:09:39.600886 29854 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260709 13:09:39.664324 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:39.670569 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:39.674086 25244 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:09:39.647140 29961 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:39.659507 29910 consensus_queue.cc:1059] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 13:09:39.700091 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:39.697726 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:39.698464 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:39.701823 25244 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
W20260709 13:09:39.714483 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:39.719537 25394 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: []
W20260709 13:09:39.723078 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 133237us
I20260709 13:09:39.720976 25359 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:39.723958 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:39.724465 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:39.728130 30052 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 13:09:39.730464 29872 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:39.731257 30049 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:39.724985 29844 consensus_queue.cc:237] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:39.731846 30049 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:39.734155 30049 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:39.727238 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:39.737242 25359 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:39.738865 25284 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:39.740224 25244 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:09:39.718683 30055 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:39.745193 29961 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:39.753881 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:39.757182 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 218739us
I20260709 13:09:39.738533 29992 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:39.759580 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:39.764803 30063 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:39.768909 25943 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:39.754384 30006 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:39.747072 30055 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:39.755776 29854 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:39.754460 30003 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:39.772704 30065 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:39.741279 29996 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260709 13:09:39.754302 30056 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260709 13:09:39.761844 29992 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:09:39.781255 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:39.762300 29872 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:39.761018 29966 consensus_queue.cc:1059] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.017s
I20260709 13:09:39.780148 29854 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:39.792603 30065 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:39.771241 30006 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:09:39.802652 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 151169us
W20260709 13:09:39.806979 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 251525us
I20260709 13:09:39.778575 30055 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:39.834878 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 98549.6us
I20260709 13:09:39.851051 25212 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:39.826479 29992 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:09:39.864508 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 92270.6us
I20260709 13:09:39.853305 30006 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:39.866559 25212 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:39.863111 29527 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:09:39.866329 25287 connection.cc:256] RPC call timeout handler was delayed by -1.2250624350001544s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15300000000000002s for a response
W20260709 13:09:39.884089 25287 connection.cc:256] RPC call timeout handler was delayed by -1.1170624350000935s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.165s for a response
W20260709 13:09:39.885717 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 185725us
I20260709 13:09:39.865541 30056 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:39.888547 25209 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:39.861037 29872 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:39.882165 29527 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:39.886907 30003 raft_consensus.cc:697] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:39.917073 30076 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:39.917402 29973 raft_consensus.cc:3055] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:39.917851 29973 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:39.917734 30076 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:39.919188 25209 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:39.920143 30076 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:39.921487 30079 consensus_queue.cc:1059] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:39.921957 30057 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:09:39.922675 30057 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:39.918902 29973 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:39.932771 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:39.935205 30065 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:39.899179 30064 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:39.915467 30056 raft_consensus.cc:697] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:09:39.976336 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 55219.8us
I20260709 13:09:39.977434 29854 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:39.977576 30081 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:09:39.971050 29974 raft_consensus.cc:3055] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:39.978561 29974 raft_consensus.cc:740] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:39.917037 30080 consensus_queue.cc:1059] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.063s
I20260709 13:09:39.979483 29974 consensus_queue.cc:260] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:39.980510 30080 consensus_queue.cc:1059] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:09:39.966101 30066 consensus_queue.cc:1059] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.084s
I20260709 13:09:39.982195 30068 raft_consensus.cc:3055] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:39.982602 30068 raft_consensus.cc:740] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:39.983458 30068 consensus_queue.cc:260] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:39.984405 30068 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:39.977425 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:39.929461 30058 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:09:39.987442 29865 raft_consensus.cc:3055] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:39.987568 29508 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:39.987973 29865 raft_consensus.cc:740] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:39.988233 29508 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:39.988844 29865 consensus_queue.cc:260] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:39.981437 30073 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:09:39.913499 29527 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:39.984536 30006 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:39.980913 29974 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:40.019443 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:40.018162 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:40.019042 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:40.021354 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:40.030179 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:40.031068 25387 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:09:40.032230 30071 log.cc:927] Time spent T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.240s	user 0.000s	sys 0.002s
I20260709 13:09:39.973510 29872 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:40.039269 25388 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:09:40.042794 25391 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:09:40.047415 25394 raft_consensus.cc:2393] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 10: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:09:40.046782 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 121153us
W20260709 13:09:40.038663 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 155860us
I20260709 13:09:40.034255 29854 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:40.052722 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:40.033978 29973 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:09:40.047256 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 119780us
I20260709 13:09:40.034986 30073 consensus_queue.cc:1243] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer a2a77ecf2c0941618b5bc79d4c04db61 log is divergent from this leader: its last log entry 13.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:09:40.055365 25284 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:40.040202 30064 raft_consensus.cc:697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:40.064805 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:40.071192 30049 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.011s
I20260709 13:09:40.077221 30003 consensus_queue.cc:237] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:40.074252 30082 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:40.095355 30082 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:40.081651 30086 raft_consensus.cc:3055] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:40.096889 25388 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:09:40.062381 29865 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:40.100970 25287 leader_election.cc:400] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:09:40.113519 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:40.110107 25212 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:40.116616 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:40.116351 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:40.118396 25238 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:40.121932 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:40.122692 25939 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:40.071722 29872 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:40.096462 29508 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:40.082682 30006 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:40.093442 30087 raft_consensus.cc:3055] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:40.082028 30073 raft_consensus.cc:3055] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:40.077226 30056 consensus_queue.cc:237] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:40.134888 25237 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:09:40.107869 30086 raft_consensus.cc:740] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:40.107546 29854 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 13:09:40.130080 29900 raft_consensus.cc:3055] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:40.091636 30038 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:09:40.119531 29988 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.000s	sys 0.021s
I20260709 13:09:40.131415 25212 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:40.143520 25358 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:40.107132 30058 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260709 13:09:40.130471 29994 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:40.138612 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:40.116295 29996 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 13:09:40.173635 29996 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:40.174045 29996 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:40.174356 29996 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:40.156278 30061 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.053s
W20260709 13:09:40.163384 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 97912.2us
I20260709 13:09:40.142854 30073 raft_consensus.cc:740] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:09:40.163729 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:40.220072 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 169776us
I20260709 13:09:40.163983 25285 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:40.164565 30089 consensus_queue.cc:1059] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:09:40.133001 29844 consensus_queue.cc:1059] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 13:09:40.160892 25212 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:40.156569 29988 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.000s	sys 0.021s
I20260709 13:09:40.169260 30038 raft_consensus.cc:697] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 13 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:40.161239 25940 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:09:40.138069 30087 raft_consensus.cc:740] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:09:40.183549 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 75685.3us
W20260709 13:09:40.243794 25909 leader_election.cc:336] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 1.861s (SENT)
I20260709 13:09:40.163089 30086 consensus_queue.cc:260] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:40.184959 29966 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:40.188318 29961 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:40.191351 25939 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
W20260709 13:09:40.260481 25358 leader_election.cc:336] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.570s (SENT)
I20260709 13:09:40.077090 30064 consensus_queue.cc:237] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:40.155392 29900 raft_consensus.cc:740] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:09:40.243284 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:40.263299 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:40.219446 29508 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:40.260906 25358 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:40.258704 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 97894.7us
I20260709 13:09:40.267805 30073 consensus_queue.cc:260] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:40.268704 30079 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:40.268952 30073 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:40.269186 30101 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:40.262712 30057 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20260709 13:09:40.269797 30101 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:40.262943 29527 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:40.272694 29527 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:40.274387 30075 raft_consensus.cc:3055] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:40.274845 30075 raft_consensus.cc:740] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:40.275723 30075 consensus_queue.cc:260] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:40.276908 30076 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:09:40.281818 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 50450.2us
I20260709 13:09:40.282727 25238 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:40.283787 29900 consensus_queue.cc:260] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:40.285205 29900 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 12
I20260709 13:09:40.285718 30082 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:40.292845 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:40.298627 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:40.260205 29862 raft_consensus.cc:3055] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:40.263317 29988 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:09:40.322225 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:40.336769 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:40.338299 25940 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:40.340730 25909 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:40.341637 25909 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:40.337056 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 43595.4us
W20260709 13:09:40.342855 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:40.342286 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 100926us
W20260709 13:09:40.344101 25358 leader_election.cc:336] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.629s (SENT)
I20260709 13:09:40.344621 25358 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:40.347188 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:40.350100 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:40.351022 25237 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:40.352559 29994 raft_consensus.cc:697] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:40.353070 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:40.353657 25237 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:40.353673 29994 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:40.361832 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:40.362643 25246 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:40.363373 25237 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
W20260709 13:09:40.370798 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 206916us
I20260709 13:09:40.297762 30082 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:40.376904 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:40.379547 29964 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:40.303872 30006 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:40.383436 29964 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:40.384747 30006 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:40.385043 29894 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:40.295707 29872 raft_consensus.cc:697] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:40.385643 29894 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:40.385917 25909 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:40.386201 29894 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:40.386348 29872 consensus_queue.cc:237] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:40.387619 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:40.389487 29973 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:40.390151 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:40.390172 29973 raft_consensus.cc:697] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:40.391119 25942 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:40.391677 29973 consensus_queue.cc:237] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:40.393366 25942 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:09:40.396965 25944 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:09:40.304256 29862 raft_consensus.cc:740] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:40.341579 30081 raft_consensus.cc:3055] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:40.400202 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:40.373389 30113 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20260709 13:09:40.404336 30064 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:40.405107 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:40.407680 29992 consensus_meta.cc:287] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.525s	user 0.008s	sys 0.000s
I20260709 13:09:40.400269 30025 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:40.374076 29827 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:40.408670 29894 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:40.409827 30088 log.cc:927] Time spent T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.145s	user 0.000s	sys 0.000s
I20260709 13:09:40.415200 30075 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:40.415982 30089 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:40.474536 30089 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:40.416811 25285 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:40.370240 29961 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:40.370623 30086 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:40.423094 25315 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:09:40.424067 25940 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:40.425223 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:40.425458 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:40.425771 30087 consensus_queue.cc:260] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:40.343011 29988 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:40.291396 30079 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:40.262720 29966 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:40.340174 29854 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.096s
I20260709 13:09:40.473011 25209 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:40.482108 25943 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:40.428213 30081 raft_consensus.cc:740] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:40.470366 30025 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:40.485250 25318 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
W20260709 13:09:40.485551 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 68832.6us
W20260709 13:09:40.486119 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 240419us
I20260709 13:09:40.469852 29827 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:40.502810 25209 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:40.477542 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:40.402498 30082 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:40.505388 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 212384us
W20260709 13:09:40.513048 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 152600us
W20260709 13:09:40.520073 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:40.494189 29988 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:40.426785 30038 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:40.526785 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:40.473227 29992 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:40.528784 25390 raft_consensus.cc:2393] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 15: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:40.530016 29992 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:40.542881 25359 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:40.507987 30076 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:40.508447 29827 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:40.506188 30025 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:40.561497 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 217415us
I20260709 13:09:40.507725 30057 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:40.522399 30061 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:40.569296 25287 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:40.504184 30089 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:40.586913 25245 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 10. Current term is 12. Ops: []
I20260709 13:09:40.544142 29508 raft_consensus.cc:697] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:40.566179 29992 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:09:40.601903 25358 leader_election.cc:336] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.092s (SENT)
W20260709 13:09:40.599895 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 97187.1us
W20260709 13:09:40.605289 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:40.584555 30076 raft_consensus.cc:697] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:40.613744 25285 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:40.570112 30003 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20260709 13:09:40.624714 25909 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:40.625877 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:40.601678 30104 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:09:40.632723 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:40.625650 30063 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.010s
I20260709 13:09:40.635510 30090 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:40.635938 30090 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:40.636302 30090 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:40.636497 30084 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.011s
I20260709 13:09:40.563740 29862 consensus_queue.cc:260] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:40.639544 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 114795us
I20260709 13:09:40.647781 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:40.637789 29508 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:40.608314 29992 raft_consensus.cc:697] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:40.667357 25287 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:40.668401 29527 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:40.634516 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:40.663203 30055 consensus_meta.cc:287] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.778s	user 0.012s	sys 0.000s
W20260709 13:09:40.664474 29974 consensus_meta.cc:287] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.628s	user 0.010s	sys 0.000s
I20260709 13:09:40.633291 30104 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:09:40.721943 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 82145.8us
I20260709 13:09:40.722831 25285 leader_election.cc:400] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12.
I20260709 13:09:40.723131 25285 leader_election.cc:403] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
W20260709 13:09:40.728920 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:40.729732 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:40.730257 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:40.730383 25237 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:40.732148 30082 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:40.732807 30124 log.cc:927] Time spent T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.131s	user 0.001s	sys 0.000s
W20260709 13:09:40.733775 30106 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.155s	user 0.000s	sys 0.000s
I20260709 13:09:40.739022 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:40.739607 25237 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:09:40.749274 25209 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:40.750602 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:40.754385 25237 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:40.708568 30104 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:09:40.756870 29865 consensus_meta.cc:287] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.644s	user 0.008s	sys 0.000s
I20260709 13:09:40.732180 30098 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:09:40.796144 25314 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 13:09:40.770619 29996 consensus_meta.cc:287] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.596s	user 0.009s	sys 0.001s
W20260709 13:09:40.805255 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:40.805976 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 83286.6us
I20260709 13:09:40.806296 25321 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:40.748414 29894 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:40.702225 30055 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:40.773795 29872 raft_consensus.cc:2764] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 13:09:40.757733 30047 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:40.829080 25237 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 13:09:40.833699 25242 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 11. Ops: []
I20260709 13:09:40.757542 29827 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:40.776530 30089 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12.
I20260709 13:09:40.846506 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:40.847555 25241 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 8. Current term is 9. Ops: []
I20260709 13:09:40.846453 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:40.856000 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:40.821691 29966 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:40.905251 30101 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:40.834585 29996 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:40.842839 30098 raft_consensus.cc:697] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:40.818478 30079 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:40.909201 30127 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.209s	user 0.001s	sys 0.000s
I20260709 13:09:40.773882 30076 consensus_queue.cc:237] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:40.839715 29900 consensus_meta.cc:287] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.554s	user 0.009s	sys 0.001s
I20260709 13:09:40.743270 30081 consensus_queue.cc:260] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:40.819101 29992 consensus_queue.cc:237] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:40.913513 25246 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
W20260709 13:09:40.915952 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 317916us
W20260709 13:09:41.142051 25358 leader_election.cc:336] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.124s (SENT)
W20260709 13:09:40.945210 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:41.142508 25358 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:40.952369 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 97452.1us
I20260709 13:09:40.918673 25237 raft_consensus.cc:1240] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
W20260709 13:09:41.122862 30125 log.cc:927] Time spent T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.487s	user 0.001s	sys 0.000s
W20260709 13:09:41.036537 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:41.104614 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:40.915634 29865 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.187s
I20260709 13:09:41.098547 25938 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:40.857672 30047 raft_consensus.cc:697] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:09:40.857200 30124 log.cc:927] Time spent T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
W20260709 13:09:41.142989 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:40.982524 29988 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:41.110030 29900 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:41.150832 25937 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:41.164642 25287 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:41.142903 29827 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:41.164132 30038 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
W20260709 13:09:41.177419 30134 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.135s	user 0.000s	sys 0.001s
W20260709 13:09:41.178867 30112 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.200s	user 0.000s	sys 0.000s
I20260709 13:09:41.179301 29988 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:41.181829 30128 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
W20260709 13:09:41.208994 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 44467.3us
I20260709 13:09:41.182410 30137 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:41.182653 30110 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.348s	user 0.000s	sys 0.001s
I20260709 13:09:41.183598 29996 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:41.185385 25241 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:41.190517 29964 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:41.193034 25390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2507) took 1129 ms. Trace:
W20260709 13:09:41.195473 30132 log.cc:927] Time spent T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 13:09:41.196780 30055 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:41.197839 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 55839.1us
W20260709 13:09:41.219903 25358 leader_election.cc:336] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.765s (SENT)
W20260709 13:09:41.200511 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:41.220640 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:41.200687 30117 log.cc:927] Time spent T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.503s	user 0.000s	sys 0.001s
W20260709 13:09:41.202313 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:41.149559 30072 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:41.156371 29974 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:41.186540 30104 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.215660 25390 rpcz_store.cc:276] 0709 13:09:40.065325 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:40.072388 (+  7063us) service_pool.cc:224] Handling call
0709 13:09:41.192788 (+1120400us) spinlock_profiling.cc:243] Waited 414 ms on lock 0x7b54003e3d54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:41.192944 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":953168256}
I20260709 13:09:41.238832 25358 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:41.239691 25937 pending_rounds.cc:85] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 9
I20260709 13:09:41.258757 25937 pending_rounds.cc:107] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 13:09:41.259068 25937 raft_consensus.cc:2889] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:41.224941 29988 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:41.274008 25240 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:41.207527 29900 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.262050 29865 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:09:41.279892 29865 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:41.280368 29865 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:09:41.283870 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 63997.2us
I20260709 13:09:41.215319 30137 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.262446 29964 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:09:41.285630 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 71212.8us
I20260709 13:09:41.205705 30098 consensus_queue.cc:237] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:41.288491 25358 leader_election.cc:336] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.132s (SENT)
I20260709 13:09:41.291854 25287 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:41.292866 25940 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.007s	sys 0.006s
W20260709 13:09:41.294081 25940 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.007s	sys 0.006s
I20260709 13:09:41.294327 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 963) took 1093 ms. Trace:
I20260709 13:09:41.299252 25393 rpcz_store.cc:276] 0709 13:09:40.198235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:40.198380 (+   145us) service_pool.cc:224] Handling call
0709 13:09:40.198734 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:40.199346 (+   612us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:40.199431 (+    85us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:40.221868 (+ 22437us) log.cc:844] Serialized 26 byte log entry
0709 13:09:40.516918 (+295050us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:40.516978 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:40.517080 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:41.291393 (+774313us) raft_consensus.cc:1685] finished
0709 13:09:41.291422 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:41.291601 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":107,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":82,"thread_start_us":239987,"threads_started":1,"wal-append.queue_time_us":443607}
I20260709 13:09:41.313968 25359 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:41.319808 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:41.320751 25244 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:41.322572 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:41.298177 25942 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.006s	sys 0.009s
W20260709 13:09:41.338109 25942 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.006s	sys 0.009s
I20260709 13:09:41.258090 30072 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.338389 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:09:41.343559 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:41.282374 29988 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.358197 25359 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:41.370139 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 820) took 1017 ms. Trace:
I20260709 13:09:41.370802 25238 rpcz_store.cc:276] 0709 13:09:40.342169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:40.342288 (+   119us) service_pool.cc:224] Handling call
0709 13:09:41.338223 (+995935us) spinlock_profiling.cc:243] Waited 3.23 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:09:41.359456 (+ 21233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7440128}
W20260709 13:09:41.378907 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 90463.1us
I20260709 13:09:41.355976 29827 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:41.275676 30047 consensus_queue.cc:237] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.373294 29992 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:41.369547 29974 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:09:41.359995 30062 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:09:41.394990 30068 consensus_meta.cc:287] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.410s	user 0.010s	sys 0.005s
I20260709 13:09:41.398255 30068 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:41.399243 30068 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:09:41.393368 29827 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.404413 29827 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:41.405105 30062 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.412735 30062 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:41.408703 25358 leader_election.cc:336] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.181s (SENT)
I20260709 13:09:41.413913 25395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 958) took 1390 ms. Trace:
I20260709 13:09:41.414623 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2391) took 1038 ms. Trace:
I20260709 13:09:41.419926 25315 rpcz_store.cc:276] 0709 13:09:40.375925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:40.376084 (+   159us) service_pool.cc:224] Handling call
0709 13:09:40.402360 (+ 26276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:40.480199 (+ 77839us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:41.414393 (+934194us) spinlock_profiling.cc:243] Waited 20.3 ms on lock 0x7b54000870d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:41.414560 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46673664}
I20260709 13:09:41.416260 25939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 878) took 1122 ms. Trace:
I20260709 13:09:41.426570 25358 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:41.428611 29961 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:41.429520 29961 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:09:41.415452 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 123686us
I20260709 13:09:41.435616 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:41.438122 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:41.438969 25238 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:41.443526 25944 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:41.438849 25287 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:41.443905 30055 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:41.444514 30055 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.454284 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:41.455403 25937 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 12: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:41.458163 30070 raft_consensus.cc:3055] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:41.458907 30070 raft_consensus.cc:740] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:41.463469 30089 raft_consensus.cc:3055] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:41.464167 30089 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:41.466626 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:41.475625 30101 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:41.476310 30101 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:41.476961 30131 log.cc:927] Time spent T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.319s	user 0.001s	sys 0.001s
I20260709 13:09:41.478561 30006 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:41.479135 30006 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.481779 30061 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:41.482342 30061 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.484439 30061 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:41.475899 30089 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:41.486382 30089 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:41.491117 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 826) took 1129 ms. Trace:
I20260709 13:09:41.491493 30129 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:41.491595 25245 rpcz_store.cc:276] 0709 13:09:40.361756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:40.361905 (+   149us) service_pool.cc:224] Handling call
0709 13:09:40.362232 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:40.588856 (+226624us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:41.490851 (+901995us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b540027f954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:41.491028 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":516138752}
I20260709 13:09:41.492422 30129 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.493520 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:41.421492 25939 rpcz_store.cc:276] 0709 13:09:40.294209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:40.294370 (+   161us) service_pool.cc:224] Handling call
0709 13:09:40.303009 (+  8639us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:40.303214 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:09:40.470337 (+167123us) log.cc:844] Serialized 14 byte log entry
0709 13:09:40.470876 (+   539us) log.cc:844] Serialized 14 byte log entry
0709 13:09:41.316835 (+845959us) spinlock_profiling.cc:243] Waited 5.16 ms on lock 0x7b1800043ae0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc5522 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:09:41.415581 (+ 98746us) spinlock_profiling.cc:243] Waited 7.99 ms on lock 0x7b1800043ae0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b3fc0a 00007f297d27f219 00007f2976868d70 00007f297685c293 00007f2976851cea 00007f297685223a 00007f2976cbca2a 00007f2976ccc0bf 00007f2976ccc038
0709 13:09:41.415860 (+   279us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:41.415934 (+    74us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:41.415986 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:41.416065 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:41.416200 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11134,"prepare.queue_time_us":116415,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":67,"raft.queue_time_us":1377,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":8455,"spinlock_wait_cycles":30234752,"thread_start_us":88503,"threads_started":1}
I20260709 13:09:41.495438 30129 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:41.492439 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:41.498556 25284 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:41.499876 25209 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:41.503501 30056 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:41.503937 30056 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:41.504252 30056 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:41.506100 25943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 987) took 1118 ms. Trace:
I20260709 13:09:41.460208 30070 consensus_queue.cc:260] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:41.507771 30070 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:41.395695 29992 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:41.506485 25943 rpcz_store.cc:276] 0709 13:09:40.387548 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:40.388734 (+  1186us) service_pool.cc:224] Handling call
0709 13:09:40.481875 (+ 93141us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:40.502085 (+ 20210us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:41.505903 (+1003818us) spinlock_profiling.cc:243] Waited 92.8 ms on lock 0x7b54001b2bd0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:41.506042 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":213460352}
I20260709 13:09:41.518472 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:41.522989 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:41.530254 25391 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 14: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:41.533528 25238 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:41.418558 25395 rpcz_store.cc:276] 0709 13:09:40.023540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:40.035787 (+ 12247us) service_pool.cc:224] Handling call
0709 13:09:40.046230 (+ 10443us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:40.052295 (+  6065us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:09:40.229866 (+177571us) log.cc:844] Serialized 14 byte log entry
0709 13:09:41.409325 (+1179459us) log.cc:844] Serialized 14 byte log entry
0709 13:09:41.409541 (+   216us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:41.409598 (+    57us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:41.409653 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:41.409743 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:41.413851 (+  4108us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":947270,"prepare.queue_time_us":971300,"prepare.run_cpu_time_us":22,"prepare.run_wall_time_us":23,"raft.queue_time_us":49310,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":19472,"spinlock_wait_cycles":33541248,"thread_start_us":358887,"threads_started":2,"wal-append.queue_time_us":263410}
I20260709 13:09:41.538195 25944 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
W20260709 13:09:41.539708 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 100955us
I20260709 13:09:41.416440 29988 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:41.522433 30025 consensus_meta.cc:287] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.939s	user 0.003s	sys 0.011s
I20260709 13:09:41.548808 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:41.553438 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 144774us
I20260709 13:09:41.537876 29827 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:41.528697 30057 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:41.582765 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:41.522509 29992 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:41.555826 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:41.584187 25939 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 14: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:41.425540 29974 raft_consensus.cc:697] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:41.569581 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:41.552279 25909 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:41.554708 30061 consensus_queue.cc:1059] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.010s
I20260709 13:09:41.571784 25314 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:41.585440 25321 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:09:41.587143 25318 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 13:09:41.590766 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:41.591744 25943 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: [9.10-9.10]
I20260709 13:09:41.593055 29508 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:41.593674 29508 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.595907 29508 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:41.597337 30091 log.cc:927] Time spent T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 1.339s	user 0.003s	sys 0.000s
W20260709 13:09:41.597635 30147 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 13:09:41.597651 25944 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 9. (term mismatch)
I20260709 13:09:41.598281 25944 pending_rounds.cc:85] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 8
I20260709 13:09:41.598573 25944 pending_rounds.cc:107] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260709 13:09:41.598796 25944 raft_consensus.cc:2889] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:41.601320 30104 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:41.602519 25358 leader_election.cc:336] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.818s (SENT)
I20260709 13:09:41.602607 25287 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:41.603138 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:41.603648 25314 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:09:41.603737 30097 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:41.604163 25287 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:41.604350 30097 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:41.604779 30097 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:41.605857 30072 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:41.587914 30025 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.607426 25941 raft_consensus.cc:1240] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:41.607793 30057 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:41.608287 29508 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:09:41.604308 25358 leader_election.cc:336] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.238s (SENT)
I20260709 13:09:41.610144 30057 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:41.609826 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:41.611733 25358 leader_election.cc:336] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.279s (SENT)
I20260709 13:09:41.610280 30025 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:41.612219 25358 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:41.613066 29961 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:41.613724 30061 raft_consensus.cc:3055] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:41.613858 29961 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:41.614250 30061 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:09:41.615041 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 62837.2us
W20260709 13:09:41.618073 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 245395us
I20260709 13:09:41.622424 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:41.554124 30003 raft_consensus.cc:3055] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:41.616974 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:41.631811 30139 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:41.613766 29992 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:41.632225 30061 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:41.633612 30121 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.725s	user 0.003s	sys 0.000s
I20260709 13:09:41.607333 30137 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:41.636963 25942 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
W20260709 13:09:41.617117 30086 consensus_meta.cc:287] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.128s	user 0.009s	sys 0.004s
I20260709 13:09:41.548254 29964 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:41.652447 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:41.652614 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:41.646411 25943 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:09:41.647450 25287 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:41.647440 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:41.648259 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:41.647440 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:09:41.636059 30075 consensus_meta.cc:287] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.155s	user 0.002s	sys 0.010s
I20260709 13:09:41.656803 25936 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:41.650307 25911 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:41.656052 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:41.647642 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:41.650295 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:41.664053 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:41.636431 30003 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:41.666844 25319 raft_consensus.cc:2410] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:09:41.668917 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:41.677922 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:41.680971 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:41.650293 30139 raft_consensus.cc:697] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:41.707307 25911 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:41.707480 30139 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.699651 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:41.709239 29928 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:41.709582 25390 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:41.700740 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:41.710769 25389 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
W20260709 13:09:41.711862 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 67111.6us
I20260709 13:09:41.704563 25321 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:41.675429 30072 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 13:09:41.715982 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 96181us
I20260709 13:09:41.717559 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:41.723637 25287 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:41.657272 29508 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:41.649183 29988 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.033s
I20260709 13:09:41.660485 30154 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:41.789249 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:41.790217 25238 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:41.659476 30058 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:41.765292 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:41.670358 29961 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:41.798591 25245 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:41.799120 25386 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260709 13:09:41.797271 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 51665.9us
I20260709 13:09:41.658857 29964 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.802285 25209 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:41.803457 30130 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 13:09:41.803737 25939 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:09:41.804170 29961 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:41.805398 30090 consensus_meta.cc:287] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.169s	user 0.009s	sys 0.003s
I20260709 13:09:41.805306 30003 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:41.806015 29900 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:41.805354 30148 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.222s	user 0.001s	sys 0.000s
I20260709 13:09:41.687379 29994 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.125s
I20260709 13:09:41.810487 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2410) took 1062 ms. Trace:
I20260709 13:09:41.767356 30061 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:41.763688 30086 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 13:09:41.832830 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 115808us
I20260709 13:09:41.807910 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:41.861261 25386 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:41.852556 25389 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:09:41.853469 30102 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:41.813400 29827 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:41.801878 30058 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:41.675637 30038 consensus_queue.cc:1059] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.153s
I20260709 13:09:41.850090 25317 rpcz_store.cc:276] 0709 13:09:40.747681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:41.285610 (+537929us) service_pool.cc:224] Handling call
0709 13:09:41.289741 (+  4131us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:41.439483 (+149742us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:41.439527 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:41.441157 (+  1630us) log.cc:844] Serialized 26 byte log entry
0709 13:09:41.648534 (+207377us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:09:41.648578 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:41.648646 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:41.810248 (+161602us) raft_consensus.cc:1685] finished
0709 13:09:41.810275 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:41.810419 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":921,"raft.queue_time_us":103932,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"thread_start_us":346247,"threads_started":2,"wal-append.queue_time_us":367212}
I20260709 13:09:41.857689 25387 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
W20260709 13:09:41.861919 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:41.873379 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:41.871362 25238 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:41.790877 29508 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.862929 30102 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:41.890964 30102 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:09:41.904369 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:41.911762 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 925) took 1826 ms. Trace:
I20260709 13:09:41.912230 25938 rpcz_store.cc:276] 0709 13:09:40.085436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:40.085579 (+   143us) service_pool.cc:224] Handling call
0709 13:09:40.112942 (+ 27363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:41.156392 (+1043450us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:41.905314 (+748922us) spinlock_profiling.cc:243] Waited 744 ms on lock 0x7b5400703854. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:41.911688 (+  6374us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1711331072}
W20260709 13:09:41.910422 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 51495.9us
I20260709 13:09:41.797523 29974 consensus_queue.cc:237] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:41.871699 30058 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:41.863775 30151 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:41.873829 30154 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:09:41.919008 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 195515us
I20260709 13:09:41.921257 25287 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:41.922358 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:41.933439 25386 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:41.847154 30076 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.068s
W20260709 13:09:41.924628 30158 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
I20260709 13:09:41.925107 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 881) took 1007 ms. Trace:
I20260709 13:09:41.845806 30090 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.942922 25941 rpcz_store.cc:276] 0709 13:09:40.917819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:40.917957 (+   138us) service_pool.cc:224] Handling call
0709 13:09:40.918255 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:41.607878 (+689623us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:41.924800 (+316922us) spinlock_profiling.cc:243] Waited 689 ms on lock 0x7b54004ad8d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:41.925037 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1584557952}
I20260709 13:09:41.875218 30062 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.046s
I20260709 13:09:41.946581 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:41.947314 25389 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:41.875617 29827 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:41.978022 30163 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 13:09:41.977615 30176 consensus_queue.cc:1059] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:41.977854 30156 consensus_queue.cc:1059] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.114s
I20260709 13:09:41.951285 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:41.978564 30144 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.357s	user 0.001s	sys 0.000s
I20260709 13:09:41.994448 30166 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:41.995114 30166 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:41.996594 30073 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:41.997344 30073 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:41.997764 30073 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:41.997887 30105 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.155s
W20260709 13:09:41.998484 30089 consensus_meta.cc:287] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.512s	user 0.008s	sys 0.003s
I20260709 13:09:41.999112 30176 raft_consensus.cc:3055] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:41.947777 29992 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:42.013418 25394 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:09:41.949388 30175 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:42.031280 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:41.934170 30151 raft_consensus.cc:697] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:42.042651 25388 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
W20260709 13:09:42.051491 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 130337us
I20260709 13:09:42.052366 25287 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:41.999521 30176 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:42.076864 25389 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:09:42.095912 25386 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:09:42.103389 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2409) took 1400 ms. Trace:
I20260709 13:09:42.104018 25312 rpcz_store.cc:276] 0709 13:09:40.703086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:41.286075 (+582989us) service_pool.cc:224] Handling call
0709 13:09:41.290652 (+  4577us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:41.290839 (+   187us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:09:41.291083 (+   244us) log.cc:844] Serialized 14 byte log entry
0709 13:09:41.927195 (+636112us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b1800271660. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f297685223a 00007f2976cbca2a
0709 13:09:42.103031 (+175836us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:42.103086 (+    55us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:42.103140 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:42.103209 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:42.103328 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3079,"prepare.queue_time_us":246,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":2234,"spinlock_wait_cycles":4611328,"thread_start_us":172475,"threads_started":1}
I20260709 13:09:42.107865 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:41.990078 30056 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:42.119915 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 67623.3us
I20260709 13:09:42.093705 30154 raft_consensus.cc:3055] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:09:42.121083 25287 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:42.035614 29854 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260709 13:09:42.067260 30057 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:42.096387 30047 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:42.035888 30064 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.075s
W20260709 13:09:42.138674 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 187413us
I20260709 13:09:42.107081 30156 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:09:42.101528 30166 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:42.142963 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:42.158325 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:42.126204 30154 raft_consensus.cc:740] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:42.186802 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:42.198539 30056 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:42.213320 25392 raft_consensus.cc:2410] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:09:42.133966 29961 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:42.144802 30176 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.143787 30098 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:42.205622 29865 consensus_meta.cc:287] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.925s	user 0.007s	sys 0.007s
W20260709 13:09:42.185792 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:42.185493 30170 log.cc:927] Time spent T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.205s	user 0.000s	sys 0.001s
W20260709 13:09:42.187088 30160 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20260709 13:09:42.185602 30104 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:42.186118 30146 log.cc:927] Time spent T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.340s	user 0.001s	sys 0.000s
I20260709 13:09:42.136888 29964 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:42.197424 30161 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.289s	user 0.000s	sys 0.001s
I20260709 13:09:42.217299 25358 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:42.197955 30166 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:42.285473 30166 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:42.144973 30047 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.288056 30166 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:42.290966 30081 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:42.230543 25394 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: [9.10-9.10]
W20260709 13:09:42.240322 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:42.244423 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:42.293402 25386 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:42.277021 30098 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:42.092013 30175 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.288995 30047 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:42.295375 30098 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:42.296041 30175 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:42.185995 30153 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:09:42.302462 30097 consensus_meta.cc:287] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.697s	user 0.009s	sys 0.001s
I20260709 13:09:42.302927 30097 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.303747 30056 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:42.303963 25237 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:42.304277 30056 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.305516 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:42.305799 29508 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:42.306658 29827 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:42.307971 30157 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.279s	user 0.000s	sys 0.001s
W20260709 13:09:42.309414 30165 log.cc:927] Time spent T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.366s	user 0.001s	sys 0.000s
I20260709 13:09:42.310341 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:42.310771 30090 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:42.311190 25238 raft_consensus.cc:2393] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 15: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:42.311460 30181 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:42.313301 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:42.271919 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 150945us
I20260709 13:09:42.186328 30167 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:42.206025 29900 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:42.224565 30073 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:42.321486 30169 log.cc:927] Time spent T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.369s	user 0.000s	sys 0.001s
I20260709 13:09:42.321626 29900 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.244884 29865 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.323889 29865 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:42.325881 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 108673us
I20260709 13:09:42.336943 25358 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:42.349556 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:42.350556 25941 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:42.359094 25287 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:42.326184 30097 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:42.366860 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:42.324484 29900 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:42.369341 25944 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:42.215158 30102 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:42.368017 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:42.373028 30177 log.cc:927] Time spent T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.215s	user 0.001s	sys 0.000s
I20260709 13:09:42.374184 30087 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:42.375607 25238 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:42.375685 30168 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:42.376420 30168 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.378822 30168 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:42.379331 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:42.381098 29865 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:42.381332 30193 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:42.382489 25237 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:42.385316 30183 consensus_queue.cc:1059] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260709 13:09:42.388556 30183 raft_consensus.cc:3055] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:42.388926 30183 raft_consensus.cc:740] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:42.340281 30151 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.389848 30006 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:42.324549 30104 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:42.356534 29508 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:42.335002 30189 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:42.394182 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:42.407111 25945 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 988) took 2019 ms. Trace:
I20260709 13:09:42.344465 30090 consensus_queue.cc:1059] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.059s
W20260709 13:09:42.410043 25358 connection.cc:256] RPC call timeout handler was delayed by -1.7345068800000263s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16010000000000002s for a response
I20260709 13:09:42.420830 25313 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:42.422091 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:42.422310 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:42.424273 25238 raft_consensus.cc:2393] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:09:42.424623 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:42.426429 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:42.427055 25240 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:42.407581 25945 rpcz_store.cc:276] 0709 13:09:40.387810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:40.388322 (+   512us) service_pool.cc:224] Handling call
0709 13:09:40.388661 (+   339us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:40.389052 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:40.389078 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:41.450667 (+1061589us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b2c000476c8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f29755aba11 00007f29755bb372 00007f2976ceda01 00007f2976cb243a 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2
0709 13:09:41.451672 (+  1005us) log.cc:844] Serialized 26 byte log entry
0709 13:09:41.691653 (+239981us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:41.691716 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:41.691821 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:42.403007 (+711186us) raft_consensus.cc:1685] finished
0709 13:09:42.403039 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:42.407024 (+  3985us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":291,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":126,"spinlock_wait_cycles":34405120,"thread_start_us":220195,"threads_started":1,"wal-append.queue_time_us":547198}
I20260709 13:09:42.321019 30167 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:42.468163 25358 leader_election.cc:336] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.099s (SENT)
I20260709 13:09:42.470681 25287 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:42.471879 25241 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:42.406805 30189 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:42.362648 29964 consensus_queue.cc:1059] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 13:09:42.476734 25358 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:42.481472 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:42.482988 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:42.476989 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 51146.2us
I20260709 13:09:42.477820 25940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 993) took 1830 ms. Trace:
I20260709 13:09:42.485780 25212 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:42.402712 30079 consensus_queue.cc:1059] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.073s
I20260709 13:09:42.414129 29508 raft_consensus.cc:697] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:42.368451 30154 consensus_queue.cc:260] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:42.474787 30182 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:42.485508 25940 rpcz_store.cc:276] 0709 13:09:40.647238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:40.664657 (+ 17419us) service_pool.cc:224] Handling call
0709 13:09:40.664984 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:40.666764 (+  1780us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:40.666791 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:40.702168 (+ 35377us) log.cc:844] Serialized 26 byte log entry
0709 13:09:41.294507 (+592339us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:41.294556 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:41.294623 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:42.477514 (+1182891us) raft_consensus.cc:1685] finished
0709 13:09:42.477550 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:42.477743 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":532835,"threads_started":1,"wal-append.queue_time_us":696986}
I20260709 13:09:42.502918 25212 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
W20260709 13:09:42.503669 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 158152us
I20260709 13:09:42.514541 25238 raft_consensus.cc:1275] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:09:42.485004 30189 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:42.519766 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.524334 25312 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:42.523355 25314 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 13:09:42.522261 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.522854 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.502519 30171 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:42.529397 25317 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:42.505292 30084 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:42.533981 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.535122 25312 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:42.537786 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:42.538702 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.539651 25318 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:42.540083 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.541080 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.522104 30187 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:42.526366 30179 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:42.547952 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.533993 29854 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:09:42.548179 30195 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 13:09:42.556891 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.557626 25321 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:42.558039 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.558967 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:42.555815 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 53035.1us
W20260709 13:09:42.561713 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 144636us
W20260709 13:09:42.562171 25358 connection.cc:256] RPC call timeout handler was delayed by -1.5650057139999944s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29100000000000004s for a response
I20260709 13:09:42.567581 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2398) took 2087 ms. Trace:
I20260709 13:09:42.568015 25320 rpcz_store.cc:276] 0709 13:09:40.480305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:40.480491 (+   186us) service_pool.cc:224] Handling call
0709 13:09:40.480945 (+   454us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:40.488137 (+  7192us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:40.488178 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:42.370026 (+1881848us) spinlock_profiling.cc:243] Waited 1.29 ms on lock 0x7b2c0009d3b8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f29755aba11 00007f29755bb372 00007f2976ceda01 00007f2976cb243a 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2
0709 13:09:42.372214 (+  2188us) log.cc:844] Serialized 26 byte log entry
0709 13:09:42.510489 (+138275us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:42.510525 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:42.510592 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:42.567261 (+ 56669us) raft_consensus.cc:1685] finished
0709 13:09:42.567281 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:42.567505 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2973568,"thread_start_us":137772,"threads_started":1,"wal-append.queue_time_us":166292}
I20260709 13:09:42.509989 30182 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:42.553333 29974 consensus_queue.cc:1048] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260709 13:09:42.553902 29854 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:42.496304 30183 consensus_queue.cc:260] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:42.576249 25358 leader_election.cc:336] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.134s (SENT)
I20260709 13:09:42.577850 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.578635 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.580181 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:42.578750 29854 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:42.582933 30102 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:42.583758 30102 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:09:42.585207 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.585848 25312 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:42.586750 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.592172 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:42.593458 25240 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
W20260709 13:09:42.595409 30197 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 13:09:42.595659 30183 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:42.597084 25359 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:42.597509 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.598256 25320 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
W20260709 13:09:42.601826 30194 log.cc:927] Time spent T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
W20260709 13:09:42.604318 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:42.554405 30187 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.611465 29862 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:42.627300 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.628618 25314 raft_consensus.cc:2410] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 13:09:42.553190 29508 consensus_queue.cc:237] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:42.619177 30070 consensus_meta.cc:287] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.111s	user 0.010s	sys 0.005s
W20260709 13:09:42.636364 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:42.641585 25315 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:42.640213 30167 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:42.642002 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 79483.8us
I20260709 13:09:42.640791 30104 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:42.642736 25358 leader_election.cc:336] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.225s (SENT)
I20260709 13:09:42.643183 25358 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:42.643958 30179 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:09:42.644214 25358 connection.cc:256] RPC call timeout handler was delayed by -1.4003057140000692s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3093s for a response
W20260709 13:09:42.644590 25358 connection.cc:256] RPC call timeout handler was delayed by -1.1747068800000307s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2223s for a response
I20260709 13:09:42.645076 30179 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:42.645661 30179 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.648726 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:42.649394 25238 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
W20260709 13:09:42.645023 25358 connection.cc:256] RPC call timeout handler was delayed by -1.1379353290000154s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2978s for a response
I20260709 13:09:42.617151 29992 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:42.654353 25359 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:42.633900 30205 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:42.624218 29964 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:09:42.575933 30198 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.060s
W20260709 13:09:42.657440 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 83833.4us
I20260709 13:09:42.658735 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:42.657253 29961 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:42.673007 25212 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:42.670050 29961 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:42.660203 29992 raft_consensus.cc:697] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:42.684258 25212 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
W20260709 13:09:42.687017 25358 leader_election.cc:336] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.526s (SENT)
I20260709 13:09:42.690552 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.693066 25212 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:42.693950 25320 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 8: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:42.699795 25314 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:42.690115 25358 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:42.680064 30064 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:09:42.700790 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:42.688962 29961 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:42.673949 30141 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.017s
I20260709 13:09:42.703544 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:42.704068 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:42.704397 30064 raft_consensus.cc:697] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:09:42.704563 25358 leader_election.cc:336] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.550s (SENT)
I20260709 13:09:42.706106 25320 raft_consensus.cc:2368] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11.
I20260709 13:09:42.677305 30191 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.025s
I20260709 13:09:42.706643 25212 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:42.694880 30104 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:09:42.707800 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:42.712935 25212 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:42.717659 30058 consensus_meta.cc:287] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.783s	user 0.013s	sys 0.000s
I20260709 13:09:42.717816 30182 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:42.720428 29992 consensus_queue.cc:237] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:42.708428 30198 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:42.708499 30174 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:42.762885 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.671288 30205 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.735046 30064 consensus_queue.cc:237] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:42.769392 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:42.741309 30063 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:42.742358 29854 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:42.744599 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2419) took 1102 ms. Trace:
I20260709 13:09:42.751303 30176 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:42.777776 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:42.742313 30065 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:42.738775 30184 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.403s	user 0.003s	sys 0.000s
I20260709 13:09:42.772073 25318 rpcz_store.cc:276] 0709 13:09:41.642342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:41.642607 (+   265us) service_pool.cc:224] Handling call
0709 13:09:41.643048 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:42.545401 (+902353us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:42.744305 (+198904us) spinlock_profiling.cc:243] Waited 894 ms on lock 0x7b5400115654. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:42.744527 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2055195008}
W20260709 13:09:42.767046 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 62498.6us
I20260709 13:09:42.779088 25320 raft_consensus.cc:2410] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
W20260709 13:09:42.786742 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 73874.7us
I20260709 13:09:42.761641 30175 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:42.771296 30104 raft_consensus.cc:697] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:42.792901 30204 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:42.794422 30141 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:42.810464 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:42.798185 25358 leader_election.cc:336] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.698s (SENT)
I20260709 13:09:42.798542 29961 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.799842 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.812945 25318 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:42.799929 30189 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.813920 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.807688 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.816211 25314 raft_consensus.cc:2410] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 13:09:42.802225 30205 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:42.809321 30061 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:42.809491 29854 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:42.766566 30058 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.811282 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:42.810802 30204 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:42.822199 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.824363 25312 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:42.823678 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.826311 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:42.826841 25318 raft_consensus.cc:2393] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 15: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:09:42.829206 25358 leader_election.cc:336] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.852s (SENT)
I20260709 13:09:42.829595 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:42.786787 30065 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.835301 25393 raft_consensus.cc:2410] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 13:09:42.835723 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:42.835907 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:42.838946 30189 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:42.839726 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:42.842005 25386 raft_consensus.cc:1275] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 5. (term mismatch)
I20260709 13:09:42.842336 25386 pending_rounds.cc:85] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 4
I20260709 13:09:42.842579 25386 pending_rounds.cc:107] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 13:09:42.842816 25386 raft_consensus.cc:2889] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:42.826414 30204 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:42.844621 25395 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: [12.7-14.8]
I20260709 13:09:42.823876 30061 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.826548 30175 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:42.847221 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:42.852787 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:42.853781 25393 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:42.855649 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.852964 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:42.856827 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:42.857537 25389 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
W20260709 13:09:42.860158 25358 leader_election.cc:336] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.910s (SENT)
I20260709 13:09:42.869834 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:42.871676 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:42.872474 25388 raft_consensus.cc:2393] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 15: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:42.877990 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:42.878827 25388 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:42.882627 25393 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:42.883813 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:42.884516 25388 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:42.886085 29961 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:42.819389 30141 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.841104 30156 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:42.889305 25314 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:42.897260 30198 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:42.897404 30215 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:42.850364 30183 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:42.892627 25388 raft_consensus.cc:2410] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 15 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 13:09:42.841758 29508 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:09:42.893438 25358 leader_election.cc:336] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.978s (SENT)
I20260709 13:09:42.941670 25358 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:42.945072 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:42.890232 30141 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:42.948714 30138 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 1.803s	user 0.000s	sys 0.009s
I20260709 13:09:42.949374 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:42.931229 30216 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:42.953593 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:42.954523 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:42.956012 25246 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:42.956193 30061 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:42.956931 29508 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:09:42.963397 30204 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:42.850503 30104 consensus_queue.cc:237] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:42.934113 30156 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.967048 30156 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:42.970116 25314 raft_consensus.cc:2410] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 13:09:42.971246 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2405) took 2365 ms. Trace:
I20260709 13:09:42.971630 25316 rpcz_store.cc:276] 0709 13:09:40.605674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:40.740104 (+134430us) service_pool.cc:224] Handling call
0709 13:09:40.740478 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:40.740914 (+   436us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:40.740953 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:40.741838 (+   885us) log.cc:844] Serialized 26 byte log entry
0709 13:09:40.799145 (+ 57307us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:40.914358 (+115213us) log.cc:844] Serialized 14 byte log entry
0709 13:09:40.914661 (+   303us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:40.914761 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:41.184976 (+270215us) spinlock_profiling.cc:243] Waited 2.55 ms on lock 0x7b18003b9200. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc5522 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:09:42.948030 (+1763054us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:09:42.971039 (+ 23009us) raft_consensus.cc:1685] finished
0709 13:09:42.971062 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:42.971189 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":145336,"prepare.queue_time_us":116544,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":8414,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":74357,"spinlock_wait_cycles":36508288,"thread_start_us":56305,"threads_started":1,"wal-append.queue_time_us":310370}
I20260709 13:09:42.927000 30215 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:42.978243 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 84846.7us
I20260709 13:09:42.941696 30183 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:42.984043 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:42.971146 30198 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:42.972859 30174 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:42.994642 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:42.995234 25392 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 12
I20260709 13:09:42.995817 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:42.996320 25387 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:42.997491 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:42.997990 25393 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:42.999712 25394 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 13:09:43.007965 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.013619 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:43.012194 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:42.971306 30216 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.016959 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:43.018949 25287 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:43.017671 25387 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:43.036094 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.038479 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:43.038889 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.039846 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.041177 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:43.041716 25387 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:43.003533 30189 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:43.045408 30189 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.046238 25287 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:43.047034 30081 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:43.047384 30081 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:43.047709 30081 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:43.048240 25287 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:43.049069 30167 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:43.055969 30081 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:43.057541 30189 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:43.061054 30156 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:43.074447 25246 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:43.075124 25387 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:43.075986 25359 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:43.062829 25316 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:42.999141 30198 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.066951 25287 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:43.066385 25392 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:43.021616 30176 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:43.083670 25287 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:43.085111 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.086019 25316 raft_consensus.cc:2376] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12. Re-sending same reply.
I20260709 13:09:43.088049 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:43.095620 25212 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:43.102960 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:43.105300 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.105908 25316 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:43.081487 30174 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:43.078073 30204 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:43.113744 30216 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:43.116096 30183 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:43.119277 30220 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 13:09:43.124146 30215 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:43.125655 25358 leader_election.cc:336] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 3.093s (SENT)
I20260709 13:09:43.126106 25358 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:43.127305 25358 leader_election.cc:336] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 3.221s (SENT)
I20260709 13:09:43.127687 30198 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:43.127667 25358 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:43.129400 30175 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:43.130172 30175 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 13:09:43.131742 25358 leader_election.cc:336] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 2.125s (SENT)
I20260709 13:09:43.097903 30171 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:43.132161 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:43.132912 30175 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:43.100966 30135 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:43.135022 30135 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:43.135469 30135 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:43.136507 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.101758 30070 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:43.137374 30070 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:43.137511 30081 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:43.137909 30070 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:43.108572 30167 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:43.141965 30174 raft_consensus.cc:697] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:43.143146 30174 consensus_queue.cc:237] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.106389 30141 consensus_queue.cc:1059] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 13:09:43.146011 30141 raft_consensus.cc:3055] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:09:43.146450 30141 raft_consensus.cc:740] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:43.147434 30141 consensus_queue.cc:260] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.148757 30141 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:43.107577 30064 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:43.160074 30064 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:43.160493 30064 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:43.149976 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:43.137367 25316 raft_consensus.cc:2410] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:09:43.165917 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.166457 25316 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:09:43.166883 25245 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 834) took 1531 ms (client timeout 1986 ms). Trace:
W20260709 13:09:43.167317 25245 rpcz_store.cc:269] 0709 13:09:41.635303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:41.635435 (+   132us) service_pool.cc:224] Handling call
0709 13:09:43.166726 (+1531291us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b54002b2d54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:43.166833 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":320300928}
W20260709 13:09:43.169659 30225 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 13:09:43.169544 30006 raft_consensus.cc:3055] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:43.170254 30006 raft_consensus.cc:740] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:43.171136 30006 consensus_queue.cc:260] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.172195 30006 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:43.174010 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.174638 25320 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:43.175458 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.176157 25320 raft_consensus.cc:2410] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 13:09:43.181814 30101 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:43.184531 25313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2396) took 2722 ms (client timeout 1874 ms). Trace:
W20260709 13:09:43.184970 25313 rpcz_store.cc:269] 0709 13:09:40.462119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:40.462317 (+   198us) service_pool.cc:224] Handling call
0709 13:09:43.184296 (+2721979us) spinlock_profiling.cc:243] Waited 1.88 s on lock 0x7b5400045454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:43.184407 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4330449280}
I20260709 13:09:43.186182 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.186889 25313 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 8: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:43.187638 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.188548 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.190414 25393 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:43.196156 25390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2518) took 1408 ms. Trace:
I20260709 13:09:43.196607 25390 rpcz_store.cc:276] 0709 13:09:41.787546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:41.790593 (+  3047us) service_pool.cc:224] Handling call
0709 13:09:42.468395 (+677802us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:42.468731 (+   336us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:42.468769 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:42.468828 (+    59us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:42.468881 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:42.468958 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:43.195964 (+727006us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b54003e4750. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:43.196091 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1394957568}
I20260709 13:09:43.200544 30055 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:43.206075 30202 log.cc:927] Time spent T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.588s	user 0.000s	sys 0.002s
I20260709 13:09:43.207305 29964 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:43.207955 29964 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:43.208321 30187 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:43.213810 30058 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:43.214478 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:43.215965 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.160511 30179 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:43.217000 25314 raft_consensus.cc:2410] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:09:43.176870 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.232390 25313 raft_consensus.cc:2410] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:09:43.235793 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.236238 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.236547 25313 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:43.237533 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.237609 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.238428 25313 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:43.240113 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.240482 29964 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:43.241214 25320 raft_consensus.cc:2410] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 13:09:43.257637 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.257710 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.238811 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.258780 25320 raft_consensus.cc:2410] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 13:09:43.259785 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.260380 25320 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:43.261384 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.262429 30003 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:43.266729 30056 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:43.258728 25314 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:43.271099 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.271968 25314 raft_consensus.cc:2410] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
W20260709 13:09:43.272281 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:43.274621 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:43.272780 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.276674 25314 raft_consensus.cc:2410] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 13:09:43.262485 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.279066 25313 raft_consensus.cc:2410] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 13:09:43.279945 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.251199 25316 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:43.291424 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:43.299697 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.299908 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:43.304483 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:43.305751 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:43.310429 25937 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:43.310308 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.315442 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:43.316427 25944 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 11: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:43.314467 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:43.319314 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:43.320076 25942 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:43.321346 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.323244 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:43.318785 25938 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:43.324045 25939 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:43.325575 25313 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:43.292919 30178 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:43.326411 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.328663 25313 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:43.327009 25358 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:43.331528 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:43.332207 25938 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:43.336376 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:43.342442 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.343948 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:43.344709 25938 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:43.348181 25245 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:43.349490 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:43.351534 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:43.357486 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:43.371114 25241 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:43.358151 30179 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:43.361516 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:43.372078 30179 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:43.372586 30179 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:43.372746 25243 raft_consensus.cc:2410] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 13:09:43.324709 30070 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:43.375512 30070 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:43.378271 25313 raft_consensus.cc:2368] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14.
I20260709 13:09:43.365474 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:43.368786 25937 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:43.353639 25909 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:43.380319 25938 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:43.386651 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:43.388244 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:43.390632 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:43.391104 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:43.345830 29964 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:43.394418 25209 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:43.394444 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:43.396152 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:43.327384 29894 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.062s
I20260709 13:09:43.397491 25246 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:09:43.398175 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:43.386018 30055 raft_consensus.cc:2764] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 13:09:43.399520 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:43.412279 25944 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 13:09:43.390391 30174 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:43.412352 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:43.389421 30168 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:43.335747 30178 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:43.418169 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:43.419632 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:43.417474 25209 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:43.414366 30173 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:43.423775 30173 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:43.424250 30173 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:43.424492 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.402514 30058 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:43.418610 25212 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:43.421765 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:43.432013 25312 raft_consensus.cc:2393] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:43.434285 25212 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:43.416898 30187 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:43.441269 30187 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:43.441807 30187 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:43.432451 30068 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:43.445693 30068 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.446132 30135 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:43.447849 30068 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:43.448072 30135 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:43.448722 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.449406 25312 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:43.449869 30178 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:43.450268 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.418578 30064 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:43.451447 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.451967 25312 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:43.452204 30064 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:43.422140 30102 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:43.452781 25358 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:43.435411 30057 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:43.394040 30056 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260709 13:09:43.460325 30137 consensus_queue.cc:1059] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:43.438548 30062 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:43.441318 29988 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:43.461782 30137 raft_consensus.cc:3055] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:43.462280 30137 raft_consensus.cc:740] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:43.462433 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2423) took 1801 ms. Trace:
I20260709 13:09:43.462896 25315 rpcz_store.cc:276] 0709 13:09:41.660848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:41.680622 (+ 19774us) service_pool.cc:224] Handling call
0709 13:09:43.462243 (+1781621us) spinlock_profiling.cc:243] Waited 954 ms on lock 0x7b5400275454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:43.462352 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2193444736}
I20260709 13:09:43.464272 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.459312 25358 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:43.469869 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 193346us
I20260709 13:09:43.458899 25909 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:43.471539 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:43.453092 30102 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:43.480288 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:43.432127 29973 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:43.465013 25315 raft_consensus.cc:2376] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11. Re-sending same reply.
I20260709 13:09:43.485788 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.438776 30061 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:43.481108 25393 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:43.476372 25936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 880) took 2551 ms. Trace:
I20260709 13:09:43.482120 30234 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:43.482591 30102 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:43.480605 29988 raft_consensus.cc:697] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:43.483876 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:43.494310 25394 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:43.486631 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:43.486344 25315 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:43.495970 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.496742 25315 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:43.491420 25319 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2393) took 3066 ms (client timeout 3382 ms). Trace:
I20260709 13:09:43.489847 25936 rpcz_store.cc:276] 0709 13:09:40.914899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:40.915022 (+   123us) service_pool.cc:224] Handling call
0709 13:09:40.986730 (+ 71708us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:41.657556 (+670826us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:43.465545 (+1807989us) spinlock_profiling.cc:243] Waited 670 ms on lock 0x7b5400690354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:43.465788 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1540456704}
I20260709 13:09:43.476788 29961 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:43.474040 30168 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:43.497720 25312 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
W20260709 13:09:43.497588 25319 rpcz_store.cc:269] 0709 13:09:40.424685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:40.424917 (+   232us) service_pool.cc:224] Handling call
0709 13:09:43.491222 (+3066305us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54000636d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:43.491351 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2724947840}
I20260709 13:09:43.474155 30057 raft_consensus.cc:697] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:43.517313 30168 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.518612 30057 consensus_queue.cc:237] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.519683 30168 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:43.520231 30168 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:43.476552 30062 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:43.521795 30062 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:43.529479 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.530301 25314 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:43.531255 25317 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:43.532222 29988 consensus_queue.cc:237] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:43.533452 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.534785 25390 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:43.535908 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:43.536543 25393 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:09:43.512745 30229 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 13:09:43.491989 29973 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.539031 30179 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.539728 29973 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:43.540351 29973 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:43.533447 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:43.540980 29973 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.004s
I20260709 13:09:43.546208 25284 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:43.547739 30178 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:43.552881 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:43.554137 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:43.557832 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:43.560976 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:43.561643 25246 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:43.558473 30068 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.004s
I20260709 13:09:43.562872 25243 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:09:43.493712 30234 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:43.559561 25284 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:43.561672 25238 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:43.564759 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:43.564973 25240 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:43.565514 25242 raft_consensus.cc:2393] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 14: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:43.566666 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:43.568192 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:43.569589 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:43.570631 25393 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:09:43.573110 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2350) took 1058 ms. Trace:
I20260709 13:09:43.573531 25317 rpcz_store.cc:276] 0709 13:09:42.514817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:42.531890 (+ 17073us) service_pool.cc:224] Handling call
0709 13:09:43.572959 (+1041069us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b540005e954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:43.573060 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2277694848}
I20260709 13:09:43.574173 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.575024 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:43.496204 30061 raft_consensus.cc:697] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:43.520704 30168 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.582372 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.582970 25287 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:43.593102 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.597846 25319 raft_consensus.cc:2410] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:09:43.590539 25394 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:43.591341 25315 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:43.592998 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:43.593351 25388 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:43.599635 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.554955 30137 consensus_queue.cc:260] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.600337 25315 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
W20260709 13:09:43.600597 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 129123us
I20260709 13:09:43.601025 30137 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:43.601528 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.602165 25315 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:43.602733 25908 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:43.604732 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:43.595201 30065 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:43.613379 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:43.616030 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:43.616313 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 182107us
I20260709 13:09:43.618228 25241 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
W20260709 13:09:43.608232 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 132334us
I20260709 13:09:43.609616 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:43.611248 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 51783.2us
I20260709 13:09:43.621029 25394 raft_consensus.cc:2393] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:43.621083 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.622197 25284 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:43.622486 25312 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:43.611233 25315 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:43.613855 25390 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:43.614027 25243 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:09:43.630194 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:43.630926 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.631397 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:43.631769 25319 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:43.631958 25942 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:43.633314 30231 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:43.633884 30231 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.635938 30231 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:43.638793 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:43.639919 25318 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:43.638577 25315 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
W20260709 13:09:43.641876 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 59001.7us
I20260709 13:09:43.642475 25284 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:43.643738 25284 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:43.638863 30240 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:43.595782 30061 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.644982 30183 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:43.646107 30171 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:09:43.646843 30171 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:43.647580 30171 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:43.647832 25942 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:43.648943 25284 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:43.650075 30076 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:43.644837 30240 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.572295 30234 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:43.582877 29961 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:43.653251 30240 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:43.653447 29961 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.654721 30076 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:09:43.655658 29961 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:43.657075 29964 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:43.658716 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:43.659500 30065 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:43.661927 30173 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:43.624357 30003 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:09:43.582069 30179 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:43.665349 25319 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:09:43.667146 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:43.677491 25317 raft_consensus.cc:2410] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:09:43.664427 25209 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:43.669118 25315 raft_consensus.cc:606] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:43.679183 25315 raft_consensus.cc:614] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:09:43.691030 25318 raft_consensus.cc:2410] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 13:09:43.692224 25209 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:43.627063 30086 raft_consensus.cc:3055] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:43.701182 25209 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:43.596024 30089 consensus_queue.cc:1059] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.094s
I20260709 13:09:43.708461 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:43.705597 30205 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:43.711691 30205 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:43.712074 30205 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:43.717339 30129 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:43.718146 30097 raft_consensus.cc:3055] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:09:43.718648 30097 raft_consensus.cc:740] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:43.719219 30168 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:43.704308 30086 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:43.611426 30166 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.083s
I20260709 13:09:43.627967 30006 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.078s
I20260709 13:09:43.752856 25315 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:43.755002 25318 raft_consensus.cc:1275] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 13:09:43.755503 25318 pending_rounds.cc:85] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 5
I20260709 13:09:43.755777 25318 pending_rounds.cc:107] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 13:09:43.756078 25318 raft_consensus.cc:2889] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:43.756892 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2383) took 1028 ms. Trace:
I20260709 13:09:43.757966 25318 rpcz_store.cc:276] 0709 13:09:42.728857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:43.704878 (+976021us) service_pool.cc:224] Handling call
0709 13:09:43.754578 (+ 49700us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:43.756596 (+  2018us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:43.756827 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:43.725026 25312 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:43.725239 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:43.759439 25939 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:43.624472 30178 consensus_queue.cc:1059] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.089s
I20260709 13:09:43.752573 25317 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:43.700304 30105 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:09:43.907869 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:43.703951 30174 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:09:43.916579 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:43.752041 30104 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:43.746798 30234 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.935698 25388 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 7. Current term is 8. Ops: []
I20260709 13:09:43.914052 30171 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:43.990567 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:43.926100 30178 consensus_queue.cc:1243] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer a2a77ecf2c0941618b5bc79d4c04db61 log is divergent from this leader: its last log entry 15.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:09:43.926342 30105 raft_consensus.cc:697] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:43.928299 30240 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:43.906183 30102 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:43.935055 30183 raft_consensus.cc:697] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:09:44.009272 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 406598us
I20260709 13:09:43.974162 30089 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:43.938673 30187 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:44.018342 25908 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:44.033708 25239 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:43.998629 30171 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:44.019991 30173 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:44.016271 30089 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:44.035017 25908 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:44.052620 25908 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:44.054234 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2512) took 2382 ms. Trace:
I20260709 13:09:44.054716 25391 rpcz_store.cc:276] 0709 13:09:41.667024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:41.667163 (+   139us) service_pool.cc:224] Handling call
0709 13:09:41.667509 (+   346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:41.787618 (+120109us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:09:41.830237 (+ 42619us) log.cc:844] Serialized 14 byte log entry
0709 13:09:42.094327 (+264090us) log.cc:844] Serialized 14 byte log entry
0709 13:09:43.202751 (+1108424us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x7b18003c73c0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc2455 00007f2976caa060 00007f2976cbc716 00007f2976ccd287 00007f2976ccd208
0709 13:09:43.390924 (+188173us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:43.391542 (+   618us) log.cc:844] Serialized 26 byte log entry
0709 13:09:43.592309 (+200767us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:09:43.592368 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:43.592466 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:44.049295 (+456829us) raft_consensus.cc:1685] finished
0709 13:09:44.049326 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:44.049503 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":80395,"prepare.queue_time_us":33367,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":86,"raft.queue_time_us":298977,"raft.run_cpu_time_us":705,"raft.run_wall_time_us":106544,"spinlock_wait_cycles":4621184,"thread_start_us":683946,"threads_started":4,"wal-append.queue_time_us":827938,"wal-append.run_cpu_time_us":2115,"wal-append.run_wall_time_us":1035555}
I20260709 13:09:44.060762 30240 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:44.061353 30240 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:44.066859 25312 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:44.072921 30234 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:44.074641 30187 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:44.078399 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:44.079057 25943 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:44.079475 30089 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:44.050520 30241 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:44.083683 25312 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:09:44.088163 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 778) took 1236 ms. Trace:
I20260709 13:09:44.091406 25312 rpcz_store.cc:276] 0709 13:09:42.851414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.066426 (+1215012us) service_pool.cc:224] Handling call
0709 13:09:44.066747 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:44.087847 (+ 21100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:44.088105 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:44.096202 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 780) took 1244 ms. Trace:
I20260709 13:09:44.096699 25312 rpcz_store.cc:276] 0709 13:09:42.851889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.095407 (+1243518us) service_pool.cc:224] Handling call
0709 13:09:44.095716 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:44.095879 (+   163us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:44.095903 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:44.095941 (+    38us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:44.095976 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:44.096044 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:44.096149 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:44.099594 25943 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:44.053072 30246 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 13:09:44.111092 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 375003us
I20260709 13:09:44.081897 30173 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:44.127281 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 74734.9us
I20260709 13:09:44.128237 25908 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:44.137814 25358 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:44.142760 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 781) took 1290 ms. Trace:
I20260709 13:09:44.142445 25908 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:44.145627 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:44.140015 25911 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:44.147541 25908 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:44.148407 25312 rpcz_store.cc:276] 0709 13:09:42.852154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.097975 (+1245821us) service_pool.cc:224] Handling call
0709 13:09:44.142267 (+ 44292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:44.142432 (+   165us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:44.142455 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:44.142502 (+    47us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:44.142540 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:44.142595 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:44.142706 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:44.045410 30183 consensus_queue.cc:237] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:44.154121 25911 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:44.044816 30105 consensus_queue.cc:237] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:44.158303 25943 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:09:44.163386 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 782) took 1311 ms. Trace:
I20260709 13:09:44.164620 25358 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:44.189565 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:44.192631 29964 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:44.193152 29964 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:44.165251 25312 rpcz_store.cc:276] 0709 13:09:42.857233 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.149788 (+1292555us) service_pool.cc:224] Handling call
0709 13:09:44.162752 (+ 12964us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:44.162962 (+   210us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:44.163000 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:44.163069 (+    69us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:44.163114 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:44.163183 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:44.163317 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:44.168257 30217 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:44.127678 30173 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:44.155023 30055 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:44.176795 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:44.151619 29973 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:44.159705 30137 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:44.155270 30247 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:44.190770 25358 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:44.193575 29964 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:44.210773 25312 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:44.212013 29973 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:44.187646 30102 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:44.216521 30065 consensus_meta.cc:287] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.557s	user 0.006s	sys 0.001s
I20260709 13:09:44.218781 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:44.221413 30171 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:44.266846 30070 raft_consensus.cc:3055] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:44.267463 30070 raft_consensus.cc:740] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:09:44.269214 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:44.222190 29894 consensus_queue.cc:1059] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:44.268563 30070 consensus_queue.cc:260] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:44.225914 30205 consensus_meta.cc:287] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.513s	user 0.012s	sys 0.000s
W20260709 13:09:44.230458 30244 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 13:09:44.231748 30173 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:44.231484 29961 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:09:44.201838 30240 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:44.201318 30217 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:44.208052 30055 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:44.276194 30240 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:44.276561 30055 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:44.189254 30246 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:44.234730 25284 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:44.235706 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:44.238713 25239 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 10. (term mismatch)
I20260709 13:09:44.239998 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 783) took 1382 ms. Trace:
W20260709 13:09:44.252761 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 62079.1us
I20260709 13:09:44.283114 25239 pending_rounds.cc:85] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 9
I20260709 13:09:44.256139 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2364) took 1598 ms. Trace:
I20260709 13:09:44.283625 25358 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:44.283758 25313 rpcz_store.cc:276] 0709 13:09:42.658157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:43.342121 (+683964us) service_pool.cc:224] Handling call
0709 13:09:44.255925 (+913804us) spinlock_profiling.cc:243] Waited 34.8 ms on lock 0x7b5400443254. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:44.256075 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":79939968}
I20260709 13:09:44.263028 25244 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:09:44.283169 25312 rpcz_store.cc:276] 0709 13:09:42.857539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.210135 (+1352596us) service_pool.cc:224] Handling call
0709 13:09:44.210544 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:44.239731 (+ 29187us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:44.239943 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:44.272442 30097 consensus_queue.cc:260] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:44.263574 25941 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:44.234032 30137 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:44.265627 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:44.266463 30102 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:09:44.266693 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 48019.2us
I20260709 13:09:44.271044 30070 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:44.270820 30205 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:44.275959 30217 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:44.276861 30240 raft_consensus.cc:3055] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:09:44.283476 25239 pending_rounds.cc:107] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 9.10
I20260709 13:09:44.286540 25312 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 13:09:44.317809 25239 raft_consensus.cc:2889] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:44.263562 25940 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: [9.9-11.10]
I20260709 13:09:44.289573 25245 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:09:44.264699 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:44.320075 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:44.294901 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:44.321194 25386 raft_consensus.cc:2393] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 14: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:44.321579 25389 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:44.297561 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:44.300572 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 65929.2us
I20260709 13:09:44.323534 25284 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:44.323943 25387 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:09:44.299674 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:44.279117 30065 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:44.308454 25388 raft_consensus.cc:1240] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:44.310469 25390 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:09:44.304628 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:44.277778 30006 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:44.318653 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:44.330071 25239 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 13:09:44.290230 30234 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:44.335306 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:44.300772 30137 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:44.298051 30038 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:44.337292 25238 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:44.340664 25390 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:09:44.299032 29961 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:44.411809 25389 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:44.344898 25388 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:09:44.326857 30240 raft_consensus.cc:740] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:44.360832 30205 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:44.323179 30102 raft_consensus.cc:697] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:44.361924 30075 consensus_queue.cc:1059] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:09:44.363803 30154 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:44.299660 30176 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:44.329891 30089 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:44.308245 30173 consensus_queue.cc:1048] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20260709 13:09:44.299499 30179 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.074s
I20260709 13:09:44.299856 30171 consensus_queue.cc:1059] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.074s
I20260709 13:09:44.373730 25285 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:44.377326 30003 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:44.331519 25209 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:44.437856 30068 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:09:44.524861 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 193448us
I20260709 13:09:44.526890 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:44.391264 30038 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:44.434896 30250 consensus_queue.cc:1059] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.160s
I20260709 13:09:44.509052 30178 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:44.565176 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:44.567135 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:44.570215 25241 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:44.547654 30205 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:44.496192 30176 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:44.579718 25943 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 13:09:44.562589 30075 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:44.576243 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:44.556555 30056 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:44.575569 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:44.585839 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:44.586583 25940 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:09:44.582015 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 420287us
I20260709 13:09:44.571566 30089 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:44.595381 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 776) took 1744 ms. Trace:
I20260709 13:09:44.549619 30003 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:44.564165 29854 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.021s
I20260709 13:09:44.552068 30068 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:44.577044 30205 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:44.622215 25387 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 13:09:44.578413 30178 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:44.628579 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 466899us
I20260709 13:09:44.606485 30174 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:44.665488 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:44.667209 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:44.671509 25940 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:44.616267 30205 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:44.675776 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:44.678460 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:44.686499 25939 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:44.571756 30240 consensus_queue.cc:260] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:44.601893 29928 consensus_queue.cc:1059] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 13:09:44.625413 30056 raft_consensus.cc:697] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:44.612617 30089 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:44.623502 30234 consensus_queue.cc:1059] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.069s
I20260709 13:09:44.703379 30101 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:44.603137 30102 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:44.717919 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:44.718403 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:44.720837 25358 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.68:39211 timed out after 1.502s (SENT) [suppressed 21 similar messages]
I20260709 13:09:44.733596 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:44.681828 30174 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:44.740209 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:44.740597 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 366962us
I20260709 13:09:44.742125 25939 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:44.730261 29854 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 13:09:44.762764 30263 raft_consensus.cc:3055] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:44.765328 30263 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:44.766310 30263 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:44.743788 25944 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:44.767836 30062 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:09:44.768918 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:44.770219 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:44.771394 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:44.772606 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:44.736368 30241 consensus_meta.cc:287] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.642s	user 0.012s	sys 0.000s
I20260709 13:09:44.773325 30241 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:44.745666 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:44.776247 25942 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:44.776907 25938 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:09:44.779096 30243 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.614s	user 0.000s	sys 0.004s
I20260709 13:09:44.746891 30086 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:44.749500 30171 raft_consensus.cc:3055] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Stepping down as leader of term 11
W20260709 13:09:44.757645 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:44.786263 25314 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.019s	sys 0.025s
I20260709 13:09:44.758523 30092 raft_consensus.cc:3055] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:44.743297 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:44.745915 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:44.786648 25314 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.019s	sys 0.025s
I20260709 13:09:44.788498 25945 raft_consensus.cc:2410] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 13:09:44.789651 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:44.790872 25939 raft_consensus.cc:2410] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 13:09:44.707278 30258 consensus_queue.cc:1059] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.086s
I20260709 13:09:44.783267 25285 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:44.782294 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 464448us
I20260709 13:09:44.795322 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:44.727376 30061 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:09:44.795783 25938 raft_consensus.cc:2410] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 13:09:44.767613 30263 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:44.762781 29854 consensus_queue.cc:1243] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer a2a77ecf2c0941618b5bc79d4c04db61 log is divergent from this leader: its last log entry 13.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:09:44.763912 30006 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:09:44.790380 29964 consensus_meta.cc:287] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.505s	user 0.010s	sys 0.002s
I20260709 13:09:44.804543 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:44.808969 25945 raft_consensus.cc:2393] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:44.807708 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:44.810293 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:44.811509 25943 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:09:44.811697 25938 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 12: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:44.789597 30092 raft_consensus.cc:740] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:44.789459 30171 raft_consensus.cc:740] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:44.764353 29844 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.042s
I20260709 13:09:44.830400 25939 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:09:44.763240 30129 consensus_queue.cc:1059] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 13:09:44.799504 30086 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:44.820407 25316 raft_consensus.cc:2955] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Committing config change with OpId 12.10: config changed from index 4 to 10, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:44.803283 30176 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:44.800526 25285 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:44.811595 30183 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:44.863603 25319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2425) took 1546 ms. Trace:
I20260709 13:09:44.873560 30038 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:44.874233 30174 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:44.874450 30038 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:44.810628 29961 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 13:09:44.875083 30174 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:09:44.873631 25319 rpcz_store.cc:276] 0709 13:09:43.317554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.241383 (+923829us) service_pool.cc:224] Handling call
0709 13:09:44.241644 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:44.241857 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:09:44.620150 (+378293us) log.cc:844] Serialized 14 byte log entry
0709 13:09:44.863149 (+242999us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:44.863211 (+    62us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:44.863268 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:44.863343 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:44.863526 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":448990,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":44,"raft.queue_time_us":103,"thread_start_us":599526,"threads_started":2}
I20260709 13:09:44.808841 30056 consensus_queue.cc:237] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:44.876493 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:44.877997 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:44.879211 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:44.880380 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:44.883899 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 894) took 1178 ms. Trace:
I20260709 13:09:44.884370 25937 rpcz_store.cc:276] 0709 13:09:43.706419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:43.706588 (+   169us) service_pool.cc:224] Handling call
0709 13:09:43.762716 (+ 56128us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:44.187027 (+424311us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:44.187068 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:44.207040 (+ 19972us) log.cc:844] Serialized 26 byte log entry
0709 13:09:44.622479 (+415439us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:44.622521 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:44.622686 (+   165us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:44.883652 (+260966us) raft_consensus.cc:1685] finished
0709 13:09:44.883680 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:44.883832 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":144458,"raft.queue_time_us":269726,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":3749,"thread_start_us":645628,"threads_started":2,"wal-append.queue_time_us":648773}
W20260709 13:09:44.864429 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 64055.4us
I20260709 13:09:44.843390 30168 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:44.857159 30065 raft_consensus.cc:3055] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:44.810375 29964 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:44.901954 25285 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:44.905206 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:44.906656 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2428) took 1583 ms. Trace:
I20260709 13:09:44.907419 25390 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 13:09:44.912087 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:44.844676 30006 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.051s
I20260709 13:09:44.864530 30166 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:44.915310 25313 rpcz_store.cc:276] 0709 13:09:43.333005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.284781 (+951776us) service_pool.cc:224] Handling call
0709 13:09:44.285089 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:44.285329 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:09:44.690156 (+404827us) log.cc:844] Serialized 14 byte log entry
0709 13:09:44.906152 (+215996us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:44.906213 (+    61us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:44.906310 (+    97us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:44.906403 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:44.906579 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":427501,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":44,"thread_start_us":603056,"threads_started":2}
W20260709 13:09:44.917541 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:44.875074 30038 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:44.913370 25386 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:44.922247 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2426) took 1604 ms. Trace:
I20260709 13:09:44.920701 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:44.934043 25390 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:44.939352 25390 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:44.933503 25320 rpcz_store.cc:276] 0709 13:09:43.317986 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.277604 (+959618us) service_pool.cc:224] Handling call
0709 13:09:44.278024 (+   420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:44.293095 (+ 15071us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:44.293160 (+    65us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:44.301562 (+  8402us) log.cc:844] Serialized 26 byte log entry
0709 13:09:44.667036 (+365474us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:09:44.667076 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:44.667142 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:44.881440 (+214298us) spinlock_profiling.cc:243] Waited 7.92 ms on lock 0x7b4800271ed8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297bbdbaa1 00007f297bbd2b37 00007f297bbd29ca 00007f297bcca26f 00007f297bcc9f37 00007f297bcde55c 00007f297bcde508 00007f297bcde488 00007f297bcde443 00007f297bcdd6ec 00007f297ac07a96
0709 13:09:44.921869 (+ 40429us) raft_consensus.cc:1685] finished
0709 13:09:44.921904 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:44.922169 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14531,"raft.queue_time_us":22889,"raft.run_cpu_time_us":3190,"raft.run_wall_time_us":578281,"spinlock_wait_cycles":18216832,"thread_start_us":284073,"threads_started":1,"wal-append.queue_time_us":478962}
I20260709 13:09:44.880980 25319 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
W20260709 13:09:44.944819 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 224021us
I20260709 13:09:44.945029 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:44.947392 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:44.926064 30137 consensus_meta.cc:287] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.549s	user 0.011s	sys 0.000s
I20260709 13:09:44.961405 25389 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:44.963037 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:44.966027 30257 consensus_queue.cc:1059] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:44.971678 30234 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:44.903264 30065 raft_consensus.cc:740] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:09:44.979084 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 243502us
I20260709 13:09:44.937594 30038 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:44.989100 25212 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:44.910763 30167 consensus_queue.cc:1059] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.074s
W20260709 13:09:45.021535 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 104230us
I20260709 13:09:44.924351 30168 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.038614 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:44.992935 29854 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:45.055678 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:45.009729 29894 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:44.904752 30171 consensus_queue.cc:260] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.017163 30061 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:45.005501 30089 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:45.078171 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:44.903798 30092 consensus_queue.cc:260] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:45.057879 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 156038us
I20260709 13:09:45.087224 25285 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:44.988863 30137 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.038260 30183 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:09:45.102389 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 63964.8us
I20260709 13:09:45.104456 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:45.104555 25358 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:45.105554 25317 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:45.023820 29872 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20260709 13:09:45.024199 30246 consensus_queue.cc:1059] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.078s
I20260709 13:09:45.115527 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:45.116240 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:45.111197 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:45.120612 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:45.122316 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:45.123927 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:45.124377 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:45.092176 30089 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:45.137027 30068 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:45.137420 30089 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.138000 29964 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:45.138885 29964 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:45.139616 29964 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:45.038348 29988 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.070s
W20260709 13:09:45.147171 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:45.102535 30061 raft_consensus.cc:697] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:45.149719 25237 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:09:45.150820 25317 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
W20260709 13:09:45.126266 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 103229us
I20260709 13:09:45.125092 30097 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:09:45.117055 30154 consensus_meta.cc:287] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.662s	user 0.013s	sys 0.000s
I20260709 13:09:45.152848 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:45.079631 29854 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.129091 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:45.131080 30168 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:45.132530 30055 consensus_meta.cc:287] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.855s	user 0.009s	sys 0.006s
W20260709 13:09:45.134021 30217 consensus_meta.cc:287] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.795s	user 0.011s	sys 0.000s
I20260709 13:09:45.135789 30273 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:45.128762 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:45.067824 30065 consensus_queue.cc:260] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:45.186551 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:45.145661 25390 pending_rounds.cc:85] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 6
I20260709 13:09:45.153620 25317 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:45.153666 25287 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:45.128621 30101 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:45.140002 30089 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:45.117324 30215 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:45.123782 30183 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:45.180675 30092 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:45.182149 25321 raft_consensus.cc:2410] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 13:09:45.123179 29961 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:45.183611 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:45.155124 30075 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:45.103349 30171 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:45.187817 25390 pending_rounds.cc:107] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 13.7
I20260709 13:09:45.140033 29964 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:45.192165 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:45.201424 30282 log.cc:927] Time spent T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.135s	user 0.000s	sys 0.001s
I20260709 13:09:45.142607 30167 consensus_queue.cc:1059] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
W20260709 13:09:45.102027 30176 consensus_queue.cc:1175] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46de6919f85341c6a9b574dff112e35a" responder_term: 12 status { last_received { term: 9 index: 9 } last_committed_idx: 9 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:09:45.206794 30178 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:45.208706 29900 raft_consensus.cc:3055] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:09:45.272480 29900 raft_consensus.cc:740] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:45.273188 29854 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:45.154018 30156 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:45.266633 29966 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:45.274227 30089 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:45.273959 30156 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:45.274654 30089 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:45.275024 30089 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:45.273535 29900 consensus_queue.cc:260] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:45.277294 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:45.277455 30167 raft_consensus.cc:3055] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:45.277696 29900 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:45.277906 30167 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:45.268448 30306 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:45.278764 30167 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.278800 30306 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.280125 30167 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:45.251081 30215 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:45.281283 30061 consensus_queue.cc:237] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:45.283232 30215 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:45.194707 30217 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.287822 30217 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:45.288848 30299 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
W20260709 13:09:45.292588 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:45.251132 30137 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:45.294077 25392 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 12: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:45.246974 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:45.195808 30101 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:45.214954 30154 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:45.325241 25314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2424) took 2034 ms. Trace:
W20260709 13:09:45.213330 30291 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 13:09:45.259563 25390 raft_consensus.cc:2889] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:09:45.250177 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 53569.7us
I20260709 13:09:45.266773 25944 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:09:45.190754 30006 consensus_queue.cc:1059] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.065s
I20260709 13:09:45.329505 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:45.249992 30065 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:45.213364 30168 consensus_queue.cc:1059] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.061s
I20260709 13:09:45.251590 30151 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:45.266464 29964 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:45.261621 30075 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:45.259107 30081 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:45.085870 29894 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.220425 25317 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:45.429411 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:45.297015 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2427) took 1974 ms. Trace:
I20260709 13:09:45.430043 25317 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:45.430197 25316 rpcz_store.cc:276] 0709 13:09:43.322376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.282067 (+959691us) service_pool.cc:224] Handling call
0709 13:09:44.282635 (+   568us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:44.302085 (+ 19450us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:44.302131 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:44.435552 (+133421us) log.cc:844] Serialized 527 byte log entry
0709 13:09:44.789912 (+354360us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:44.951197 (+161285us) log.cc:844] Serialized 14 byte log entry
0709 13:09:45.258898 (+307701us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:45.259038 (+   140us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:45.296702 (+ 37664us) raft_consensus.cc:1685] finished
0709 13:09:45.296732 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:45.296941 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29558,"raft.queue_time_us":149013,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":23766,"thread_start_us":570753,"threads_started":2,"wal-append.queue_time_us":468768}
W20260709 13:09:45.209100 30205 consensus_meta.cc:287] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.509s	user 0.008s	sys 0.000s
I20260709 13:09:45.317503 25244 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
W20260709 13:09:45.299028 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:45.294065 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1893) took 1036 ms. Trace:
W20260709 13:09:45.268784 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 76744.3us
I20260709 13:09:45.435285 25287 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:45.395500 30288 log.cc:927] Time spent T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 13:09:45.241400 30183 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.266937 30301 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:45.191736 30055 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.326903 25314 rpcz_store.cc:276] 0709 13:09:43.316876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.208972 (+892096us) service_pool.cc:224] Handling call
0709 13:09:44.209412 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:44.209673 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:09:44.788439 (+578766us) log.cc:844] Serialized 14 byte log entry
0709 13:09:45.324811 (+536372us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:45.324875 (+    64us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:45.324932 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:45.325004 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:45.325173 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17914,"prepare.queue_time_us":414126,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":18679,"raft.queue_time_us":85526,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":15383,"thread_start_us":1073558,"threads_started":3}
I20260709 13:09:45.349798 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:45.445441 25237 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 12: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:45.314846 30086 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:45.258490 29961 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:45.326853 30280 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:45.330570 30154 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:45.409497 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2241) took 1104 ms. Trace:
I20260709 13:09:45.317219 30038 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:45.454339 30070 consensus_meta.cc:287] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.126s	user 0.015s	sys 0.000s
W20260709 13:09:45.495525 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:45.498287 30101 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:45.514897 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:45.459854 30316 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:45.268889 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 85026.6us
I20260709 13:09:45.432950 25391 rpcz_store.cc:276] 0709 13:09:44.257743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.258630 (+   887us) service_pool.cc:224] Handling call
0709 13:09:44.259040 (+   410us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:44.259557 (+   517us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:09:44.585728 (+326171us) log.cc:844] Serialized 14 byte log entry
0709 13:09:45.022719 (+436991us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:45.036914 (+ 14195us) log.cc:844] Serialized 26 byte log entry
0709 13:09:45.037190 (+   276us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:09:45.048413 (+ 11223us) log.cc:844] Serialized 14 byte log entry
0709 13:09:45.048672 (+   259us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:45.048774 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:45.293801 (+245027us) raft_consensus.cc:1685] finished
0709 13:09:45.293820 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:45.294003 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2524,"prepare.queue_time_us":493954,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":78,"raft.queue_time_us":231919,"raft.run_cpu_time_us":382,"raft.run_wall_time_us":21024,"thread_start_us":625946,"threads_started":3,"wal-append.queue_time_us":630442}
I20260709 13:09:45.559736 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:45.438506 25908 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:45.386422 30151 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:45.443674 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:45.444331 25316 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:09:45.432214 30081 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.449404 30075 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:45.452510 25314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2430) took 2062 ms. Trace:
I20260709 13:09:45.389991 30318 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.081s
I20260709 13:09:45.436462 30068 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:45.583277 30248 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:45.587131 30068 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:45.588127 30248 raft_consensus.cc:697] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:45.589557 30248 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:45.561431 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 123047us
I20260709 13:09:45.480365 30154 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:45.440121 30205 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.577525 25316 raft_consensus.cc:606] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:45.580680 25358 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:45.583261 25314 rpcz_store.cc:276] 0709 13:09:43.390162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.451533 (+2061371us) service_pool.cc:224] Handling call
0709 13:09:45.451915 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:45.452131 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:45.452160 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:45.452205 (+    45us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:09:45.452246 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:45.452303 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:45.452420 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:45.316743 30178 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:45.586385 30075 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:45.590373 30068 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:45.598944 30318 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:45.606060 30178 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:45.606902 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:45.607540 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49638 (request call id 29) took 1309 ms. Trace:
I20260709 13:09:45.456791 30301 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.598883 30101 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:45.557636 30070 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:09:45.473955 25242 rpcz_store.cc:276] 0709 13:09:44.304755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.304920 (+   165us) service_pool.cc:224] Handling call
0709 13:09:44.305368 (+   448us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:44.316593 (+ 11225us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:09:44.700839 (+384246us) log.cc:844] Serialized 14 byte log entry
0709 13:09:44.980011 (+279172us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:44.985625 (+  5614us) log.cc:844] Serialized 26 byte log entry
0709 13:09:44.985830 (+   205us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:44.985921 (+    91us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:44.986010 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:45.409114 (+423104us) raft_consensus.cc:1685] finished
0709 13:09:45.409189 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:45.409401 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55498,"prepare.queue_time_us":414130,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":3950,"raft.queue_time_us":66516,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":31840,"thread_start_us":564049,"threads_started":2,"wal-append.queue_time_us":373304}
I20260709 13:09:45.594569 30097 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:45.625360 25239 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:09:45.615275 25358 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:45.612521 25316 rpcz_store.cc:276] 0709 13:09:44.295649 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.431581 (+1135932us) service_pool.cc:224] Handling call
0709 13:09:45.604950 (+173369us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:45.591828 30302 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:45.631420 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2566) took 1185 ms. Trace:
I20260709 13:09:45.634333 30151 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:45.607168 30006 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 13:09:45.636247 30275 log.cc:927] Time spent T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.619s	user 0.004s	sys 0.000s
I20260709 13:09:45.460930 30086 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:45.665078 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 49947.5us
I20260709 13:09:45.631876 25388 rpcz_store.cc:276] 0709 13:09:44.445937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.450859 (+  4922us) service_pool.cc:224] Handling call
0709 13:09:44.451281 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:44.455662 (+  4381us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:09:44.804964 (+349302us) log.cc:844] Serialized 14 byte log entry
0709 13:09:44.805383 (+   419us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:44.805437 (+    54us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:44.805488 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:45.629858 (+824370us) spinlock_profiling.cc:243] Waited 1.73 ms on lock 0x7b18001f9b40. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b932a6 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419
0709 13:09:45.631154 (+  1296us) spinlock_profiling.cc:243] Waited 4.05 ms on lock 0x7b54003d4ad4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:45.631206 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:45.631344 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55452,"prepare.queue_time_us":314061,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"raft.queue_time_us":97,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":5321,"spinlock_wait_cycles":13303040,"thread_start_us":295595,"threads_started":1}
I20260709 13:09:45.666963 25317 raft_consensus.cc:2410] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:09:45.672008 25358 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:45.611028 30075 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:45.618170 30178 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:45.616600 30300 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:45.585150 30316 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:45.675051 30006 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:45.684407 25285 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:45.686681 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:45.690583 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:45.691203 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:45.692379 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:45.692829 25938 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:45.669363 30151 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:45.686197 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:45.713850 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:45.607472 30318 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.724642 25358 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:45.716375 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:45.718182 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:45.714608 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:45.743817 25945 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 12: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:45.744917 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:45.642161 30097 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:45.748478 30314 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:45.749186 30315 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:45.749106 30314 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.749769 30315 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:45.750486 30097 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:45.751487 30314 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:45.733630 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:45.715582 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:45.720216 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 777) took 2869 ms. Trace:
I20260709 13:09:45.753065 25318 rpcz_store.cc:276] 0709 13:09:42.851175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:43.926943 (+1075768us) service_pool.cc:224] Handling call
0709 13:09:43.975263 (+ 48320us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:43.996129 (+ 20866us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:09:44.287256 (+291127us) log.cc:844] Serialized 14 byte log entry
0709 13:09:44.710612 (+423356us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:44.766969 (+ 56357us) spinlock_profiling.cc:243] Waited 1.02 ms on lock 0x7b2c0009d3b8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f29755aba11 00007f29755bb372 00007f2976ceda01 00007f2976cb243a 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2
0709 13:09:44.767405 (+   436us) log.cc:844] Serialized 26 byte log entry
0709 13:09:44.767568 (+   163us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:44.792431 (+ 24863us) log.cc:844] Serialized 14 byte log entry
0709 13:09:44.792676 (+   245us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:44.792752 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:45.635555 (+842803us) spinlock_profiling.cc:243] Waited 31.4 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:09:45.719884 (+ 84329us) raft_consensus.cc:1685] finished
0709 13:09:45.719919 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:45.720136 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24556,"prepare.queue_time_us":579582,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":3736,"raft.queue_time_us":22159,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":15530,"spinlock_wait_cycles":74615936,"thread_start_us":647444,"threads_started":2,"wal-append.queue_time_us":594854}
I20260709 13:09:45.755431 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:45.756040 25318 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:45.756938 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:45.757601 25318 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:45.696524 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:45.696007 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:45.714278 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:45.726907 25321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2356) took 3174 ms (client timeout 1570 ms). Trace:
I20260709 13:09:45.730362 25394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2565) took 1346 ms. Trace:
I20260709 13:09:45.751374 30292 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:45.769846 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:45.698128 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:45.762544 25321 rpcz_store.cc:269] 0709 13:09:42.552493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:42.563851 (+ 11358us) service_pool.cc:224] Handling call
0709 13:09:45.726596 (+3162745us) spinlock_profiling.cc:243] Waited 2.58 s on lock 0x7b54000b22d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:45.726808 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5931355904}
I20260709 13:09:45.683225 30168 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:45.773247 25940 raft_consensus.cc:2393] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:09:45.755555 30266 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.776s	user 0.003s	sys 0.003s
I20260709 13:09:45.762952 25394 rpcz_store.cc:276] 0709 13:09:44.384192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.391103 (+  6911us) service_pool.cc:224] Handling call
0709 13:09:44.447924 (+ 56821us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:44.448250 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:09:44.736302 (+288052us) log.cc:844] Serialized 14 byte log entry
0709 13:09:45.285734 (+549432us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b1800043e40. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:09:45.729575 (+443841us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b1800043e40. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f297685a4bc 00007f2973ba848b 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc2455 00007f2976caa060 00007f2976cbc716 00007f2976ccd287
0709 13:09:45.729850 (+   275us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:45.729938 (+    88us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:45.730017 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:45.730094 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:45.730278 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":321308,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"raft.queue_time_us":152168,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":9003,"spinlock_wait_cycles":63724160,"thread_start_us":593050,"threads_started":3,"wal-append.queue_time_us":363678}
I20260709 13:09:45.770594 25388 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:45.681946 30055 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:45.694133 30178 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:45.694106 29854 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:45.748927 30321 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:45.722739 30086 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:45.811437 30240 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:09:45.722860 30156 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:45.724471 30006 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:45.811836 30321 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.804749 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:45.781935 25939 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 13:09:45.785806 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:45.814400 30321 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:45.815297 25943 raft_consensus.cc:2410] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 13:09:45.785529 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:45.758239 30300 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:45.816180 30300 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:45.816593 30300 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:45.677194 30101 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.097s
I20260709 13:09:45.822368 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:45.795768 30135 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:09:45.823287 25314 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 12: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:45.824134 25314 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:09:45.824503 30316 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:45.777002 30292 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:45.827191 30292 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:45.797477 30314 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:45.828665 30314 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:45.830027 25395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2540) took 2848 ms. Trace:
I20260709 13:09:45.830334 25312 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 13:09:45.798144 30097 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:45.831192 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2432) took 2410 ms. Trace:
I20260709 13:09:45.831185 30097 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:45.831741 25312 rpcz_store.cc:276] 0709 13:09:43.421295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.829735 (+2408440us) service_pool.cc:224] Handling call
0709 13:09:45.830094 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:45.830927 (+   833us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:45.831136 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:45.798899 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:45.833530 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:45.834352 25945 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 11: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:45.834399 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2456) took 1501 ms. Trace:
I20260709 13:09:45.834877 25321 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:09:45.834818 25317 rpcz_store.cc:276] 0709 13:09:44.333382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.430782 (+1097400us) service_pool.cc:224] Handling call
0709 13:09:45.834214 (+403432us) spinlock_profiling.cc:243] Waited 93.7 ms on lock 0x7b54000622d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:45.834343 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":215482752}
I20260709 13:09:45.836025 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2396) took 2394 ms. Trace:
I20260709 13:09:45.836685 30183 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:45.783710 29900 consensus_meta.cc:287] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.506s	user 0.010s	sys 0.001s
I20260709 13:09:45.837908 25937 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:09:45.838160 29961 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:45.839583 30081 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:45.839804 30318 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:45.830845 30314 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:45.833498 30097 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:45.685885 30217 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.092s
I20260709 13:09:45.841958 30217 consensus_queue.cc:1243] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer f646dfd532974dceadfe00005d2bbf59 log is divergent from this leader: its last log entry 11.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:09:45.800057 25209 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:45.843894 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 44021.6us
I20260709 13:09:45.844008 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:45.796778 25243 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:09:45.806222 30003 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:44.604483 25315 rpcz_store.cc:276] 0709 13:09:42.850937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:43.753948 (+903011us) service_pool.cc:224] Handling call
0709 13:09:43.754271 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:43.754533 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:43.754560 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:43.754992 (+   432us) log.cc:844] Serialized 26 byte log entry
0709 13:09:44.101288 (+346296us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:09:44.575654 (+474366us) log.cc:844] Serialized 14 byte log entry
0709 13:09:44.594951 (+ 19297us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:44.595058 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:44.595114 (+    56us) raft_consensus.cc:1685] finished
0709 13:09:44.595132 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:44.595305 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9270,"prepare.queue_time_us":486596,"raft.queue_time_us":27156,"thread_start_us":643515,"threads_started":2,"wal-append.queue_time_us":374831}
I20260709 13:09:45.855453 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2435) took 2412 ms. Trace:
I20260709 13:09:45.808673 29966 raft_consensus.cc:3055] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Stepping down as leader of term 12
W20260709 13:09:45.811264 30089 consensus_meta.cc:287] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.536s	user 0.012s	sys 0.001s
I20260709 13:09:45.812580 30006 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:45.815510 25129 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f75b60146e7d4b9b9954b4fb2c414679 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 46de6919f85341c6a9b574dff112e35a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 11 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:45.827991 25238 raft_consensus.cc:2410] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 13:09:45.784471 25938 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:45.716759 30258 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:45.862036 25938 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
W20260709 13:09:45.860054 25129 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 74667.9us
I20260709 13:09:45.847278 25318 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:45.836369 25321 rpcz_store.cc:276] 0709 13:09:43.441718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.834276 (+2392558us) service_pool.cc:224] Handling call
0709 13:09:45.834572 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:45.835401 (+   829us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:45.835576 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:45.863771 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:45.830435 25395 rpcz_store.cc:276] 0709 13:09:42.981600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:43.078547 (+ 96947us) service_pool.cc:224] Handling call
0709 13:09:43.078901 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:43.079423 (+   522us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:43.079466 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:43.103313 (+ 23847us) log.cc:844] Serialized 26 byte log entry
0709 13:09:44.228082 (+1124769us) spinlock_profiling.cc:243] Waited 3.06 ms on lock 0x7b18002fb420. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f29768520d1 00007f2976c6da0f
0709 13:09:44.697205 (+469123us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:44.697265 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:44.697363 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:45.754955 (+1057592us) spinlock_profiling.cc:243] Waited 70.3 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:09:45.829804 (+ 74849us) raft_consensus.cc:1685] finished
0709 13:09:45.829825 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:45.829964 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2874,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":13658,"spinlock_wait_cycles":168841216,"thread_start_us":468841,"threads_started":1,"wal-append.queue_time_us":1761482}
W20260709 13:09:45.778213 30298 log.cc:927] Time spent T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.299s	user 0.000s	sys 0.001s
I20260709 13:09:45.681052 30061 consensus_queue.cc:1059] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.104s
I20260709 13:09:45.877734 30061 raft_consensus.cc:3055] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:45.877588 30258 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:45.804049 30167 consensus_meta.cc:287] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.523s	user 0.010s	sys 0.001s
I20260709 13:09:45.836455 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2434) took 2393 ms. Trace:
I20260709 13:09:45.879302 25317 rpcz_store.cc:276] 0709 13:09:43.442802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.835736 (+2392934us) service_pool.cc:224] Handling call
0709 13:09:45.836013 (+   277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:45.836167 (+   154us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:45.836187 (+    20us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:45.836221 (+    34us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:09:45.836257 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:45.836305 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:45.836397 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:45.880396 30258 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:09:45.855855 25315 rpcz_store.cc:276] 0709 13:09:43.443127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.854605 (+2411478us) service_pool.cc:224] Handling call
0709 13:09:45.854902 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:45.855080 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:45.855103 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:45.855143 (+    40us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:45.855173 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:45.855232 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:45.855391 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:45.881291 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2568) took 1326 ms. Trace:
I20260709 13:09:45.858845 25358 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:45.859808 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:45.865686 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:45.860411 25937 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:45.866436 25908 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:45.785298 30315 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:45.867364 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:45.857493 29966 raft_consensus.cc:740] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:45.888070 25314 raft_consensus.cc:606] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:45.888582 25314 raft_consensus.cc:614] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:09:45.865371 30089 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:45.870157 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 897) took 1533 ms. Trace:
I20260709 13:09:45.874132 25938 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:45.874696 30292 consensus_queue.cc:1059] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:09:45.878234 30061 raft_consensus.cc:740] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:45.856482 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:45.890971 25942 rpcz_store.cc:276] 0709 13:09:44.336555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.336825 (+   270us) service_pool.cc:224] Handling call
0709 13:09:44.357548 (+ 20723us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:44.776916 (+419368us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:45.869929 (+1093013us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x7b5400644954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:45.870098 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":961868672}
I20260709 13:09:45.868748 30323 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:45.886899 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:45.881701 25389 rpcz_store.cc:276] 0709 13:09:44.554608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.554970 (+   362us) service_pool.cc:224] Handling call
0709 13:09:44.555390 (+   420us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:44.978093 (+422703us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:45.880993 (+902900us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b54001e3f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:45.881223 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":933230336}
I20260709 13:09:45.883361 25312 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:45.887873 25936 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:45.746378 30151 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.869436 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 896) took 1632 ms. Trace:
I20260709 13:09:45.899155 25941 rpcz_store.cc:276] 0709 13:09:44.237505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.238102 (+   597us) service_pool.cc:224] Handling call
0709 13:09:44.250714 (+ 12612us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:44.290345 (+ 39631us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:45.869182 (+1578837us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b54007044d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:45.869374 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29135104}
I20260709 13:09:45.858481 30135 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election won for term 12
W20260709 13:09:45.890251 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 64267.2us
I20260709 13:09:45.855193 30321 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:45.862840 30003 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:45.909578 30151 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:45.909487 30003 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:45.912022 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:45.912436 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:45.872887 30006 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:45.856099 29894 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:45.900903 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:45.938674 25312 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:45.903142 29966 consensus_queue.cc:260] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.904163 30089 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:45.896047 30315 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:45.947738 25940 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:09:45.883325 30097 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:45.898619 30217 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:45.908564 30321 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:45.896790 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:45.913187 25936 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:45.919139 25909 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:45.901523 25318 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:45.963608 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:45.940318 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:45.941319 25908 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:45.960085 25321 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:45.945935 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:45.946102 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2573) took 1229 ms. Trace:
I20260709 13:09:45.922159 30135 raft_consensus.cc:697] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:45.899268 30258 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:45.905315 30323 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:45.971989 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 52988.4us
W20260709 13:09:45.954421 30092 consensus_meta.cc:287] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.710s	user 0.008s	sys 0.004s
I20260709 13:09:45.972029 25313 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260709 13:09:45.969249 25392 rpcz_store.cc:276] 0709 13:09:44.717003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.718059 (+  1056us) service_pool.cc:224] Handling call
0709 13:09:45.945892 (+1227833us) spinlock_profiling.cc:243] Waited 575 ms on lock 0x7b54003d59d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:45.946039 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1321388800}
I20260709 13:09:45.972981 25320 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 13:09:45.976698 25909 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:45.966195 25908 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:46.003180 25313 pending_rounds.cc:85] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 7
I20260709 13:09:45.951788 30205 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:45.961861 30003 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:46.013994 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2571) took 1387 ms. Trace:
W20260709 13:09:46.014801 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 48732.7us
I20260709 13:09:46.014590 25387 rpcz_store.cc:276] 0709 13:09:44.627025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.633317 (+  6292us) service_pool.cc:224] Handling call
0709 13:09:44.679061 (+ 45744us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:44.793251 (+114190us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:44.793295 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:44.793339 (+    44us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:44.793370 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.013707 (+1220337us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54003d54d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:46.013764 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:46.013916 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":262047616}
I20260709 13:09:46.004307 25209 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:45.950565 30006 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:45.944602 30061 consensus_queue.cc:260] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:46.003466 29964 consensus_meta.cc:287] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.606s	user 0.009s	sys 0.002s
I20260709 13:09:46.003698 25313 pending_rounds.cc:107] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 13:09:46.036976 25313 raft_consensus.cc:2889] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:46.012033 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:46.012213 25318 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:46.014212 25317 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:09:45.977463 30240 raft_consensus.cc:2764] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 13:09:46.005702 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2447) took 2471 ms. Trace:
I20260709 13:09:46.039956 25320 rpcz_store.cc:276] 0709 13:09:43.534732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.968232 (+2433500us) service_pool.cc:224] Handling call
0709 13:09:45.968638 (+   406us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:46.003519 (+ 34881us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.005618 (+  2099us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:45.977448 30259 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:45.975919 29966 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:45.961288 30097 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:46.052613 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2449) took 2517 ms. Trace:
I20260709 13:09:46.053333 25313 rpcz_store.cc:276] 0709 13:09:43.535290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.971347 (+2436057us) service_pool.cc:224] Handling call
0709 13:09:45.971680 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:46.051189 (+ 79509us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.052553 (+  1364us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:46.053805 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:46.046661 25940 pending_rounds.cc:85] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 9
W20260709 13:09:46.050379 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:46.007490 30258 raft_consensus.cc:697] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:46.054782 25940 pending_rounds.cc:107] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 13:09:46.057467 25944 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 947) took 1298 ms. Trace:
I20260709 13:09:46.057650 25940 raft_consensus.cc:2889] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:46.017740 30315 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:45.971499 30217 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:46.055133 25389 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:46.019018 29928 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:46.033742 29872 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:46.057893 25944 rpcz_store.cc:276] 0709 13:09:44.747566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:44.772326 (+ 24760us) service_pool.cc:224] Handling call
0709 13:09:46.045635 (+1273309us) spinlock_profiling.cc:243] Waited 440 ms on lock 0x7b5400690fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:46.045787 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1013031680}
I20260709 13:09:46.062598 25284 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:46.036628 29894 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:46.097376 29894 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:46.097847 29894 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:46.033550 29900 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:46.093521 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:46.043011 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2454) took 2312 ms. Trace:
W20260709 13:09:46.076875 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:46.100096 25394 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:46.087730 30129 consensus_queue.cc:1059] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:09:46.089866 30215 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:09:46.093518 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:46.096539 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:46.051692 30241 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:46.067833 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2455) took 1742 ms. Trace:
I20260709 13:09:46.076977 25941 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:46.100024 25320 rpcz_store.cc:276] 0709 13:09:43.730734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.041870 (+2311136us) service_pool.cc:224] Handling call
0709 13:09:46.042297 (+   427us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:46.042553 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:46.042595 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:46.042660 (+    65us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:46.042715 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.042792 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:46.042936 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:46.047463 30259 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:09:46.099535 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 183393us
I20260709 13:09:46.059950 30234 raft_consensus.cc:3055] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:46.107278 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:46.112483 25320 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:09:46.107296 25244 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:46.105858 25313 rpcz_store.cc:276] 0709 13:09:44.325893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.064123 (+1738230us) service_pool.cc:224] Handling call
0709 13:09:46.067168 (+  3045us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:46.067404 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:46.067440 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:46.067501 (+    61us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:46.067555 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.067626 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:46.067763 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:46.111238 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:46.119275 25394 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:46.064617 30240 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:46.037807 30006 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:46.113063 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2458) took 1546 ms. Trace:
I20260709 13:09:46.040375 29964 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:46.126236 25320 rpcz_store.cc:276] 0709 13:09:44.566722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.111961 (+1545239us) service_pool.cc:224] Handling call
0709 13:09:46.112258 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:46.112851 (+   593us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.113020 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:46.131191 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2452) took 2459 ms. Trace:
I20260709 13:09:46.132133 25320 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:46.052320 30154 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:46.132726 25911 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:46.052196 30205 consensus_queue.cc:1059] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.057s
I20260709 13:09:46.067185 30315 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:46.135322 30205 raft_consensus.cc:3055] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:46.135789 30205 raft_consensus.cc:740] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:46.110160 30259 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:46.075958 29872 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:46.133013 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 798) took 1267 ms. Trace:
I20260709 13:09:46.060752 30245 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.061s
I20260709 13:09:46.099942 30003 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:46.060941 30065 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
I20260709 13:09:46.075620 30307 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:46.138309 30323 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:46.120412 30168 raft_consensus.cc:3055] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:46.111657 30234 raft_consensus.cc:740] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:46.145447 25320 rpcz_store.cc:276] 0709 13:09:44.866098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.131584 (+1265486us) service_pool.cc:224] Handling call
0709 13:09:46.131917 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:46.132721 (+   804us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.132948 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:46.131772 30241 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:46.147132 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:46.154425 25392 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:46.157848 25911 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:46.161438 25313 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:46.102943 29900 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:46.163066 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 55938.2us
I20260709 13:09:46.147980 29872 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:46.130115 30240 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:46.174304 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2459) took 1605 ms. Trace:
I20260709 13:09:46.155056 29928 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:46.175652 25319 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:09:46.183727 25319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 801) took 1315 ms. Trace:
I20260709 13:09:46.184226 25319 rpcz_store.cc:276] 0709 13:09:44.868608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.175041 (+1306433us) service_pool.cc:224] Handling call
0709 13:09:46.175333 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:46.183362 (+  8029us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.183652 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:46.162094 30092 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:46.160826 30168 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:09:46.185508 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:46.142863 29994 consensus_queue.cc:1059] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 13:09:46.176658 25358 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:46.179628 25313 rpcz_store.cc:276] 0709 13:09:44.568514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.125021 (+1556507us) service_pool.cc:224] Handling call
0709 13:09:46.160653 (+ 35632us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:46.167542 (+  6889us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.174037 (+  6495us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:46.199465 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:46.166747 30304 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:46.212864 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2431) took 2808 ms. Trace:
I20260709 13:09:46.213433 25316 rpcz_store.cc:276] 0709 13:09:43.404033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.629645 (+2225612us) service_pool.cc:224] Handling call
0709 13:09:45.630086 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:45.758768 (+128682us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:45.758816 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:45.758877 (+    61us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:45.758935 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.212567 (+453632us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b540005e454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:46.212632 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:46.212791 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":295268096}
I20260709 13:09:46.208559 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2399) took 2759 ms. Trace:
I20260709 13:09:46.208711 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:46.216246 25942 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:46.217294 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:46.217422 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:46.180753 30248 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:46.121330 30135 consensus_queue.cc:237] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:46.234198 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:46.147548 30258 consensus_queue.cc:237] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:46.345882 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:46.170064 30241 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:46.173796 30242 consensus_queue.cc:1059] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.090s
W20260709 13:09:46.212057 30330 log.cc:927] Time spent T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 13:09:46.265883 29961 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:46.199785 30315 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:46.164091 30307 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:46.393734 25316 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:09:46.221285 30304 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:46.360345 30248 raft_consensus.cc:697] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:46.359961 30245 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:46.390177 30315 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:46.485827 30248 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:46.487684 30089 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:46.488440 30089 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:46.489271 30245 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:46.491137 30089 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:46.491524 30245 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:46.470805 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2453) took 2799 ms. Trace:
I20260709 13:09:46.356205 30234 consensus_queue.cc:260] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:46.492219 25321 rpcz_store.cc:276] 0709 13:09:43.671867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.003715 (+2331848us) service_pool.cc:224] Handling call
0709 13:09:46.026458 (+ 22743us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:46.026899 (+   441us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:09:46.149465 (+122566us) log.cc:844] Serialized 14 byte log entry
0709 13:09:46.470416 (+320951us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:46.470473 (+    57us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:09:46.470524 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.470604 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:46.470737 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":157197,"prepare.queue_time_us":243538,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":4346,"raft.queue_time_us":109657,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":9610,"thread_start_us":302045,"threads_started":2}
I20260709 13:09:46.206518 30092 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:46.495433 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:46.525497 25394 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:46.526784 25911 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:46.206355 30321 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:46.220949 29928 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.151s
W20260709 13:09:46.450017 30300 consensus_meta.cc:287] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.633s	user 0.011s	sys 0.000s
I20260709 13:09:46.355835 30065 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:46.484882 25316 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:46.443595 30304 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:46.493818 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:46.392251 30242 consensus_queue.cc:1243] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer 2329e428ebbf484fa3ea9452133df567 log is divergent from this leader: its last log entry 13.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:09:46.268870 30179 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.129s
I20260709 13:09:46.543606 30234 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:46.584206 25321 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:09:46.447517 30217 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:46.485858 30315 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:46.529922 30245 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:46.443979 30062 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:46.543493 30300 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:46.545280 30137 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:09:46.623535 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 437506us
I20260709 13:09:46.662200 30137 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:46.662722 30137 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:09:46.665247 30327 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.524s	user 0.000s	sys 0.001s
I20260709 13:09:46.629881 25321 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:09:46.641502 25316 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20260709 13:09:46.668048 25316 pending_rounds.cc:85] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 9
I20260709 13:09:46.668331 25316 pending_rounds.cc:107] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 13:09:46.668469 30240 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:46.668627 25316 raft_consensus.cc:2889] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:46.669363 29900 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:46.614802 30065 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:46.671921 30065 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:46.672510 30065 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:46.673731 30217 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:46.673568 29961 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:46.674356 30217 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:46.674497 29961 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:46.622675 30315 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:46.677591 29961 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:46.681825 30337 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.477s	user 0.000s	sys 0.001s
I20260709 13:09:46.632370 30006 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:46.627206 30154 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:46.685885 30302 consensus_meta.cc:287] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.055s	user 0.006s	sys 0.006s
I20260709 13:09:46.686692 30302 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:46.605845 30179 consensus_queue.cc:1243] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer 2329e428ebbf484fa3ea9452133df567 log is divergent from this leader: its last log entry 14.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:09:46.687760 25388 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:46.695930 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:46.696900 25394 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
W20260709 13:09:46.645903 30342 log.cc:927] Time spent T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
W20260709 13:09:46.647126 30336 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.255s	user 0.001s	sys 0.000s
I20260709 13:09:46.391538 30168 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:46.648135 30329 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.476s	user 0.000s	sys 0.001s
I20260709 13:09:46.658499 30215 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:46.659329 30300 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:46.659741 30129 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:46.667586 30038 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:46.706526 25911 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:46.715813 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:46.714670 25910 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.005s	sys 0.030s
W20260709 13:09:46.744057 25910 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.006s	sys 0.031s
W20260709 13:09:46.723690 30055 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.006s	sys 0.032s
I20260709 13:09:46.677023 30217 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:46.760226 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 526088us
I20260709 13:09:46.762890 25358 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:46.764700 25358 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:46.766726 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:46.774338 25358 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:46.755771 30055 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.006s	sys 0.032s
I20260709 13:09:46.791042 30092 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:46.673015 30065 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:46.637544 30245 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:46.766355 30168 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:46.764072 29964 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:46.767457 30300 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:46.720808 30062 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:46.741822 30215 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:46.780778 30179 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:46.794428 30006 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:46.781056 30302 raft_consensus.cc:2764] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 13:09:46.750001 30129 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:46.815193 30345 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:09:46.882108 25313 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.664s	user 0.018s	sys 0.043s
W20260709 13:09:46.882753 25313 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.018s	sys 0.043s
I20260709 13:09:46.870651 30241 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:46.927040 30092 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:46.932286 30092 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:46.933259 30055 raft_consensus.cc:3055] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:09:46.933701 30055 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:46.934602 30092 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:46.934552 30055 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:46.935978 30055 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:46.939539 30205 consensus_queue.cc:260] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:46.946900 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 2199) took 1250 ms. Trace:
I20260709 13:09:46.947368 25238 rpcz_store.cc:276] 0709 13:09:45.696831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.696988 (+   157us) service_pool.cc:224] Handling call
0709 13:09:46.946686 (+1249698us) spinlock_profiling.cc:243] Waited 74.7 ms on lock 0x7b540028bed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:46.946830 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":171841792}
W20260709 13:09:46.978853 30334 log.cc:927] Time spent T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
I20260709 13:09:46.979590 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:46.980219 25238 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:46.980465 25939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 961) took 1143 ms. Trace:
I20260709 13:09:46.952244 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2398) took 3505 ms. Trace:
I20260709 13:09:46.981302 25317 rpcz_store.cc:276] 0709 13:09:43.446421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.880586 (+2434165us) service_pool.cc:224] Handling call
0709 13:09:45.880929 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:46.038606 (+157677us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.951884 (+913278us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b54000f8a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:46.952158 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":305731840}
W20260709 13:09:46.982683 30340 log.cc:927] Time spent T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.189s	user 0.000s	sys 0.001s
I20260709 13:09:46.983507 25317 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:09:46.984813 30306 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:46.988740 25945 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260709 13:09:46.988967 30344 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260709 13:09:46.988776 25942 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:46.993441 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:46.956557 25314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49638 (request call id 30) took 1319 ms. Trace:
I20260709 13:09:46.995493 25314 rpcz_store.cc:276] 0709 13:09:45.637242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.823964 (+186722us) service_pool.cc:224] Handling call
0709 13:09:46.956294 (+1132330us) spinlock_profiling.cc:243] Waited 63.3 ms on lock 0x7b5400115654. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cac853 00007f297bc7a60a 00007f29752007d4 00007f2975200762 00007f2975200691 00007f297520060c 00007f29751ff8c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:46.956439 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":145687424}
I20260709 13:09:46.984126 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2467) took 1126 ms. Trace:
I20260709 13:09:46.953281 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 1036) took 1117 ms. Trace:
I20260709 13:09:46.997150 25937 rpcz_store.cc:276] 0709 13:09:45.835489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.839495 (+  4006us) service_pool.cc:224] Handling call
0709 13:09:45.860243 (+ 20748us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:45.882394 (+ 22151us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.953007 (+1070613us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x7b54005baed0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:46.953204 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":47010048}
I20260709 13:09:46.996623 25317 rpcz_store.cc:276] 0709 13:09:45.857513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.982886 (+1125373us) service_pool.cc:224] Handling call
0709 13:09:46.983184 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:46.983867 (+   683us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.984054 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:46.957765 29854 raft_consensus.cc:3055] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:46.960505 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:46.962391 30215 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:46.963775 30328 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.758s	user 0.001s	sys 0.001s
I20260709 13:09:46.970180 25139 catalog_manager.cc:5697] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:47.021899 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2579) took 1278 ms. Trace:
I20260709 13:09:47.022454 25391 rpcz_store.cc:276] 0709 13:09:45.743783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.744924 (+  1141us) service_pool.cc:224] Handling call
0709 13:09:45.745385 (+   461us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:45.747727 (+  2342us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:45.747775 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:45.779414 (+ 31639us) log.cc:844] Serialized 26 byte log entry
0709 13:09:46.047413 (+267999us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:46.047459 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.047533 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:46.963031 (+915498us) spinlock_profiling.cc:243] Waited 8.61 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:09:47.021469 (+ 58438us) raft_consensus.cc:1685] finished
0709 13:09:47.021501 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:47.021818 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1744,"raft.queue_time_us":2475,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":18042,"spinlock_wait_cycles":19801984,"thread_start_us":224312,"threads_started":1,"wal-append.queue_time_us":360503}
I20260709 13:09:47.030074 25319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 802) took 1922 ms. Trace:
I20260709 13:09:47.030582 25319 rpcz_store.cc:276] 0709 13:09:45.107994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.186692 (+1078698us) service_pool.cc:224] Handling call
0709 13:09:46.187000 (+   308us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:46.204305 (+ 17305us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:46.204355 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:46.213899 (+  9544us) log.cc:844] Serialized 26 byte log entry
0709 13:09:46.827018 (+613119us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:46.874753 (+ 47735us) log.cc:844] Serialized 14 byte log entry
0709 13:09:46.875061 (+   308us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.875178 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:47.029820 (+154642us) raft_consensus.cc:1685] finished
0709 13:09:47.029842 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:47.030005 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1480,"prepare.queue_time_us":50223,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":57,"raft.queue_time_us":19980,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":19674,"thread_start_us":446188,"threads_started":1,"wal-append.queue_time_us":682295}
I20260709 13:09:47.032712 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:47.034060 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:47.034340 29894 consensus_meta.cc:287] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.936s	user 0.012s	sys 0.000s
I20260709 13:09:47.034792 25319 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 12: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:47.034785 29894 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:47.035914 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:47.036794 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:47.037104 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:47.037765 25319 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:47.038632 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:47.039841 29966 consensus_meta.cc:287] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.975s	user 0.007s	sys 0.002s
I20260709 13:09:46.975572 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.041455 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.042887 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.044273 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:46.976140 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:46.976378 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.045567 25237 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:47.046159 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.046481 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:47.047660 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.047803 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:46.977375 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:46.977790 30259 consensus_meta.cc:287] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.822s	user 0.013s	sys 0.000s
I20260709 13:09:47.049211 30259 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:47.051364 30259 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:09:46.934662 25940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 963) took 1026 ms. Trace:
I20260709 13:09:47.051973 25940 rpcz_store.cc:276] 0709 13:09:45.908118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.948953 (+ 40835us) service_pool.cc:224] Handling call
0709 13:09:46.046426 (+ 97473us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:46.058575 (+ 12149us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:46.058609 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:46.059119 (+   510us) log.cc:844] Serialized 52 byte log entry
0709 13:09:46.127374 (+ 68255us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:46.127430 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.127525 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:46.683534 (+556009us) raft_consensus.cc:1685] finished
0709 13:09:46.683564 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:46.934394 (+250830us) spinlock_profiling.cc:243] Waited 78.6 ms on lock 0x7b5400168850. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:46.934596 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1079,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":1898,"spinlock_wait_cycles":180772608,"thread_start_us":67960,"threads_started":1,"wal-append.queue_time_us":108121}
I20260709 13:09:46.960078 30075 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:47.054164 30075 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:47.054634 30075 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:47.054934 30065 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:47.056423 30038 raft_consensus.cc:993] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567: : Instructing follower a2a77ecf2c0941618b5bc79d4c04db61 to start an election
I20260709 13:09:47.057036 30038 raft_consensus.cc:1081] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Signalling peer a2a77ecf2c0941618b5bc79d4c04db61 to start an election
I20260709 13:09:47.057981 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:47.059448 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.059661 30129 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:47.060647 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:46.981021 25939 rpcz_store.cc:276] 0709 13:09:45.837529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.883565 (+ 46036us) service_pool.cc:224] Handling call
0709 13:09:45.893206 (+  9641us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:45.893725 (+   519us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:45.893768 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:45.912589 (+ 18821us) log.cc:844] Serialized 26 byte log entry
0709 13:09:46.112833 (+200244us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:46.112895 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.112991 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:46.980188 (+867197us) raft_consensus.cc:1685] finished
0709 13:09:46.980217 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:46.980390 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":136,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":9403,"thread_start_us":154145,"threads_started":1,"wal-append.queue_time_us":816317}
I20260709 13:09:47.063836 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:47.064685 25942 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:46.879182 30345 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:47.011057 25390 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:47.065099 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.003136 30142 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:46.971180 30057 raft_consensus.cc:3055] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:47.066062 30142 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:47.066800 25390 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:09:47.015552 30347 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:47.016997 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.017911 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.015902 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:47.077535 25317 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:47.018736 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:47.039745 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:47.056797 30075 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:47.083966 25319 raft_consensus.cc:2393] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:47.065372 30345 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:47.066711 30057 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:47.016319 30154 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:47.011551 25394 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:09:47.004422 29854 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:47.067655 25239 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:47.008512 30306 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:47.069620 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.103004 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.070329 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.105582 25245 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
W20260709 13:09:47.073665 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 61200.3us
I20260709 13:09:46.889935 30006 raft_consensus.cc:697] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:47.076012 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2415) took 1158 ms. Trace:
I20260709 13:09:47.111277 25318 rpcz_store.cc:276] 0709 13:09:45.917279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.963200 (+ 45921us) service_pool.cc:224] Handling call
0709 13:09:47.075845 (+1112645us) spinlock_profiling.cc:243] Waited 47.7 ms on lock 0x7b5400274f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:47.075959 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":109725568}
I20260709 13:09:47.081024 25314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 808) took 1216 ms. Trace:
I20260709 13:09:47.112928 25314 rpcz_store.cc:276] 0709 13:09:45.864840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.996771 (+1131931us) service_pool.cc:224] Handling call
0709 13:09:47.080395 (+ 83624us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:47.080599 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:47.080635 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:47.080690 (+    55us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:47.080743 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:47.080813 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:47.080941 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:46.941030 30205 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:46.131665 25312 rpcz_store.cc:276] 0709 13:09:43.671537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.975171 (+2303634us) service_pool.cc:224] Handling call
0709 13:09:46.009059 (+ 33888us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:46.009373 (+   314us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:09:46.109635 (+100262us) log.cc:844] Serialized 14 byte log entry
0709 13:09:46.130827 (+ 21192us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:46.130879 (+    52us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:46.130933 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.131009 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:46.131131 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":127,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":4644,"thread_start_us":93892,"threads_started":2}
I20260709 13:09:47.082511 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.062340 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:47.085179 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 421611us
I20260709 13:09:47.085856 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.086556 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.119985 25390 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:47.120218 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.121897 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 1010) took 1060 ms. Trace:
I20260709 13:09:47.093235 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.095548 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.122274 25388 rpcz_store.cc:276] 0709 13:09:46.061906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.063471 (+  1565us) service_pool.cc:224] Handling call
0709 13:09:46.063863 (+   392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:46.689141 (+625278us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:47.121680 (+432539us) spinlock_profiling.cc:243] Waited 624 ms on lock 0x7b54002837d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:47.121841 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1434114432}
I20260709 13:09:47.122761 25394 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:47.122952 25237 raft_consensus.cc:2410] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 13:09:47.123469 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.104729 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.015681 29994 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:09:47.094813 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:46.216048 25315 rpcz_store.cc:276] 0709 13:09:43.448682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.881344 (+2432662us) service_pool.cc:224] Handling call
0709 13:09:45.881680 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:45.881923 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:09:46.121915 (+239992us) log.cc:844] Serialized 14 byte log entry
0709 13:09:46.206471 (+ 84556us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:46.206623 (+   152us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:46.206767 (+   144us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:46.207058 (+   291us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:46.208328 (+  1270us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3993,"prepare.queue_time_us":293802,"prepare.run_wall_time_us":37,"raft.queue_time_us":1481,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"thread_start_us":310119,"threads_started":2}
I20260709 13:09:47.075572 30302 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:47.117079 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 1040) took 1258 ms. Trace:
I20260709 13:09:47.127658 25941 rpcz_store.cc:276] 0709 13:09:45.859048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.900089 (+ 41041us) service_pool.cc:224] Handling call
0709 13:09:45.937540 (+ 37451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:46.105658 (+168118us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:47.116834 (+1011176us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b54006748d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:47.117010 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":320087808}
I20260709 13:09:47.120268 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:47.123620 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.128791 25942 raft_consensus.cc:2376] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14. Re-sending same reply.
I20260709 13:09:47.091442 25392 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
W20260709 13:09:47.094780 25320 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.704s	user 0.005s	sys 0.018s
I20260709 13:09:47.118700 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2574) took 1975 ms. Trace:
I20260709 13:09:47.126932 25388 raft_consensus.cc:2393] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:47.120827 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:47.132207 25320 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.006s	sys 0.018s
I20260709 13:09:47.132748 25391 raft_consensus.cc:2410] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 16 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 13:09:47.134102 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.134196 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:47.134629 25390 raft_consensus.cc:2410] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 16 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 13:09:47.084275 29964 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:47.130563 25386 rpcz_store.cc:276] 0709 13:09:45.143338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.143486 (+   148us) service_pool.cc:224] Handling call
0709 13:09:45.143840 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:45.514373 (+370533us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:45.514421 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:45.514488 (+    67us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:09:45.514544 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:47.118484 (+1603940us) spinlock_profiling.cc:243] Waited 370 ms on lock 0x7b5400283cd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:47.118526 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:47.118647 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":851424640}
I20260709 13:09:47.135357 25940 raft_consensus.cc:2393] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:47.136046 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 799) took 2269 ms. Trace:
I20260709 13:09:47.140214 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.155202 25392 raft_consensus.cc:2393] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 12: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:47.068372 30105 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 13:09:47.095670 30179 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:47.131160 25285 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:47.156520 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.144476 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.144893 29994 consensus_queue.cc:1243] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer 2329e428ebbf484fa3ea9452133df567 log is divergent from this leader: its last log entry 11.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:09:47.167137 30142 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:47.140527 30302 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:47.148626 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2466) took 1368 ms. Trace:
I20260709 13:09:47.099654 30057 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:47.129395 29894 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:47.154809 25320 rpcz_store.cc:276] 0709 13:09:44.867001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.161360 (+1294359us) service_pool.cc:224] Handling call
0709 13:09:46.161695 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:46.163126 (+  1431us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:46.163157 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:46.163640 (+   483us) log.cc:844] Serialized 26 byte log entry
0709 13:09:46.350373 (+186733us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:47.132631 (+782258us) log.cc:844] Serialized 14 byte log entry
0709 13:09:47.135679 (+  3048us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:47.135758 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:47.135833 (+    75us) raft_consensus.cc:1685] finished
0709 13:09:47.135846 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:47.135984 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6897,"prepare.queue_time_us":78452,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":15898,"raft.queue_time_us":5085,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":7430,"thread_start_us":925317,"threads_started":2,"wal-append.queue_time_us":451559}
I20260709 13:09:47.170746 25316 rpcz_store.cc:276] 0709 13:09:45.779740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.669632 (+889892us) service_pool.cc:224] Handling call
0709 13:09:46.669967 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:46.670180 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:09:46.670483 (+   303us) log.cc:844] Serialized 14 byte log entry
0709 13:09:47.148183 (+477700us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:47.148247 (+    64us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:09:47.148299 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:47.148372 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:47.148534 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":22033,"prepare.run_cpu_time_us":118,"prepare.run_wall_time_us":3818,"raft.queue_time_us":5529,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":7909,"thread_start_us":477312,"threads_started":1}
I20260709 13:09:47.084328 30347 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:47.182715 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:47.181551 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 50582.2us
I20260709 13:09:47.099807 30154 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:47.186980 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:47.193432 25285 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:47.117336 30306 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:47.192596 30234 consensus_meta.cc:287] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.577s	user 0.011s	sys 0.001s
W20260709 13:09:47.189067 30304 consensus_meta.cc:287] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.557s	user 0.012s	sys 0.000s
I20260709 13:09:47.190199 30057 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:47.190759 30280 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:47.190526 30217 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:47.157210 29964 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:47.215975 25317 pending_rounds.cc:85] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 8
I20260709 13:09:47.200419 30176 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:47.219831 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.221203 30347 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:47.222085 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:47.222811 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:47.230472 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:47.239583 25317 pending_rounds.cc:107] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260709 13:09:47.253836 25317 raft_consensus.cc:2889] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:47.218958 30306 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:47.253933 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:47.253377 25316 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
W20260709 13:09:47.224852 30315 consensus_meta.cc:287] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.547s	user 0.007s	sys 0.003s
W20260709 13:09:47.243165 29872 consensus_meta.cc:287] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.060s	user 0.008s	sys 0.005s
I20260709 13:09:47.224962 30345 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:47.243624 25386 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:47.244421 29964 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:47.246143 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:47.248521 30137 consensus_meta.cc:287] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.585s	user 0.011s	sys 0.000s
I20260709 13:09:47.253664 25939 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 8.
I20260709 13:09:47.225544 30245 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:47.211066 30304 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:47.231134 30280 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:47.273151 30280 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:47.273751 30154 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:47.274192 29994 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:47.274734 29994 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:47.276732 29994 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:47.263587 25320 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:47.277237 30062 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:47.256948 29966 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:47.268998 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2410) took 1579 ms. Trace:
I20260709 13:09:47.263106 29872 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:47.170441 30179 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:47.253345 30176 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:47.271937 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2413) took 1421 ms. Trace:
I20260709 13:09:47.272487 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.269014 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.281560 25313 rpcz_store.cc:276] 0709 13:09:45.689366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.215343 (+525977us) service_pool.cc:224] Handling call
0709 13:09:46.215647 (+   304us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:46.217081 (+  1434us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:46.217111 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:46.217593 (+   482us) log.cc:844] Serialized 26 byte log entry
0709 13:09:46.883269 (+665676us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:09:47.015064 (+131795us) log.cc:844] Serialized 14 byte log entry
0709 13:09:47.094061 (+ 78997us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:47.094171 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:47.267749 (+173578us) raft_consensus.cc:1685] finished
0709 13:09:47.268664 (+   915us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:47.268826 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":126708,"prepare.queue_time_us":131479,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":79264,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":54256,"thread_start_us":665223,"threads_started":1,"wal-append.queue_time_us":996995}
I20260709 13:09:47.318188 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:47.313161 25321 rpcz_store.cc:276] 0709 13:09:45.850314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.671849 (+821535us) service_pool.cc:224] Handling call
0709 13:09:46.672763 (+   914us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:46.673547 (+   784us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:46.673576 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:46.674564 (+   988us) log.cc:844] Serialized 26 byte log entry
0709 13:09:47.090941 (+416377us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:47.109194 (+ 18253us) log.cc:844] Serialized 14 byte log entry
0709 13:09:47.109420 (+   226us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:47.109491 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:47.271706 (+162215us) raft_consensus.cc:1685] finished
0709 13:09:47.271725 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:47.271876 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":24249,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":11826,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":296,"spinlock_wait_cycles":17111936,"thread_start_us":415754,"threads_started":1,"wal-append.queue_time_us":574309}
I20260709 13:09:47.323393 25393 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:47.241603 30217 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:47.324397 25212 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:47.330971 25940 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:47.260380 30315 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:47.333904 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.213920 30006 consensus_queue.cc:237] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:47.340209 25246 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:47.207815 29854 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:47.287528 30142 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:47.341471 30154 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:47.341956 25390 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:47.338066 25212 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:47.342582 30215 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:09:47.349586 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:47.370714 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 1042) took 1505 ms. Trace:
I20260709 13:09:47.320824 30176 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:47.371416 25938 rpcz_store.cc:276] 0709 13:09:45.865647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:45.903728 (+ 38081us) service_pool.cc:224] Handling call
0709 13:09:45.904011 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:46.005040 (+101029us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:46.005086 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:46.005174 (+    88us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:09:46.005230 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:47.370433 (+1365203us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b5400671e54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:47.370488 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:47.370650 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":231677696}
I20260709 13:09:47.375701 25392 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:47.272135 30137 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:47.381822 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:47.312292 29966 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:47.383721 30105 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:47.398686 30105 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:47.399204 30105 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:47.401741 30355 log.cc:927] Time spent T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 13:09:47.373523 30301 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:47.407156 25937 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:47.383410 29854 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:47.398154 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.377648 30302 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:47.428910 25358 leader_election.cc:336] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Timed out: RequestConsensusVote RPC to 127.23.140.68:39211 timed out after 1.745s (SENT)
I20260709 13:09:47.374038 30154 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:47.440007 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.374549 30142 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:47.459388 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:47.410058 30062 consensus_queue.cc:1059] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 13:09:47.479934 25391 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
W20260709 13:09:47.481518 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 116264us
I20260709 13:09:47.491541 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:47.492194 25313 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:47.494499 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:47.494635 25313 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:09:47.500622 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:47.501478 25313 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:47.506842 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:47.508951 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:47.527208 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:47.538121 29966 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:47.539243 30142 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:47.539389 25313 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
W20260709 13:09:47.542676 30168 consensus_meta.cc:287] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.723s	user 0.007s	sys 0.004s
I20260709 13:09:47.554500 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:47.557508 25316 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 13:09:47.563500 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:47.564221 25313 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:47.575105 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:47.575598 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:47.575951 25944 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
W20260709 13:09:47.577627 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:47.575403 25287 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:47.603472 25313 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:47.604303 25287 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:47.586176 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:47.605711 25315 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 8.
I20260709 13:09:47.588711 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:47.561837 30345 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:47.596817 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 201395us
I20260709 13:09:47.612025 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:47.606943 25316 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:09:47.615393 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:47.619604 25287 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:47.622632 25287 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:47.512226 30347 consensus_queue.cc:1059] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.073s
I20260709 13:09:47.632061 30306 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:47.634464 30304 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:47.635511 30097 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:47.648113 30154 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:47.587736 29872 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:47.616478 30089 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:47.615681 30217 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:47.621980 30038 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:47.618284 30315 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:47.684782 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:47.654985 30300 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:47.655426 30092 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:09:47.707305 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 198391us
I20260709 13:09:47.674451 30304 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:47.676218 30038 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:47.677001 30097 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:47.675923 30089 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:47.675236 30137 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:47.670917 30280 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:47.671475 30105 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:47.704411 30300 raft_consensus.cc:697] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:47.717572 30038 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:47.719910 30089 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:47.731374 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 120073us
I20260709 13:09:47.708874 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:47.746364 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:47.739629 30057 consensus_meta.cc:287] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.510s	user 0.011s	sys 0.000s
I20260709 13:09:47.718403 30304 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:47.753480 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:47.774181 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 165324us
W20260709 13:09:47.770074 30364 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.183s	user 0.000s	sys 0.001s
I20260709 13:09:47.755898 30097 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:09:47.792629 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:47.795838 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:47.797482 30363 log.cc:927] Time spent T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260709 13:09:47.799507 30307 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:47.819844 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:47.801772 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 65088.5us
I20260709 13:09:47.831461 25938 raft_consensus.cc:2368] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12.
W20260709 13:09:47.774708 30360 log.cc:927] Time spent T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 13:09:47.819381 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:47.768860 30305 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:47.852605 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:47.822232 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:47.857515 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:47.822849 25209 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:47.858683 25938 raft_consensus.cc:2410] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 13:09:47.823400 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:47.831676 30258 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:47.805095 30373 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:47.862530 30292 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:09:47.863479 25212 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:47.864334 30304 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:47.864588 30292 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:47.860906 30092 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:47.865253 30154 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:47.865677 30092 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:47.858234 25284 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:47.866936 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:47.892051 25941 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 8.
W20260709 13:09:47.824869 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:47.895335 25284 leader_election.cc:400] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12.
I20260709 13:09:47.896806 25284 leader_election.cc:403] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 13:09:47.858114 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:47.899874 25940 raft_consensus.cc:2410] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 13:09:47.902626 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.902761 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.903478 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.903704 25395 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:47.903931 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.904243 25391 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:47.908283 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.908499 25212 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:47.908982 25391 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 9.
I20260709 13:09:47.909060 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:47.892201 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 69240.2us
I20260709 13:09:47.925060 25359 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:47.865890 30154 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:47.927718 25359 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:47.823414 30179 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:47.811152 30205 consensus_meta.cc:287] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.674s	user 0.011s	sys 0.000s
I20260709 13:09:47.911717 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:47.901690 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:47.861275 30373 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:47.866377 30105 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:47.887372 30351 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.747s	user 0.006s	sys 0.000s
W20260709 13:09:47.921322 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:47.951527 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 128945us
I20260709 13:09:47.831465 30137 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:47.913295 25387 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:47.867153 30301 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:47.896970 30305 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:09:47.936174 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 313647us
W20260709 13:09:47.944368 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:47.946602 25391 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:47.809929 30370 consensus_queue.cc:1059] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.118s
I20260709 13:09:47.947523 25945 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
W20260709 13:09:47.950605 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 55471.3us
I20260709 13:09:47.812141 30280 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:47.865291 30057 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.071s
I20260709 13:09:47.810113 30378 consensus_queue.cc:1059] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.140s
I20260709 13:09:47.980458 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:47.975544 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 67961.7us
I20260709 13:09:47.807727 30176 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:47.975651 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:47.986202 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.806327 30379 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.146s
I20260709 13:09:47.989365 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.858781 30097 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:47.936985 30307 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:47.982834 25284 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:48.003202 25387 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:47.861003 30258 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.003999 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:48.007088 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:48.037021 25936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 1047) took 1049 ms. Trace:
I20260709 13:09:48.037822 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:48.037513 25936 rpcz_store.cc:276] 0709 13:09:46.987522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.988604 (+  1082us) service_pool.cc:224] Handling call
0709 13:09:46.988955 (+   351us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:46.989529 (+   574us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:09:46.989922 (+   393us) log.cc:844] Serialized 14 byte log entry
0709 13:09:47.370815 (+380893us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:47.375489 (+  4674us) log.cc:844] Serialized 52 byte log entry
0709 13:09:47.375699 (+   210us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:47.376413 (+   714us) log.cc:844] Serialized 14 byte log entry
0709 13:09:47.376648 (+   235us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:47.376734 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:48.036782 (+660048us) raft_consensus.cc:1685] finished
0709 13:09:48.036807 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:48.036959 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1331,"prepare.queue_time_us":2596,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":93,"raft.queue_time_us":16644,"raft.run_cpu_time_us":436,"raft.run_wall_time_us":73079,"spinlock_wait_cycles":9242240,"thread_start_us":379898,"threads_started":1,"wal-append.queue_time_us":509973}
I20260709 13:09:48.039072 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:48.039371 30258 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:48.040185 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:48.032159 30062 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.009s
I20260709 13:09:48.042421 30154 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:48.043176 30154 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12.
I20260709 13:09:48.043960 30154 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:09:47.817255 30300 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.045450 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:48.047466 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:48.047806 25387 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:48.048473 25242 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:48.001279 25212 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:48.013159 25237 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 13:09:48.014909 30382 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:09:48.015815 30176 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:48.017333 30370 raft_consensus.cc:3055] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:48.058110 30370 raft_consensus.cc:740] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:48.059111 30370 consensus_queue.cc:260] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.060585 30370 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:47.980855 30358 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:48.021365 30371 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:48.074932 30371 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.024178 25320 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 12: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:48.028990 30378 raft_consensus.cc:3055] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:09:48.077598 30378 raft_consensus.cc:740] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:48.078668 30378 consensus_queue.cc:260] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.079171 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:48.080287 30378 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:48.080879 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:47.992278 30038 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.085316 30176 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:48.086484 25240 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:09:48.054621 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 54399.1us
I20260709 13:09:47.990442 30301 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.087720 25212 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:48.089517 30301 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:48.090302 30061 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:48.091391 25212 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:48.061590 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:48.092482 30215 consensus_queue.cc:1059] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:09:47.982578 30305 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:48.053434 30307 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:09:48.069600 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 86859.8us
I20260709 13:09:48.069841 25242 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:47.985428 29872 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:48.062548 30062 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:48.091439 30105 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:48.092916 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:48.070485 29894 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:48.021037 30373 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:48.093796 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 1014) took 1298 ms. Trace:
I20260709 13:09:47.984664 30137 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.094411 30092 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:48.096154 30379 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:48.098588 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 2202) took 1125 ms. Trace:
I20260709 13:09:48.102161 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:48.109018 30102 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:48.106439 25284 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:48.108898 30062 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:48.110460 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:48.029358 30280 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:48.115962 30383 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:48.113919 25285 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:48.119328 30258 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:48.121982 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:48.086663 30038 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:48.134714 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:48.184027 25242 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 12: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:48.104020 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:48.134346 25239 rpcz_store.cc:276] 0709 13:09:46.972862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.977149 (+  4287us) service_pool.cc:224] Handling call
0709 13:09:48.098374 (+1121225us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x7b54002b2354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:48.098525 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":43386240}
I20260709 13:09:47.951190 30168 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:48.070868 30358 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:09:48.181586 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 67802.6us
I20260709 13:09:48.118227 25393 rpcz_store.cc:276] 0709 13:09:46.795741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.796052 (+   311us) service_pool.cc:224] Handling call
0709 13:09:46.796442 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:47.323840 (+527398us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:48.093484 (+769644us) spinlock_profiling.cc:243] Waited 527 ms on lock 0x7b54003d5ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:48.093715 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1211443584}
I20260709 13:09:48.194936 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:48.171834 29854 consensus_meta.cc:287] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.696s	user 0.011s	sys 0.006s
I20260709 13:09:48.179877 30062 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:48.112181 30105 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:48.117995 29928 consensus_queue.cc:1059] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.064s
I20260709 13:09:48.175871 30092 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:48.172034 30371 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:48.188396 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 157896us
I20260709 13:09:48.182205 30102 raft_consensus.cc:697] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:09:48.200863 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 216681us
W20260709 13:09:48.201752 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 95386.8us
I20260709 13:09:48.124784 29894 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.112751 30307 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.197755 30358 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:48.199736 25285 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:48.200984 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:48.217486 25393 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:48.196957 30168 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:48.209455 25287 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:48.218374 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:48.218955 25393 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:48.219190 25287 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:48.204891 25246 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:48.191679 30392 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:48.209936 25284 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:48.221346 25285 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:48.207228 30105 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:48.205001 30376 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:48.228940 25287 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:48.175690 30379 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.207659 30302 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:48.254354 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:48.193099 30258 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.214758 30242 raft_consensus.cc:3055] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:48.253719 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:48.265687 25284 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:48.202502 29900 consensus_queue.cc:1059] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.022s
I20260709 13:09:48.253784 30280 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:48.224630 29872 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:48.256228 30075 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:09:48.226233 30038 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:48.224493 30057 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.056s
I20260709 13:09:48.268922 30387 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:48.272104 30301 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:09:48.251699 29994 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 13:09:48.272162 25287 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:48.273206 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:48.213333 29854 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20260709 13:09:48.299994 25244 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:09:48.186141 30383 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:48.367969 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 95914.8us
I20260709 13:09:48.369143 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:48.307385 25393 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:48.221277 30371 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
I20260709 13:09:48.255749 30092 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:09:48.276808 30367 log.cc:927] Time spent T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.564s	user 0.001s	sys 0.001s
W20260709 13:09:48.341579 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:48.223613 30168 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:09:48.291663 30089 consensus_meta.cc:287] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.547s	user 0.020s	sys 0.016s
I20260709 13:09:48.272791 30176 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:09:48.351279 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1916) took 1172 ms. Trace:
I20260709 13:09:48.279337 30097 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:48.282891 30242 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:48.284667 30392 raft_consensus.cc:697] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:09:48.389324 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:48.264338 30376 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:48.309648 30038 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:48.346807 30387 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:48.279402 30302 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.407670 30397 raft_consensus.cc:3055] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:48.412681 25386 rpcz_store.cc:276] 0709 13:09:47.181804 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:47.181933 (+   129us) service_pool.cc:224] Handling call
0709 13:09:48.350340 (+1168407us) spinlock_profiling.cc:243] Waited 56.8 ms on lock 0x7b54003d4ad4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:48.351176 (+   836us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":130649472}
I20260709 13:09:48.411379 29900 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:09:48.392138 30075 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:48.396696 25287 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:48.308426 29872 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.409077 30280 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:09:48.393025 30396 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:48.445634 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:48.363390 30102 consensus_queue.cc:237] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:48.449962 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:48.452854 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 1051) took 1448 ms. Trace:
I20260709 13:09:48.454778 29928 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:48.462376 30345 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:48.469200 25139 catalog_manager.cc:5697] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 7 to 10, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 10 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:48.484704 30377 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:48.487113 29964 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:48.412415 30092 raft_consensus.cc:697] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:48.491802 25390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1915) took 1335 ms. Trace:
W20260709 13:09:48.439584 30384 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.261s	user 0.000s	sys 0.001s
I20260709 13:09:48.438956 30038 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:48.437896 30137 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:48.503543 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:48.411031 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 156748us
W20260709 13:09:48.511346 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 114826us
I20260709 13:09:48.511015 25937 rpcz_store.cc:276] 0709 13:09:47.004565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:47.005628 (+  1063us) service_pool.cc:224] Handling call
0709 13:09:47.079657 (+ 74029us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:47.410606 (+330949us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:48.452474 (+1041868us) spinlock_profiling.cc:243] Waited 327 ms on lock 0x7b54005d0054. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:48.452765 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":752603008}
I20260709 13:09:48.443030 29894 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:48.442924 30387 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:48.414060 30089 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:48.448110 30397 raft_consensus.cc:740] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:48.541694 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:48.409868 30395 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.096s
I20260709 13:09:48.558655 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:48.523401 25390 rpcz_store.cc:276] 0709 13:09:47.156108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:47.156272 (+   164us) service_pool.cc:224] Handling call
0709 13:09:48.491643 (+1335371us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b54001ec154. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:48.491742 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":416751488}
I20260709 13:09:48.501498 30057 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:48.516786 30388 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.306s	user 0.000s	sys 0.001s
I20260709 13:09:48.533353 25287 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:48.516763 30176 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:48.531311 29900 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:48.542686 25940 raft_consensus.cc:2410] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 12 because replica has last-logged OpId of term: 10 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 10.
W20260709 13:09:48.581589 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 48328.3us
I20260709 13:09:48.594389 25287 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:48.586522 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:48.597702 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:48.588466 25239 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:09:48.588470 25911 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:48.590132 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 324526us
I20260709 13:09:48.602620 25284 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:48.612278 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:48.620236 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:48.620433 25318 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:09:48.620992 25319 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:48.621714 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:48.622048 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:48.622412 25318 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:48.623421 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:48.624221 25318 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:48.628700 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:48.613303 25316 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:09:48.520576 29928 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.543247 30307 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:48.525969 30345 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.527580 29964 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:48.622689 25319 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:48.646939 25318 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:48.694262 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:48.694559 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:48.593247 30154 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:48.584029 30129 raft_consensus.cc:3055] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Stepping down as leader of term 13
W20260709 13:09:48.599934 30391 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.326s	user 0.000s	sys 0.001s
I20260709 13:09:48.580389 30280 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:48.600370 30390 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.318s	user 0.001s	sys 0.000s
I20260709 13:09:48.690651 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:48.691601 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:48.694867 25318 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:48.672953 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:48.695252 25319 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:09:48.694933 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:48.520182 30242 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.688035 30379 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:48.527874 30377 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:48.800889 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:48.721792 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:48.803262 25319 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:09:48.803398 25315 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:09:48.591336 30057 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.517350 30396 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.725366 25321 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 13:09:48.641717 30097 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:48.722080 25287 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:48.796444 25386 raft_consensus.cc:2410] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
W20260709 13:09:48.644271 30399 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.297s	user 0.000s	sys 0.001s
I20260709 13:09:48.688045 30258 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:48.798749 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 189596us
I20260709 13:09:48.801545 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:48.616590 30301 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:48.796932 25312 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
W20260709 13:09:48.773093 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 287122us
I20260709 13:09:48.804405 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:48.806280 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 217925us
I20260709 13:09:48.613852 30205 consensus_queue.cc:1059] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.173s
I20260709 13:09:48.804329 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:48.714428 30129 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:09:48.819437 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 97463.3us
I20260709 13:09:48.643260 29966 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.154s
I20260709 13:09:48.615604 29900 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.837973 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:48.815289 30398 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.403s	user 0.000s	sys 0.001s
I20260709 13:09:48.808861 30403 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:48.537750 30392 consensus_queue.cc:237] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.821965 30154 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:48.842296 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:48.843087 30345 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:48.844611 29900 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:48.807900 30383 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:48.799161 30280 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.833067 25390 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:48.818013 30097 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:48.694085 30092 consensus_queue.cc:237] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:48.834121 30301 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:48.856700 30395 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:48.849107 30402 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.355s	user 0.001s	sys 0.000s
I20260709 13:09:48.788156 30300 consensus_queue.cc:1059] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.072s
I20260709 13:09:48.870834 30383 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:48.875203 30401 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.290s	user 0.000s	sys 0.001s
I20260709 13:09:48.875066 30383 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:48.874600 30378 consensus_meta.cc:287] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.794s	user 0.011s	sys 0.010s
I20260709 13:09:48.866356 29966 raft_consensus.cc:3055] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:48.882119 30129 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:48.889698 30418 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:09:48.889973 30129 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:48.886214 29966 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:48.869021 30097 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:48.870821 30301 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:48.896744 30065 raft_consensus.cc:3055] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:48.871297 30403 raft_consensus.cc:697] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:48.871254 30137 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:48.786855 30397 consensus_queue.cc:260] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.876641 30345 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:48.883503 30383 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:48.877147 30302 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:48.890092 30396 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:48.877599 30395 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.913563 30403 consensus_queue.cc:237] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:48.914429 30362 log.cc:927] Time spent T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 1.319s	user 0.014s	sys 0.001s
I20260709 13:09:48.915906 30407 raft_consensus.cc:3055] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:09:48.916332 30407 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:48.917198 30407 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:48.918463 30407 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:09:48.918582 30358 consensus_meta.cc:287] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.686s	user 0.010s	sys 0.000s
I20260709 13:09:49.126379 30358 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.129952 30378 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:49.131099 30378 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:48.823256 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:49.133643 30400 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:49.134871 30280 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:49.134680 30400 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.135507 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:49.136608 25237 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:48.835826 25287 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:48.840687 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:48.820842 25318 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:09:48.849270 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:48.978174 25314 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 13:09:48.847321 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:48.989151 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 857) took 1034 ms. Trace:
I20260709 13:09:48.866647 25386 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:48.996308 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:48.997810 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2604) took 1069 ms. Trace:
I20260709 13:09:49.285337 25388 rpcz_store.cc:276] 0709 13:09:47.928678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:47.931124 (+  2446us) service_pool.cc:224] Handling call
0709 13:09:47.932639 (+  1515us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:47.933445 (+   806us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:09:48.118956 (+185511us) log.cc:844] Serialized 14 byte log entry
0709 13:09:48.927486 (+808530us) spinlock_profiling.cc:243] Waited 69.7 ms on lock 0x7b18003266e0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:09:48.996766 (+ 69280us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:48.996916 (+   150us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:48.997073 (+   157us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:48.997238 (+   165us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:48.997545 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2882,"prepare.queue_time_us":14992,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":4731,"raft.queue_time_us":63415,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":20757,"spinlock_wait_cycles":160417664,"thread_start_us":1058958,"threads_started":3,"wal-append.queue_time_us":481110}
I20260709 13:09:49.287822 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:48.999065 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2588) took 1987 ms. Trace:
I20260709 13:09:49.288401 25314 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:09:48.999336 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:49.000197 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:49.003243 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:48.984593 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:49.004956 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 814) took 1511 ms. Trace:
I20260709 13:09:49.003320 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:48.810889 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 135756us
I20260709 13:09:49.289938 25320 rpcz_store.cc:276] 0709 13:09:47.493168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:47.494297 (+  1129us) service_pool.cc:224] Handling call
0709 13:09:49.004496 (+1510199us) spinlock_profiling.cc:243] Waited 529 ms on lock 0x7b5400276354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:49.004756 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1216182528}
I20260709 13:09:49.295737 25318 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: [13.7-13.7]
W20260709 13:09:49.020960 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:49.027185 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:49.031122 25139 catalog_manager.cc:5697] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 10 to 12, now has a pending config: VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65), VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66), VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 12 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:49.050205 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2607) took 1096 ms. Trace:
I20260709 13:09:49.053321 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:49.056102 25394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2598) took 1153 ms. Trace:
I20260709 13:09:49.057410 25395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2605) took 1125 ms. Trace:
I20260709 13:09:48.885063 29872 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:48.894733 30061 consensus_meta.cc:287] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.802s	user 0.020s	sys 0.011s
I20260709 13:09:48.900264 30065 raft_consensus.cc:740] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:09:49.119220 30075 consensus_meta.cc:287] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.620s	user 0.001s	sys 0.011s
I20260709 13:09:49.119477 29928 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:49.073756 29961 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 13:09:48.919380 30370 consensus_meta.cc:287] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.858s	user 0.009s	sys 0.002s
I20260709 13:09:48.925745 30376 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:49.342242 25243 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:09:48.930141 30377 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:48.932263 30395 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:48.932830 30397 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:48.922873 30414 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
W20260709 13:09:48.932307 30415 log.cc:927] Time spent T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.332s	user 0.002s	sys 0.000s
W20260709 13:09:49.347289 25241 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.006s	sys 0.019s
W20260709 13:09:49.348045 25241 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.007s	sys 0.019s
I20260709 13:09:48.902710 30137 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.349749 25237 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:49.351953 30137 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:49.148106 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:48.907847 30345 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.357852 30057 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:49.194335 30400 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:49.210362 30142 raft_consensus.cc:3055] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:49.213769 30154 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:49.358870 30142 raft_consensus.cc:740] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:49.213773 30217 raft_consensus.cc:3055] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:49.359570 30217 raft_consensus.cc:740] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:49.360360 30142 consensus_queue.cc:260] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:49.360918 30217 consensus_queue.cc:260] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.362057 30142 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:49.362372 30217 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:49.214564 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 378662us
I20260709 13:09:49.309852 30075 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:49.229534 30431 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:49.366708 25313 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:49.298283 25394 rpcz_store.cc:276] 0709 13:09:47.902777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:47.905143 (+  2366us) service_pool.cc:224] Handling call
0709 13:09:47.905438 (+   295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:47.989425 (+ 83987us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:09:48.022106 (+ 32681us) log.cc:844] Serialized 14 byte log entry
0709 13:09:48.192748 (+170642us) log.cc:844] Serialized 14 byte log entry
0709 13:09:48.202429 (+  9681us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:48.202508 (+    79us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:48.202568 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:48.275127 (+ 72559us) spinlock_profiling.cc:243] Waited 6.78 ms on lock 0x7b18003ae6a0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:09:48.869441 (+594314us) spinlock_profiling.cc:243] Waited 4.8 ms on lock 0x7b18003ae6a0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc5522 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:09:49.055047 (+185606us) spinlock_profiling.cc:243] Waited 8.96 ms on lock 0x7b18003ae6a0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b932a6 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419
0709 13:09:49.055558 (+   511us) spinlock_profiling.cc:243] Waited 83.7 ms on lock 0x7b5400481554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:49.055634 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:49.055924 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6328,"prepare.queue_time_us":41172,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":82,"raft.queue_time_us":413,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":6830,"spinlock_wait_cycles":239769856,"thread_start_us":166651,"threads_started":1,"wal-append.queue_time_us":228934}
I20260709 13:09:49.367195 30431 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.368057 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:49.231715 30306 consensus_meta.cc:287] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.566s	user 0.022s	sys 0.000s
I20260709 13:09:49.232290 30242 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:49.232892 29966 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:49.235366 30358 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:49.236408 30089 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:49.238106 30105 consensus_meta.cc:287] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.974s	user 0.012s	sys 0.009s
I20260709 13:09:49.239979 30378 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:49.288652 25389 rpcz_store.cc:276] 0709 13:09:47.011136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:47.014230 (+  3094us) service_pool.cc:224] Handling call
0709 13:09:47.014679 (+   449us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:47.079970 (+ 65291us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:47.080015 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:47.090475 (+ 10460us) log.cc:844] Serialized 26 byte log entry
0709 13:09:47.187796 (+ 97321us) spinlock_profiling.cc:243] Waited 879 us on lock 0x7b18002fe240. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f29768520d1 00007f2976c6da0f
0709 13:09:47.389331 (+201535us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:09:47.389369 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:47.389440 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:48.864129 (+1474689us) spinlock_profiling.cc:243] Waited 1.82 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:09:48.998486 (+134357us) raft_consensus.cc:1685] finished
0709 13:09:48.998549 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:48.998862 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64741,"raft.queue_time_us":68914,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":4457,"spinlock_wait_cycles":6216320,"thread_start_us":201339,"threads_started":1,"wal-append.queue_time_us":402028}
I20260709 13:09:49.258963 25358 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:49.249068 30301 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:49.262956 25319 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:49.277107 25245 rpcz_store.cc:276] 0709 13:09:47.954925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:47.955052 (+   127us) service_pool.cc:224] Handling call
0709 13:09:48.098560 (+143508us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:48.099071 (+   511us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:48.099113 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:48.099892 (+   779us) log.cc:844] Serialized 52 byte log entry
0709 13:09:48.549870 (+449978us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:48.549929 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:48.550016 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:48.973642 (+423626us) raft_consensus.cc:1685] finished
0709 13:09:48.973728 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:48.988558 (+ 14830us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54002b2350. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:48.988872 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2475,"raft.queue_time_us":1992,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":5106,"spinlock_wait_cycles":329048960,"thread_start_us":449443,"threads_started":1,"wal-append.queue_time_us":768321}
I20260709 13:09:49.283802 25321 raft_consensus.cc:2410] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 12 because replica has last-logged OpId of term: 10 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 10.
I20260709 13:09:49.293468 25390 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:49.290262 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:49.290150 25240 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:49.298348 25395 rpcz_store.cc:276] 0709 13:09:47.931742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:47.931938 (+   196us) service_pool.cc:224] Handling call
0709 13:09:48.350583 (+418645us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:48.350801 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:48.350825 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:48.350880 (+    55us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:48.350916 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:48.350971 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:49.057033 (+706062us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x7b54003d4ad0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:49.057248 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":961589504}
I20260709 13:09:49.325610 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:49.326399 30065 consensus_queue.cc:260] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:49.327783 25910 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.000s	sys 0.016s
W20260709 13:09:49.331630 30038 consensus_meta.cc:287] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.793s	user 0.007s	sys 0.015s
I20260709 13:09:49.332293 25393 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:09:49.336701 25143 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39582 (request call id 308) took 36032 ms (client timeout 15000 ms). Trace:
W20260709 13:09:49.338109 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:49.297883 25392 rpcz_store.cc:276] 0709 13:09:47.953577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:47.953747 (+   170us) service_pool.cc:224] Handling call
0709 13:09:47.954275 (+   528us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:48.003054 (+ 48779us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:48.003094 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:48.003641 (+   547us) log.cc:844] Serialized 26 byte log entry
0709 13:09:48.215912 (+212271us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:48.215952 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:48.216022 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:49.049443 (+833421us) raft_consensus.cc:1685] finished
0709 13:09:49.049523 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:49.049933 (+   410us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33946,"raft.queue_time_us":51571,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":171,"thread_start_us":154353,"threads_started":1,"wal-append.queue_time_us":438889}
I20260709 13:09:49.355502 25139 catalog_manager.cc:5697] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 7 to 9, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 9 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:49.369691 25242 raft_consensus.cc:2393] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:09:49.360296 30305 consensus_meta.cc:287] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.244s	user 0.013s	sys 0.005s
W20260709 13:09:49.254263 30387 consensus_meta.cc:287] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.675s	user 0.011s	sys 0.004s
I20260709 13:09:49.380837 30306 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:09:49.387147 25910 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.000s	sys 0.017s
I20260709 13:09:49.338176 29964 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:49.396552 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:49.398983 25246 raft_consensus.cc:2410] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 13:09:49.402100 25238 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:49.403000 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:49.403255 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:49.404793 25314 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:49.407686 25391 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:49.405856 30369 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:49.417412 30089 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:09:49.417855 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 428978us
I20260709 13:09:49.418237 30089 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:49.419093 25358 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:49.420522 30436 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 13:09:49.425590 30358 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:49.425951 30431 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:49.426234 30358 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.401396 30387 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.428710 30358 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:49.429900 30376 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:49.431200 30345 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:49.431617 25312 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:49.433632 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:49.429792 25242 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:09:49.426684 30431 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:49.432163 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:49.386154 30105 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.397753 30038 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.416743 30301 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:49.434367 25389 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
W20260709 13:09:49.450306 25908 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.023s	sys 0.050s
I20260709 13:09:49.462155 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:49.408826 30369 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:49.468887 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:49.412264 30065 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:49.420938 30089 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:49.464663 29964 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:49.475859 29964 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.479056 29964 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:49.461491 25139 catalog_manager.cc:5697] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 10 to 12, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 12 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:49.482123 30038 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:49.483279 25315 raft_consensus.cc:2393] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:49.420972 25358 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:49.484731 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 63910.1us
I20260709 13:09:49.486069 29961 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:09:49.490724 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:49.492101 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:49.494673 25358 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:49.517903 29964 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:49.518536 29964 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:49.519079 29964 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:49.496610 25359 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:49.386279 25143 rpcz_store.cc:269] 0709 13:09:13.316189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:13.316358 (+   169us) service_pool.cc:224] Handling call
0709 13:09:49.336383 (+36020025us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:09:37.907285 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:09:38.109780 (+202495us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:09:38.120415 (+ 10635us) write_op.cc:435] Acquired schema lock
0709 13:09:38.121675 (+  1260us) tablet.cc:662] Decoding operations
0709 13:09:38.227697 (+106022us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:09:38.244252 (+ 16555us) write_op.cc:641] Partition lock acquired for write op
0709 13:09:38.245600 (+  1348us) tablet.cc:685] Acquiring locks for 14 operations
0709 13:09:38.341678 (+ 96078us) tablet.cc:701] Row locks acquired
0709 13:09:38.341742 (+    64us) write_op.cc:260] PREPARE: finished
0709 13:09:38.341968 (+   226us) write_op.cc:270] Start()
0709 13:09:38.342172 (+   204us) write_op.cc:276] Timestamp: P: 1783602578341890 usec, L: 0
0709 13:09:38.342221 (+    49us) op_driver.cc:348] REPLICATION: starting
0709 13:09:38.343177 (+   956us) log.cc:844] Serialized 5619 byte log entry
0709 13:09:38.728268 (+385091us) op_driver.cc:464] REPLICATION: finished
0709 13:09:38.973696 (+245428us) write_op.cc:301] APPLY: starting
0709 13:09:38.973820 (+   124us) tablet.cc:1367] starting BulkCheckPresence
0709 13:09:39.482738 (+508918us) tablet.cc:1370] finished BulkCheckPresence
0709 13:09:39.489770 (+  7032us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:09:39.966727 (+476957us) spinlock_profiling.cc:243] Waited 5.27 ms on lock 0x7b18004708c0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f29769f10ec 00007f2976a3dc0a 00007f2976a3d4f7 00007f2976a3d490 00007f29769f4f48 00007f297a7091bb 00007f297a7505d4 00007f297a5fc8da 00007f297a5ff3cb
0709 13:09:40.666229 (+699502us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:09:40.840436 (+174207us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=28,mrs_lookups=0
0709 13:09:40.850479 (+ 10043us) write_op.cc:312] APPLY: finished
0709 13:09:41.183039 (+332560us) log.cc:844] Serialized 129 byte log entry
0709 13:09:41.283723 (+100684us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:09:41.394144 (+110421us) write_op.cc:454] Released schema lock
0709 13:09:41.414403 (+ 20259us) write_op.cc:341] FINISH: Updating metrics
0709 13:09:41.639194 (+224791us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:09:41.644110 (+  4916us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":273374,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":160005,"apply.run_cpu_time_us":61421,"apply.run_wall_time_us":2610200,"cfile_cache_hit":54,"cfile_cache_hit_bytes":56376,"cfile_cache_miss":4,"cfile_cache_miss_bytes":6215,"cfile_init":2,"lbm_read_time_us":83066,"lbm_reads_10-100_ms":4,"lbm_reads_lt_1ms":8,"num_ops":14,"prepare.queue_time_us":4081141,"prepare.run_cpu_time_us":24799,"prepare.run_wall_time_us":576463,"raft.queue_time_us":336602,"raft.run_cpu_time_us":12943,"raft.run_wall_time_us":104604,"replication_time_us":385647,"spinlock_wait_cycles":12122880,"thread_start_us":820863,"threads_started":5,"wal-append.queue_time_us":227887,"wal-append.run_cpu_time_us":20458,"wal-append.run_wall_time_us":2429596}]]}
I20260709 13:09:49.497535 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:49.497723 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:49.498692 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:49.500329 30105 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:49.511363 25317 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 13:09:49.513552 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:49.514071 30129 consensus_meta.cc:287] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.623s	user 0.005s	sys 0.010s
W20260709 13:09:49.474689 25908 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.027s	sys 0.051s
I20260709 13:09:49.520984 30373 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:49.399164 30305 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.525750 30419 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:49.437191 30097 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:49.530215 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:49.530974 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:49.537808 29928 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:49.558674 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:49.480618 25139 catalog_manager.cc:5697] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 9 to 11, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 11 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:49.461817 30431 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:49.563144 30437 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:49.563293 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:49.563897 25287 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:49.564419 25392 raft_consensus.cc:2410] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 14 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 13:09:49.564107 30437 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.566645 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:49.569469 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:49.570421 25243 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:09:49.571309 30089 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:49.572623 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:49.573449 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:49.560940 25389 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:49.576529 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:49.575215 30431 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:49.579723 25359 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:49.569515 25911 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:49.551312 30373 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:49.585971 25319 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 13:09:49.578720 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:49.584012 25393 raft_consensus.cc:2393] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:49.588351 25316 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 13:09:49.588248 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:49.589216 25314 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:49.590231 30300 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:49.590304 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:49.591267 25312 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 13:09:49.591902 30345 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:09:49.592340 30345 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:49.592468 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:49.592760 30345 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:49.593263 25312 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:49.594255 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:49.595388 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:49.596900 30421 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:49.597517 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:49.597841 25389 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:60946
I20260709 13:09:49.598647 25392 raft_consensus.cc:2410] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 13:09:49.585819 25242 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:49.591238 25314 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 15: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:49.556449 30387 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:49.596024 25312 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:09:49.598685 25389 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:09:49.602486 25389 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:49.577980 29961 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:49.603701 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:49.602756 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:49.608107 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:49.612095 25315 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:49.599146 30373 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:49.621922 25395 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:49.626009 25316 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:49.597476 30421 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:49.627090 25312 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:49.629801 25321 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:49.620260 30419 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:09:49.638833 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:49.641258 30168 consensus_meta.cc:287] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.202s	user 0.013s	sys 0.009s
I20260709 13:09:49.641808 30168 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:49.643167 25238 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32994 (request call id 2262) took 2665 ms (client timeout 1620 ms). Trace:
W20260709 13:09:49.643728 25238 rpcz_store.cc:269] 0709 13:09:46.977830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:46.980912 (+  3082us) service_pool.cc:224] Handling call
0709 13:09:49.642792 (+2661880us) spinlock_profiling.cc:243] Waited 2.36 s on lock 0x7b5400192b54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:49.643054 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5437052160}
I20260709 13:09:49.646648 25393 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:49.649958 25287 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:49.651140 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 823) took 1031 ms. Trace:
I20260709 13:09:49.651707 25317 rpcz_store.cc:276] 0709 13:09:48.620173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:48.976344 (+356171us) service_pool.cc:224] Handling call
0709 13:09:49.510756 (+534412us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:49.548955 (+ 38199us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:49.650842 (+101887us) spinlock_profiling.cc:243] Waited 533 ms on lock 0x7b54000f9450. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:49.651062 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1226996480}
I20260709 13:09:49.652599 25287 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:49.654052 30403 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.031s
I20260709 13:09:49.655946 30387 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:49.656431 30387 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:49.656837 30387 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:49.630036 30421 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:49.635177 25139 catalog_manager.cc:5697] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:49.661942 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:49.664953 25386 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 12: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:49.625624 30437 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:49.674724 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 95135.6us
I20260709 13:09:49.644805 30168 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:49.677111 25387 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:49.608011 29928 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.677042 25139 catalog_manager.cc:5697] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:49.691865 25389 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:49.692566 30062 consensus_meta.cc:287] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.484s	user 0.005s	sys 0.014s
I20260709 13:09:49.637921 30419 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:49.699697 25246 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:49.684000 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:49.714846 30142 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:49.717900 30097 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:49.718916 25238 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
W20260709 13:09:49.730508 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 77994.5us
I20260709 13:09:49.731012 29928 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:49.738946 25390 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
W20260709 13:09:49.741468 25387 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41830 (request call id 1017) took 1638 ms (client timeout 1955 ms). Trace:
W20260709 13:09:49.741992 25387 rpcz_store.cc:269] 0709 13:09:48.103354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:48.103813 (+   459us) service_pool.cc:224] Handling call
0709 13:09:49.732326 (+1628513us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54001e4954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:49.741371 (+  9045us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3221886720}
I20260709 13:09:49.693526 30369 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:49.727380 30419 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:49.716845 29964 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:49.750124 25390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1938) took 1101 ms. Trace:
W20260709 13:09:49.765658 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:49.766585 25389 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:49.768002 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:49.754793 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 186478us
I20260709 13:09:49.765815 25911 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:49.767213 25390 rpcz_store.cc:276] 0709 13:09:48.638120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:48.998465 (+360345us) service_pool.cc:224] Handling call
0709 13:09:49.739832 (+741367us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:49.771113 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:49.771919 25243 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:49.759377 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2489) took 1040 ms. Trace:
I20260709 13:09:49.769833 25359 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:49.775156 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:49.775772 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:60946 (request call id 1927) took 1154 ms. Trace:
I20260709 13:09:49.737556 30154 consensus_queue.cc:1059] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 13:09:49.759912 25139 catalog_manager.cc:5697] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:49.792490 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:49.785821 25391 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:49.795993 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:49.759052 29928 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:49.804602 25389 rpcz_store.cc:276] 0709 13:09:48.630611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:49.585397 (+954786us) service_pool.cc:224] Handling call
0709 13:09:49.767774 (+182377us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:49.737533 30062 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.811612 25287 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:49.812127 25395 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
W20260709 13:09:49.815034 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 45275.9us
I20260709 13:09:49.760021 30396 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:09:49.819902 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:49.820469 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:49.821337 25395 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 12: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:49.822909 25392 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 9. Current term is 10. Ops: []
I20260709 13:09:49.796092 30305 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:49.825040 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:49.828778 25240 raft_consensus.cc:3055] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:49.829113 25240 raft_consensus.cc:740] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:49.829308 30242 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:49.834834 30242 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.829466 30176 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:49.838238 30242 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:49.769197 30142 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.818790 25139 catalog_manager.cc:5697] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:49.809451 29928 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:49.842126 29928 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:49.843912 30062 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:49.845770 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 209230us
I20260709 13:09:49.847172 29964 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:49.827886 30437 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:49.846994 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2492) took 1023 ms. Trace:
W20260709 13:09:49.857574 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 83720.9us
I20260709 13:09:49.828063 30154 consensus_queue.cc:1243] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 11.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:09:49.860260 25390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2617) took 1022 ms. Trace:
I20260709 13:09:49.883785 25390 rpcz_store.cc:276] 0709 13:09:48.837654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:49.770746 (+933092us) service_pool.cc:224] Handling call
0709 13:09:49.796797 (+ 26051us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:49.840049 (+ 43252us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:49.840082 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:49.859634 (+ 19552us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:09:49.859724 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:49.859842 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:49.860109 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:49.860282 30431 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:49.862027 25911 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:49.863539 29966 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:49.846957 30376 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:49.869710 25243 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:49.776311 25320 rpcz_store.cc:276] 0709 13:09:48.718683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:49.295247 (+576564us) service_pool.cc:224] Handling call
0709 13:09:49.295765 (+   518us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:09:49.296405 (+   640us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:49.296438 (+    33us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:09:49.308998 (+ 12560us) log.cc:844] Serialized 78 byte log entry
0709 13:09:49.601683 (+292685us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:09:49.752811 (+151128us) log.cc:844] Serialized 14 byte log entry
0709 13:09:49.754560 (+  1749us) log.cc:844] Serialized 14 byte log entry
0709 13:09:49.757476 (+  2916us) log.cc:844] Serialized 14 byte log entry
0709 13:09:49.758840 (+  1364us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:49.758965 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:49.759033 (+    68us) raft_consensus.cc:1685] finished
0709 13:09:49.759045 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:49.759292 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6884,"raft.queue_time_us":11289,"raft.run_cpu_time_us":575,"raft.run_wall_time_us":67137,"spinlock_wait_cycles":9386880,"thread_start_us":383043,"threads_started":2,"wal-append.queue_time_us":325075}
I20260709 13:09:49.790079 30097 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 13:09:49.856501 30373 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.881670 30437 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:49.882009 25908 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:49.888027 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 61250.3us
I20260709 13:09:49.888892 25287 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:49.922339 30397 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:49.922971 30217 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:49.923643 30217 raft_consensus.cc:697] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:49.924834 30217 consensus_queue.cc:237] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:49.926848 30097 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:49.897991 30376 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:49.928732 30376 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:49.929248 30437 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:49.931236 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:49.889001 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 105510us
I20260709 13:09:49.932459 25358 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:49.932801 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:49.933246 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:49.933945 25945 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:09:49.936071 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:49.936794 25941 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:49.942391 25358 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:49.943137 30431 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:49.943416 30168 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:09:49.943890 30168 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:49.943943 30431 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:09:49.944317 30168 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:49.879287 25318 rpcz_store.cc:276] 0709 13:09:48.823948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:49.329598 (+505650us) service_pool.cc:224] Handling call
0709 13:09:49.330025 (+   427us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:49.330552 (+   527us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:49.330576 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:49.331232 (+   656us) log.cc:844] Serialized 26 byte log entry
0709 13:09:49.622042 (+290810us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:49.622084 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:49.622155 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:49.661078 (+ 38923us) spinlock_profiling.cc:243] Waited 38.6 ms on lock 0x7b4800271ed8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297bbdbaa1 00007f297bbd2b37 00007f297bbd29ca 00007f297bcca26f 00007f297bcc9f37 00007f297bcde55c 00007f297bcde508 00007f297bcde488 00007f297bcde443 00007f297bcdd6ec 00007f297ac07a96
0709 13:09:49.846692 (+185614us) raft_consensus.cc:1685] finished
0709 13:09:49.846712 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:49.846934 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1366,"raft.run_cpu_time_us":2139,"raft.run_wall_time_us":329390,"spinlock_wait_cycles":88850816,"thread_start_us":290315,"threads_started":1,"wal-append.queue_time_us":367822}
I20260709 13:09:49.946188 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:49.947389 25389 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:49.923249 25287 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:49.949946 25287 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:49.950788 30038 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:09:49.951442 30038 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:09:49.954981 25287 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:49.936260 30403 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:49.961426 30403 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:49.961913 30403 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:49.962435 25240 consensus_queue.cc:260] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:49.962810 30387 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:49.963598 25240 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:49.922477 30065 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:49.965065 30065 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.966007 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:49.967278 30065 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:49.956492 30373 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:49.970595 25246 raft_consensus.cc:2410] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 13:09:49.899411 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2612) took 1058 ms. Trace:
I20260709 13:09:49.970995 25392 rpcz_store.cc:276] 0709 13:09:48.832593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:49.680510 (+847917us) service_pool.cc:224] Handling call
0709 13:09:49.815309 (+134799us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:49.825661 (+ 10352us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:49.891200 (+ 65539us) spinlock_profiling.cc:243] Waited 67.2 ms on lock 0x7b5400120cd0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:49.891352 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":154541952}
I20260709 13:09:49.973894 25392 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 11. Current term is 12. Ops: []
W20260709 13:09:49.909601 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:49.923213 30397 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:49.924685 25139 catalog_manager.cc:5697] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:49.959447 25209 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:49.934327 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 72471.2us
I20260709 13:09:49.984529 25941 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:49.977488 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2615) took 1144 ms. Trace:
I20260709 13:09:49.993884 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:09:49.994035 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 61871.7us
I20260709 13:09:49.987468 25139 catalog_manager.cc:5697] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 9 to 12, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 12 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:49.987470 25386 rpcz_store.cc:276] 0709 13:09:48.833614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:49.708455 (+874841us) service_pool.cc:224] Handling call
0709 13:09:49.709065 (+   610us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:49.715583 (+  6518us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:49.715631 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:49.716330 (+   699us) log.cc:844] Serialized 26 byte log entry
0709 13:09:49.812662 (+ 96332us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:09:49.812695 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:49.812741 (+    46us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:49.977271 (+164530us) raft_consensus.cc:1685] finished
0709 13:09:49.977292 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:49.977431 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4889,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":160,"thread_start_us":95905,"threads_started":1,"wal-append.queue_time_us":240947}
I20260709 13:09:50.005697 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2619) took 1027 ms. Trace:
I20260709 13:09:50.009400 25391 rpcz_store.cc:276] 0709 13:09:48.978823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:49.864955 (+886132us) service_pool.cc:224] Handling call
0709 13:09:49.880959 (+ 16004us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:50.004810 (+123851us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:50.004859 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:50.004924 (+    65us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:50.004988 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:50.005084 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:50.005611 (+   527us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:09:50.017455 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:49.887179 30345 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.022779 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:50.023720 25239 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:09:50.024433 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:50.035519 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:50.019384 30062 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:50.000638 30373 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:50.048877 25139 catalog_manager.cc:5697] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:50.001297 30421 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:50.059233 30438 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:09:50.060199 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 114100us
I20260709 13:09:50.063753 30437 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.067044 30345 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:50.068223 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 113331us
I20260709 13:09:50.041858 30387 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:50.016780 30242 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.030s
I20260709 13:09:50.019821 30419 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.047279 30062 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:09:50.087193 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 188968us
I20260709 13:09:50.038918 30038 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:09:50.090731 25287 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:50.084338 25358 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:50.053659 30373 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:50.098702 30242 consensus_queue.cc:1059] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:09:50.101480 30419 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:50.113801 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:50.123821 25387 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:50.118017 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2618) took 1264 ms. Trace:
I20260709 13:09:50.124671 25389 rpcz_store.cc:276] 0709 13:09:48.853855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:49.820065 (+966210us) service_pool.cc:224] Handling call
0709 13:09:49.873378 (+ 53313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:49.947840 (+ 74462us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:50.117769 (+169929us) spinlock_profiling.cc:243] Waited 73.8 ms on lock 0x7b5400145d54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:50.117953 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":169639296}
I20260709 13:09:50.070020 30142 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:50.134784 25393 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:50.089865 25391 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:09:50.072121 25386 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:50.121766 30062 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:50.124650 30373 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:50.127775 30387 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:50.156813 30421 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:50.145642 30438 raft_consensus.cc:3055] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:50.156673 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:50.164316 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:50.161957 25239 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:50.137593 30396 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:50.162472 25139 catalog_manager.cc:5697] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 7 to 11, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 11 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:50.150746 30419 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:09:50.150765 30176 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:09:50.169433 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 88112.5us
W20260709 13:09:50.171525 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 99720.9us
I20260709 13:09:50.175146 25359 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:50.170166 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:50.163674 30438 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:50.182039 25941 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:09:50.182165 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 300290us
I20260709 13:09:50.183096 25287 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:50.168418 30419 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:50.186883 25240 raft_consensus.cc:2410] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 13:09:50.188838 25238 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 12. Ops: []
I20260709 13:09:50.193612 25287 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:50.168459 30387 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.198405 25908 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:50.184067 30397 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:50.200053 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:50.174801 30396 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:50.186790 30421 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:50.184751 30447 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:50.190248 30419 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:50.209913 25394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 1941) took 1490 ms. Trace:
I20260709 13:09:50.212725 25394 rpcz_store.cc:276] 0709 13:09:48.719771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:49.610639 (+890868us) service_pool.cc:224] Handling call
0709 13:09:49.663135 (+ 52496us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:49.753591 (+ 90456us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:49.753632 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:49.785441 (+ 31809us) log.cc:844] Serialized 52 byte log entry
0709 13:09:49.977970 (+192529us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:50.207185 (+229215us) log.cc:844] Serialized 14 byte log entry
0709 13:09:50.208023 (+   838us) log.cc:844] Serialized 14 byte log entry
0709 13:09:50.209549 (+  1526us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:50.209626 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:50.209672 (+    46us) raft_consensus.cc:1685] finished
0709 13:09:50.209683 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:50.209836 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1180,"raft.queue_time_us":12412,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":23537,"thread_start_us":330981,"threads_started":2,"wal-append.queue_time_us":301778}
I20260709 13:09:50.215042 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:50.199453 30306 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:50.224187 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:50.224296 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:50.225493 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:50.226143 25936 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:50.229341 25937 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:09:50.239984 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:50.243882 25941 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:50.207398 30038 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:50.255817 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:50.259078 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:50.219116 30176 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:50.269271 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:50.266297 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 168329us
I20260709 13:09:50.267681 30075 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:50.271577 30446 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:50.271898 30075 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.273710 30396 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:50.274225 30075 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:50.274325 30437 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:50.233690 30168 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:50.270256 25936 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:50.275576 30217 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:50.276378 30217 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:50.276855 30168 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:50.245990 30306 raft_consensus.cc:697] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:50.244946 30421 raft_consensus.cc:697] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:50.289464 25937 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
W20260709 13:09:50.286765 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 92877.7us
I20260709 13:09:50.259811 30397 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:50.297757 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:50.294684 25911 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:50.301503 25908 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:50.302750 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 112703us
I20260709 13:09:50.285410 30038 raft_consensus.cc:697] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:50.246912 30403 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.310344 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:50.284827 30376 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:50.316345 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:50.316908 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:50.299290 29928 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.336612 30387 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:50.337523 25388 raft_consensus.cc:2410] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 13:09:50.338057 29928 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:50.339183 30454 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260709 13:09:50.339041 30306 consensus_queue.cc:237] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:50.339980 25388 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1931) took 1708 ms (client timeout 1806 ms). Trace:
W20260709 13:09:50.340381 25388 rpcz_store.cc:269] 0709 13:09:48.631948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:49.287519 (+655571us) service_pool.cc:224] Handling call
0709 13:09:50.339776 (+1052257us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54001e4454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:50.339924 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2411816192}
I20260709 13:09:50.329891 30242 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:50.330310 30456 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:09:50.287760 30176 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:50.255116 30447 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.364289 30217 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:50.366925 30376 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:50.367434 30403 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:50.344350 30242 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:50.368361 30242 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:50.350533 30154 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:50.373709 30154 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.376313 30154 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:50.377761 30453 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260709 13:09:50.379559 30038 consensus_queue.cc:237] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:50.336097 30455 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 13:09:50.343601 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:50.352768 25390 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 14. Ops: []
W20260709 13:09:50.353921 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 62881.3us
I20260709 13:09:50.354765 30421 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:50.357401 30450 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
I20260709 13:09:50.362653 30176 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:50.265331 30438 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.405856 30438 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:50.406153 25908 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:50.408221 30345 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:09:50.408804 30345 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:50.409268 30345 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:09:50.409725 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2616) took 1575 ms. Trace:
I20260709 13:09:50.410177 25390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2620) took 1430 ms. Trace:
I20260709 13:09:50.410111 25387 rpcz_store.cc:276] 0709 13:09:48.834097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:49.743137 (+909040us) service_pool.cc:224] Handling call
0709 13:09:49.743539 (+   402us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:50.135058 (+391519us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:50.409452 (+274394us) spinlock_profiling.cc:243] Waited 376 ms on lock 0x7b54003237d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:50.409662 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":863887232}
I20260709 13:09:50.389803 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:50.411396 25936 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 13:09:50.390828 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:50.391868 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:50.413347 30456 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:09:50.413807 25944 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 14: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:50.415508 25245 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:09:50.393267 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2611) took 1561 ms. Trace:
I20260709 13:09:50.393347 30301 raft_consensus.cc:3055] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:50.417477 25393 rpcz_store.cc:276] 0709 13:09:48.832300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:49.647920 (+815620us) service_pool.cc:224] Handling call
0709 13:09:49.648347 (+   427us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:50.135275 (+486928us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:50.392965 (+257690us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b5400234954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:50.393200 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1096318720}
I20260709 13:09:50.396816 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:50.397466 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 54298.6us
I20260709 13:09:50.380515 30419 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:50.402480 25242 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:50.410564 25390 rpcz_store.cc:276] 0709 13:09:48.979347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:49.885192 (+905845us) service_pool.cc:224] Handling call
0709 13:09:49.900781 (+ 15589us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:50.402523 (+501742us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:50.409930 (+  7407us) spinlock_profiling.cc:243] Waited 452 ms on lock 0x7b54001e3f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:50.410119 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1039039360}
I20260709 13:09:50.425402 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:50.434217 25245 raft_consensus.cc:2410] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:09:50.432332 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:50.439731 25242 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:09:50.447853 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:50.449401 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:50.421602 30301 raft_consensus.cc:740] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:50.453876 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:50.454553 25241 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:50.450213 25285 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:50.457626 25908 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:50.465644 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:50.469251 25359 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:50.470961 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:50.471848 25389 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:50.473508 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:50.475206 25240 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:09:50.477558 25287 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:50.496932 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 59956.4us
I20260709 13:09:50.501230 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:50.502486 25390 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:50.438563 30397 raft_consensus.cc:697] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:50.510419 25387 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 12. Ops: []
W20260709 13:09:50.514006 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 157566us
W20260709 13:09:50.515357 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 225182us
I20260709 13:09:50.517165 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:50.517601 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 145152us
I20260709 13:09:50.517880 25389 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:50.455291 30097 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:50.521971 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 44550us
I20260709 13:09:50.527910 25287 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:50.522250 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:50.529234 25395 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:50.518350 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:50.524060 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:50.533780 25388 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:09:50.474875 30456 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:50.555261 25908 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:50.437991 30447 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:50.551772 25386 raft_consensus.cc:2410] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:09:50.512804 29964 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:09:50.455108 30154 consensus_queue.cc:1059] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
I20260709 13:09:50.562758 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:50.563750 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:50.516901 30465 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
W20260709 13:09:50.574208 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:50.558332 30396 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:50.554793 30456 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:50.589056 25941 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 15: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:50.597535 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:50.574796 30447 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:50.593258 30306 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:50.617782 30306 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:50.593468 30376 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.015s
I20260709 13:09:50.608201 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:50.590078 29964 raft_consensus.cc:697] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:50.610818 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:50.622463 25393 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:50.623386 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:50.623148 29964 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:50.627147 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:50.627712 25386 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:50.555291 30437 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 13:09:50.608971 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:50.571581 25944 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:45428
I20260709 13:09:50.641525 25944 raft_consensus.cc:606] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:50.642222 25944 raft_consensus.cc:614] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:09:50.552246 30097 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.614130 30461 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:50.614295 30460 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:50.614564 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:50.614645 25284 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:50.615774 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:50.611598 25389 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:09:50.582721 30154 consensus_queue.cc:1243] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 11.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260709 13:09:50.639770 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 123765us
I20260709 13:09:50.648032 25908 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:50.592057 30456 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:50.671424 25395 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:50.678787 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:50.680603 25391 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:50.662962 25911 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:50.592491 30062 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:50.690448 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:50.699837 25394 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:50.621035 30306 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:50.639032 30458 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:50.630879 30396 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:50.664614 30460 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:50.705637 25911 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:50.683302 25285 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:50.710526 25246 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:50.709961 30473 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:50.714417 30464 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:09:50.715191 30464 raft_consensus.cc:697] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:50.716292 30464 consensus_queue.cc:237] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.719101 30462 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:50.719575 25241 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:09:50.719648 30462 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:50.720372 30097 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:50.720501 30462 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:50.723031 25358 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:50.707630 25209 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:50.768658 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 55234.5us
I20260709 13:09:50.772104 29964 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:50.729934 29928 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:50.777788 29928 raft_consensus.cc:697] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:09:50.747650 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 219858us
I20260709 13:09:50.708451 30458 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:50.784405 25393 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:09:50.711323 30460 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:09:50.778600 25909 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 2.184s (SENT) [suppressed 17 similar messages]
W20260709 13:09:50.789647 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 152893us
I20260709 13:09:50.731005 30097 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:50.796605 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:50.698912 30447 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.684608 30397 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.11, Last appended by leader: 11, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.710889 30142 consensus_queue.cc:1059] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.085s
W20260709 13:09:50.802348 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 118767us
W20260709 13:09:50.803092 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 369226us
I20260709 13:09:50.805433 25284 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:50.803663 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 103138us
I20260709 13:09:50.783631 30419 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:50.784081 30458 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:50.815248 25287 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:50.780961 30396 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:50.817166 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 93720.1us
I20260709 13:09:50.800333 30097 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:50.797473 30469 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:50.814792 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:09:50.835299 25287 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:50.828930 29928 consensus_queue.cc:237] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:50.853597 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 48301.4us
I20260709 13:09:50.822510 30345 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.800598 30242 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.823756 30387 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:50.824571 30097 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:50.854730 30473 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:50.875681 25287 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:50.875921 25390 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:50.877914 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:50.872828 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 66924.5us
W20260709 13:09:50.887445 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 108854us
I20260709 13:09:50.880604 25937 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 9: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:50.861730 30461 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:50.875404 30473 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:50.908191 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:50.909332 25315 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 12: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:09:50.911407 30475 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
W20260709 13:09:50.913301 30467 log.cc:927] Time spent T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.096s	user 0.000s	sys 0.003s
W20260709 13:09:50.914474 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 100149us
I20260709 13:09:50.915143 25358 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:50.915807 30419 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:50.916298 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:50.917114 30419 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:50.908522 25908 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:50.917729 30419 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:50.918269 30419 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:50.920120 30396 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:50.921301 30464 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:50.921734 30478 consensus_queue.cc:1059] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:09:50.922896 30478 raft_consensus.cc:3055] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:50.923111 30447 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:50.923249 30478 raft_consensus.cc:740] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:50.923543 30473 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:50.923732 30447 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:50.924470 30438 consensus_meta.cc:287] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.518s	user 0.009s	sys 0.000s
I20260709 13:09:50.924292 30447 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:50.927543 30447 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:50.924284 30478 consensus_queue.cc:260] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.928787 30478 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:50.896513 30075 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:50.908463 25318 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:50.931450 30464 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:50.931990 30464 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.935473 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:50.890074 30472 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:09:50.937672 30464 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:50.938341 30345 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:50.938524 30464 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:50.940083 30464 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.945588 25944 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:50.946919 30464 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:50.959930 30473 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.961488 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:50.905263 30373 consensus_meta.cc:287] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.736s	user 0.012s	sys 0.000s
I20260709 13:09:50.947232 30242 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:50.947321 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:50.948977 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:50.949766 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:50.962205 30473 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:50.990865 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:09:50.992017 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:50.953132 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:50.992849 25394 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:50.993880 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:50.994498 25386 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:50.994313 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:50.995457 25938 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:50.953580 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:50.996881 25940 raft_consensus.cc:2410] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 12 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 13:09:50.994218 25911 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:50.997704 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:50.998140 30478 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:50.998579 30478 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:50.998934 30478 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:50.954092 25943 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:09:51.000437 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:50.962684 25318 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:09:51.001359 25943 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:50.954840 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:50.955370 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:50.962765 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:51.005870 25319 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:09:51.006031 25939 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 15: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:51.006812 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:50.956446 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:51.008123 25316 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:50.957844 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:51.008420 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 153862us
I20260709 13:09:50.890769 30038 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:09:51.007762 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:50.998157 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:50.945041 25392 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:51.006763 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:50.962292 25312 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 9: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:50.990727 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:51.007728 25319 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:51.009191 25284 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:51.009315 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:51.017939 25318 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:50.949011 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:50.895869 29964 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:50.885182 30470 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:51.011420 25358 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:50.962998 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:50.976576 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:51.033778 25393 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 9: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:09:50.983474 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:50.987152 25209 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:50.987344 25944 raft_consensus.cc:2393] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:51.024551 25911 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:50.962544 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:50.879469 30176 consensus_queue.cc:1059] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20260709 13:09:51.014776 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:51.022770 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2301) took 1008 ms. Trace:
I20260709 13:09:51.023428 25940 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 15: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:09:51.025170 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 206759us
I20260709 13:09:51.037034 25313 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:51.031348 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:50.978063 25359 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:51.035871 25321 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:51.044093 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:51.044425 30447 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:51.045085 30447 raft_consensus.cc:697] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:51.045604 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:51.046250 25315 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:51.046085 30447 consensus_queue.cc:237] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:51.046723 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.047122 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:51.047578 25245 raft_consensus.cc:2410] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 13:09:51.047720 25315 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:51.048439 25358 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:51.048700 25238 raft_consensus.cc:3055] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:09:51.049170 25238 raft_consensus.cc:740] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:51.037071 25246 rpcz_store.cc:276] 0709 13:09:50.014710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:50.014902 (+   192us) service_pool.cc:224] Handling call
0709 13:09:50.036859 (+ 21957us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:50.711052 (+674193us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:51.022455 (+311403us) spinlock_profiling.cc:243] Waited 673 ms on lock 0x7b54003ec454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:51.022695 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1548787456}
I20260709 13:09:51.049737 30142 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:51.050164 30142 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:51.049949 25238 consensus_queue.cc:260] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:51.050885 30446 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:51.014096 30038 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:51.051848 30038 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:51.053721 30376 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:51.055991 30456 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.056361 30462 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.967762 30419 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:50.963444 30373 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.059571 30419 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:51.059733 30446 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:51.060114 30373 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:51.050557 30142 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:51.038265 30470 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:51.061679 30470 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:51.063151 30464 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:51.063710 30464 raft_consensus.cc:697] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:51.064659 30464 consensus_queue.cc:237] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.065596 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:51.044093 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 62733us
W20260709 13:09:51.044235 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 66272.6us
I20260709 13:09:51.051731 30305 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:51.051378 25358 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:51.044203 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.051024 25238 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:51.070597 25318 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:09:51.071884 25312 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:09:51.072186 25387 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:51.072304 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:51.073519 25938 raft_consensus.cc:2410] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 12 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 13:09:51.076133 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:51.077174 25318 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:51.078560 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:51.079330 25312 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:09:51.081984 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:51.082078 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:50.962680 30062 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:51.082943 25393 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:09:51.088214 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:51.071692 30075 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:51.091037 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.091962 25243 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:51.092419 25358 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:50.973737 30461 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.101234 30461 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:51.101409 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.103791 30456 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:51.105964 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:51.075093 30306 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:51.106612 25321 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:09:51.113444 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.114245 25393 raft_consensus.cc:1275] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:09:51.115447 25386 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:51.115558 25943 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:51.087885 30431 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:51.118043 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:51.094367 30421 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:51.122838 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:51.118870 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 109780us
I20260709 13:09:51.120105 25321 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:09:51.118397 25316 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:51.124667 25284 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:51.124841 30387 raft_consensus.cc:2764] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 13:09:51.125288 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:51.126015 25284 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:51.127228 25321 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:09:51.122958 25318 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:51.134158 30487 consensus_queue.cc:1059] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:09:51.135324 30487 consensus_queue.cc:1059] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:09:51.136319 30487 raft_consensus.cc:3055] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:51.136682 30487 raft_consensus.cc:740] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:51.137415 30062 raft_consensus.cc:3055] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:51.137856 30062 raft_consensus.cc:740] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:51.138680 30062 consensus_queue.cc:260] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:51.139868 30062 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:51.142280 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:51.144169 30176 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:51.117298 29964 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:51.145216 30176 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:51.121888 30421 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:51.146142 30421 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:09:51.146670 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:51.149307 30431 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.151103 30462 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:51.093763 30075 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.156984 25284 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:51.158324 25386 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:51.163269 25315 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:09:51.160480 25212 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:51.167649 25312 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:09:51.172376 30075 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:51.180202 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 96800.6us
I20260709 13:09:51.155714 30097 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.185714 30431 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:51.187304 30097 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:51.189628 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.190202 25245 raft_consensus.cc:1813] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Handling vote request from an unknown peer a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:51.190733 25245 raft_consensus.cc:2410] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
W20260709 13:09:51.192163 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:51.193950 25358 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:51.117236 30306 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.113039 30038 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:51.198320 25238 raft_consensus.cc:1275] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:09:51.202600 25319 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:09:51.202852 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:51.203522 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:51.206061 25936 raft_consensus.cc:1813] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Handling vote request from an unknown peer a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:51.207098 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:51.207762 25319 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:51.153038 30460 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.208365 25936 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 15: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:51.168861 30458 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.166059 30305 consensus_queue.cc:1059] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 13:09:51.216611 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:51.168969 30396 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.041s
I20260709 13:09:51.226560 30306 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:51.233461 25911 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:51.208856 30373 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:51.205507 30065 consensus_queue.cc:1059] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 13:09:51.245622 25941 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: []
W20260709 13:09:51.246943 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 104764us
W20260709 13:09:51.244146 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 87015.8us
I20260709 13:09:51.259109 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:51.259759 25386 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:51.259260 25387 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:51.260849 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:51.262774 25212 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:51.264422 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:51.219868 30369 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:51.257156 30462 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:51.269259 30373 raft_consensus.cc:697] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:09:51.277608 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 120723us
I20260709 13:09:51.278633 30376 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:51.269635 30369 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:51.279196 30075 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:09:51.279573 30495 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:51.233140 30478 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.283068 25284 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:51.283974 25212 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:51.272513 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:51.286360 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:51.289739 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 44354.7us
I20260709 13:09:51.291728 25312 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:51.291744 25317 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 12: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:09:51.295418 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:51.302703 25209 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:51.303321 25386 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:51.257643 30403 consensus_queue.cc:1059] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 13:09:51.307180 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:51.312335 25386 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:09:51.312620 25395 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:51.311650 25389 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:09:51.311949 25391 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:51.314702 25312 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:09:51.317633 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:51.318436 25937 raft_consensus.cc:2410] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 13:09:51.318619 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:51.319813 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:51.289450 30369 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:51.322796 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:51.323249 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:51.323802 25387 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:51.325073 25944 raft_consensus.cc:2368] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12.
I20260709 13:09:51.326443 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:51.326887 25945 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:09:51.298974 30462 raft_consensus.cc:697] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:51.327711 25942 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 15: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:51.265666 30470 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.326483 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:51.331773 25312 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 14: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:51.316690 30097 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:51.290982 30065 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260709 13:09:51.322763 30495 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:51.334632 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:51.336087 25284 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:51.328454 30075 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:09:51.342396 30496 log.cc:927] Time spent T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260709 13:09:51.342839 30498 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:51.343657 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:51.357482 25315 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 9: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:51.352880 25358 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:51.343827 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:51.344421 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:51.344552 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:51.336788 30483 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:51.303731 30376 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:51.348214 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:51.350828 25320 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:51.328271 30469 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:51.360598 25318 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:51.358415 25313 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:51.348377 30154 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:51.364540 30075 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.328231 30472 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20260709 13:09:51.351325 30306 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:51.391644 25908 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:51.372581 25237 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:09:51.380672 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:51.394295 25238 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:51.375433 30176 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:09:51.384132 25318 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:51.358784 30498 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:51.397513 25908 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:51.399260 25285 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:51.376425 30495 raft_consensus.cc:697] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:51.387439 30483 raft_consensus.cc:697] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:51.386984 30373 consensus_queue.cc:237] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.410272 30466 raft_consensus.cc:3055] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:51.410780 30466 raft_consensus.cc:740] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:51.394259 30306 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:51.402976 30456 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:09:51.415252 30456 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:51.415305 30306 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:51.395474 30038 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:51.415782 30456 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:51.407773 30458 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:51.418365 30376 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:51.396358 30462 consensus_queue.cc:237] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.421783 30478 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:51.422803 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:51.423168 25238 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:51.393626 30154 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.427500 25908 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:51.428566 25940 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:51.431528 25908 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:51.433513 25245 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 12. Ops: []
I20260709 13:09:51.434968 25285 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:51.436504 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 151236us
I20260709 13:09:51.437170 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.409756 30097 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:09:51.439502 25209 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:51.441632 30483 consensus_queue.cc:237] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.444736 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:51.445585 30038 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:51.446135 30038 raft_consensus.cc:697] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:51.446331 29928 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:51.446782 29928 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:51.447177 29928 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:51.447103 30038 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.426558 30460 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:51.449966 30495 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.402887 30498 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:51.454006 30470 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:51.456549 30301 consensus_queue.cc:260] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:51.457655 30477 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:51.457835 30301 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:51.458053 30464 raft_consensus.cc:3055] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:09:51.458431 25209 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:51.459936 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:51.458518 30464 raft_consensus.cc:740] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:09:51.461912 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 125902us
W20260709 13:09:51.462353 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 109528us
I20260709 13:09:51.464756 25389 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:09:51.466328 25284 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:51.408263 30437 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:51.468788 25284 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:51.476364 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.479275 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:51.477301 25316 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:51.480268 25390 raft_consensus.cc:2393] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 11: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:51.478617 25245 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 13:09:51.480821 25284 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:51.486977 30501 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:09:51.487468 30501 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:51.487882 30501 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:51.488242 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:51.488397 30217 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 13:09:51.488798 30492 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.270s	user 0.000s	sys 0.003s
I20260709 13:09:51.489656 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:51.490141 30387 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:51.535393 30387 raft_consensus.cc:697] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 13 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:51.490442 30419 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:09:51.490067 30487 consensus_queue.cc:260] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:51.490232 29966 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:09:51.477294 30477 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:51.491528 30469 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:51.493084 30507 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
W20260709 13:09:51.510545 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:51.465289 30097 raft_consensus.cc:697] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:51.469493 30447 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:51.459354 30242 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:51.469749 30472 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:51.533944 25284 leader_election.cc:400] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12.
I20260709 13:09:51.579131 25284 leader_election.cc:403] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:09:51.579659 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 96004.2us
I20260709 13:09:51.537058 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.549660 30477 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:51.484658 30306 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:51.541775 29964 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:51.545970 30502 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:51.589025 30502 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:51.557317 25316 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:51.589481 30502 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:09:51.542179 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 50772.5us
I20260709 13:09:51.590816 29966 raft_consensus.cc:2764] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
W20260709 13:09:51.591781 30503 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260709 13:09:51.583597 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:51.596127 25940 raft_consensus.cc:2393] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 14: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:51.584095 25944 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 13:09:51.581328 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 91737.1us
I20260709 13:09:51.574432 30403 consensus_queue.cc:1059] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.013s
I20260709 13:09:51.586422 30473 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 12
I20260709 13:09:51.578177 30154 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:51.587905 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:51.587908 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:51.599036 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:51.578380 30487 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:51.586530 30477 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:51.551970 30421 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.586709 30306 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:51.626719 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:51.552052 30062 consensus_queue.cc:1059] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.071s
I20260709 13:09:51.632346 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:51.633729 25393 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:51.635825 29928 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.641319 30517 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:51.641500 30097 consensus_queue.cc:237] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.645084 30458 raft_consensus.cc:3055] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:51.645630 30458 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:51.634313 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:51.651463 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:51.668849 25358 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:51.671442 25395 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:09:51.629949 30306 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:51.629341 30075 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:51.601317 29964 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:51.649674 30403 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:51.626773 30483 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.052s
I20260709 13:09:51.694509 30075 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:51.727787 30075 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.730149 30075 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:51.731765 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 63006.2us
I20260709 13:09:51.732573 25358 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:51.726156 30509 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260709 13:09:51.670763 30376 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:51.733449 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:51.733750 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 186206us
I20260709 13:09:51.680408 29966 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:51.630606 30387 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.745534 30512 consensus_queue.cc:1059] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.141s
I20260709 13:09:51.741753 30419 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:51.750658 25395 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 13:09:51.752271 25393 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:51.733361 30518 raft_consensus.cc:3055] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:51.672268 30456 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.771847 25393 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:09:51.748059 30373 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:51.748994 29966 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:09:51.767066 30492 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 13:09:51.729001 30498 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:51.798343 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:51.768803 30518 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:51.805259 25393 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
W20260709 13:09:51.801414 30142 consensus_meta.cc:287] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.733s	user 0.008s	sys 0.004s
I20260709 13:09:51.784479 30373 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:51.787261 29966 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:09:51.807548 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 216943us
W20260709 13:09:51.815264 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:51.817955 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:51.789469 30154 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:51.696386 30458 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.805481 30517 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:51.825033 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 91648.2us
I20260709 13:09:51.829375 25358 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:51.809556 30373 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:51.759200 30403 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.805318 30523 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:51.815002 29928 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:51.809087 30142 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:51.844471 30518 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:51.845371 30511 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:51.857779 29964 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:51.858443 30531 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:51.858285 30511 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:51.858886 30531 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:51.859285 30531 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:51.860558 30511 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:51.861634 25239 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:51.841810 30458 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:51.862713 30142 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:51.833680 30517 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:51.867403 30518 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:51.869333 30421 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:51.873956 30456 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:51.875859 30403 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:51.876539 30517 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:51.877019 25314 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:51.884299 25314 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:51.835286 30462 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 13:09:51.883944 30502 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:51.887392 30502 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:51.889375 25243 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:51.892737 29966 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:51.893937 30369 consensus_meta.cc:287] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.564s	user 0.001s	sys 0.010s
I20260709 13:09:51.894398 30369 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:51.895112 29966 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:51.896548 30369 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:51.907303 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 65258.7us
I20260709 13:09:51.826025 30154 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:51.911017 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:51.911731 25312 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:51.912798 30154 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:51.918095 30498 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:51.930500 25312 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:51.920720 25318 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: []
W20260709 13:09:51.916931 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:51.928393 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:51.935992 30473 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12.
I20260709 13:09:51.937842 25318 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:09:51.939255 25318 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:09:51.940495 25318 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
W20260709 13:09:51.942651 25209 connection.cc:256] RPC call timeout handler was delayed by -1.4883940499998971s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24380000000000002s for a response
W20260709 13:09:51.943324 25209 connection.cc:256] RPC call timeout handler was delayed by -1.21917319399995s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1549s for a response
W20260709 13:09:51.943655 25209 connection.cc:256] RPC call timeout handler was delayed by -1.1822731939998903s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15900000000000003s for a response
I20260709 13:09:51.945516 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.945979 25319 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:09:51.946156 25241 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:51.947119 30529 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:51.947192 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.947763 30529 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:51.947762 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.950153 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.950773 25240 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
W20260709 13:09:51.955742 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 126462us
I20260709 13:09:51.959151 25241 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:51.959555 25318 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:51.960397 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.962162 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.962756 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.963039 25244 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 12: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:51.964032 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.961474 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.965378 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.966045 30431 raft_consensus.cc:3055] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:09:51.966315 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:51.966512 30431 raft_consensus.cc:740] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:51.966606 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:51.967267 25240 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:51.967360 30431 consensus_queue.cc:260] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:51.968475 30431 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:51.969736 30529 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:51.964025 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:51.971818 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:09:51.972621 30515 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.345s	user 0.002s	sys 0.000s
I20260709 13:09:51.972993 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:51.975576 25242 raft_consensus.cc:2393] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:51.976352 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.973109 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:51.986037 25240 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:51.987884 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:51.990649 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:51.988071 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.990880 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:51.991919 25240 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:51.990694 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:51.992702 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:51.993700 25242 raft_consensus.cc:2393] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:51.991133 25358 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:51.997249 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:52.000933 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:52.001829 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:09:52.002003 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:52.003237 25241 raft_consensus.cc:2410] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 13 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 13:09:51.996098 30498 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:52.006139 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:52.002032 30217 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:52.018528 25240 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 13:09:52.021749 25315 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:52.018819 25358 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:52.037206 25240 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:52.042644 25246 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:09:52.044049 25246 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:09:51.995707 30517 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 13:09:52.045691 30217 raft_consensus.cc:697] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:52.046725 30217 consensus_queue.cc:237] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:52.028537 30501 consensus_meta.cc:287] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.538s	user 0.006s	sys 0.006s
I20260709 13:09:52.053785 25358 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:52.069828 25315 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
W20260709 13:09:52.050768 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 235190us
I20260709 13:09:52.089491 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2457) took 1128 ms. Trace:
I20260709 13:09:52.089962 25320 rpcz_store.cc:276] 0709 13:09:50.960736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:51.006444 (+ 45708us) service_pool.cc:224] Handling call
0709 13:09:52.089238 (+1082794us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b540035c0d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:52.089421 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":757245696}
I20260709 13:09:52.091997 25911 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:52.096205 25317 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:52.068063 30376 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:09:52.038127 30306 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:52.050189 30419 raft_consensus.cc:697] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:52.070777 30523 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 13:09:52.069748 30501 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.137241 25245 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:52.120522 25209 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:52.138654 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:52.076356 30517 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:09:52.122576 25244 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:52.118382 30373 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.142676 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:52.148291 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:52.135673 30495 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.675s	user 0.028s	sys 0.052s
I20260709 13:09:52.089965 30498 consensus_queue.cc:1059] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
I20260709 13:09:52.094074 30465 consensus_queue.cc:1059] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 13:09:52.094821 30458 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260709 13:09:52.130419 30536 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:52.099984 30511 consensus_queue.cc:1059] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.054s
I20260709 13:09:52.168751 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:52.172264 25389 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:52.137035 30376 raft_consensus.cc:697] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:52.145803 30154 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:52.140533 30403 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:52.131976 30242 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.034s
I20260709 13:09:52.152190 30217 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:52.158990 30495 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.029s	sys 0.052s
I20260709 13:09:52.155544 30502 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:52.157783 30512 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:52.181504 30536 raft_consensus.cc:697] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:52.181558 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:52.140357 30397 consensus_queue.cc:1059] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.035s
I20260709 13:09:52.187882 30498 raft_consensus.cc:3055] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:52.191627 30498 raft_consensus.cc:740] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:52.186583 30536 consensus_queue.cc:237] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:52.167408 30532 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:52.192791 30501 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:52.192519 30498 consensus_queue.cc:260] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:52.192914 30532 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:52.181836 30217 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.194087 30498 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:52.195151 30532 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:52.195608 30217 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:52.183593 30403 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:52.199180 30403 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:52.200271 30493 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.986s	user 0.000s	sys 0.008s
I20260709 13:09:52.200023 30376 consensus_queue.cc:237] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.164007 30517 raft_consensus.cc:697] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:52.211352 25314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2448) took 1253 ms. Trace:
I20260709 13:09:52.211917 25316 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:52.217629 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:52.218614 25321 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:09:52.217407 25314 rpcz_store.cc:276] 0709 13:09:50.957464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:50.957585 (+   121us) service_pool.cc:224] Handling call
0709 13:09:52.211173 (+1253588us) spinlock_profiling.cc:243] Waited 868 ms on lock 0x7b540035a2d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:52.211292 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1996662656}
I20260709 13:09:52.221366 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:52.222071 25389 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:09:52.191851 30477 consensus_meta.cc:287] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.558s	user 0.009s	sys 0.002s
I20260709 13:09:52.201756 30531 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.241217 30483 raft_consensus.cc:3055] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:09:52.241669 30483 raft_consensus.cc:740] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:52.241539 30477 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:52.242542 30502 consensus_queue.cc:1059] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:09:52.243062 25238 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 13:09:52.225687 25315 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 13:09:52.225975 30075 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:09:52.244073 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:52.241051 25212 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:52.230144 25392 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:52.230746 25394 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:52.231189 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:52.231444 25318 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:52.231348 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:52.233392 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:52.235284 30495 consensus_queue.cc:1059] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:09:52.237003 30494 consensus_queue.cc:1059] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:52.202435 30512 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:52.218438 30301 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:52.239315 25909 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:52.240119 25209 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:52.240803 30431 consensus_queue.cc:1059] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:09:52.241039 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:52.224828 25391 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:52.246069 25386 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
W20260709 13:09:52.267927 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:52.246978 25387 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:52.247807 25320 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:52.251315 25389 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:52.273075 25212 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:52.275367 25939 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 11. (term mismatch)
I20260709 13:09:52.275794 25939 pending_rounds.cc:85] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 10
I20260709 13:09:52.275657 30465 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:09:52.276022 30431 raft_consensus.cc:3055] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:09:52.276211 25939 pending_rounds.cc:107] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 11.11
I20260709 13:09:52.276575 25939 raft_consensus.cc:2889] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:52.271454 30458 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:52.280508 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:52.277328 30458 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:52.276510 30431 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:52.282229 29928 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:52.281687 25945 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:09:52.283033 30431 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:52.285379 25909 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:52.285521 25942 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 13:09:52.289860 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:52.290779 25941 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:52.278394 25285 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:52.207902 30154 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.297842 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:52.298180 25209 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:52.266371 30469 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:52.300107 25209 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:52.298817 25940 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:09:52.266603 30512 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:52.304054 25390 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:52.271838 30242 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:52.283607 30458 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:52.302937 30466 consensus_queue.cc:260] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.316244 30466 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:52.322377 30373 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:52.323400 30517 consensus_queue.cc:237] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.306456 30387 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:52.332211 30369 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:52.333516 30306 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:52.336223 30531 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:52.336870 30532 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:09:52.303553 30469 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:52.338941 30469 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:52.290036 30494 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:52.341460 30494 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:52.341847 30494 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:52.246058 30419 consensus_queue.cc:237] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.298805 30502 consensus_queue.cc:1059] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.047s
I20260709 13:09:52.346897 30502 raft_consensus.cc:3055] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:09:52.347445 30502 raft_consensus.cc:740] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:52.298509 30478 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:09:52.349834 30465 consensus_queue.cc:1175] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "2329e428ebbf484fa3ea9452133df567" responder_term: 12 status { last_received { term: 11 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:09:52.289535 29928 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:09:52.352772 30217 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 13:09:52.290371 30537 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.264s	user 0.001s	sys 0.000s
I20260709 13:09:52.247807 30501 consensus_queue.cc:1059] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 13:09:52.355221 30501 raft_consensus.cc:3055] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:09:52.355639 30501 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:52.308794 25209 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:52.272698 30301 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.305823 30142 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 13:09:52.358584 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 49852.9us
I20260709 13:09:52.371417 25941 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
W20260709 13:09:52.382622 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 82090.5us
I20260709 13:09:52.387038 30495 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20260709 13:09:52.418330 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:52.419366 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:52.424413 25314 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:52.378487 30498 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:52.368546 30478 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:52.297535 29966 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.088s
I20260709 13:09:52.368921 30431 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:09:52.387485 30447 consensus_meta.cc:287] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.809s	user 0.011s	sys 0.001s
I20260709 13:09:52.442113 25321 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:09:52.428395 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:52.445199 25393 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:09:52.447674 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:52.448189 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 64813.7us
I20260709 13:09:52.356609 30501 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.455125 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:52.455727 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:52.348735 30502 consensus_queue.cc:260] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.459975 25321 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:52.459905 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:52.464993 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:52.465332 25312 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:09:52.466125 25314 raft_consensus.cc:2393] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:52.466574 25312 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
W20260709 13:09:52.460708 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 159495us
I20260709 13:09:52.467204 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:52.468173 30498 raft_consensus.cc:697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:52.469442 30498 consensus_queue.cc:237] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.368132 30242 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:52.481624 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:52.481786 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:52.482621 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:52.482730 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:52.483377 25943 raft_consensus.cc:2368] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14.
I20260709 13:09:52.483502 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:52.486155 25237 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 13:09:52.486230 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:52.489013 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:52.488931 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:52.489686 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:52.442754 30075 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:52.498555 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:52.461047 30447 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:52.501819 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:52.503437 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:52.506522 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:52.506901 25314 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:09:52.509814 25943 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:09:52.511026 25937 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:52.511782 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:52.511878 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:52.512888 25945 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:09:52.489863 30549 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:52.516669 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:52.517817 25939 raft_consensus.cc:3055] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:52.518380 25939 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:52.522574 25284 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:52.520668 25939 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.12, Last appended by leader: 12, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:52.456884 30478 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:52.524168 25939 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:52.464349 30456 consensus_queue.cc:1059] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 13:09:52.464773 30536 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.051s
I20260709 13:09:52.508074 30306 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 13:09:52.530081 25209 leader_election.cc:400] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14.
I20260709 13:09:52.536598 25945 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:09:52.537523 25209 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:52.531144 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 45401.2us
I20260709 13:09:52.516007 30501 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:09:52.545667 30513 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:09:52.510994 30075 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.547505 25936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 935) took 1211 ms. Trace:
I20260709 13:09:52.521292 30502 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:52.550779 25287 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:52.558216 25237 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:09:52.560619 25209 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:52.523794 30154 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:52.564953 30556 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:52.574265 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:52.547909 25936 rpcz_store.cc:276] 0709 13:09:51.336032 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:51.350266 (+ 14234us) service_pool.cc:224] Handling call
0709 13:09:51.350603 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:51.350944 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:51.350971 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:51.386861 (+ 35890us) log.cc:844] Serialized 26 byte log entry
0709 13:09:51.534543 (+147682us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:09:51.546974 (+ 12431us) log.cc:844] Serialized 14 byte log entry
0709 13:09:51.550325 (+  3351us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:51.550402 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:52.396338 (+845936us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b1800230e00. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:09:52.547202 (+150864us) raft_consensus.cc:1685] finished
0709 13:09:52.547300 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:52.547423 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3087,"prepare.queue_time_us":10802,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":9799,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":36406,"spinlock_wait_cycles":23069568,"thread_start_us":127723,"threads_started":1,"wal-append.queue_time_us":177493}
I20260709 13:09:52.577020 25209 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:52.547915 30075 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:52.584793 25909 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:52.581279 30456 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.005s
W20260709 13:09:52.594310 30550 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 13:09:52.560276 30301 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:52.596797 25209 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:52.577319 30142 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:09:52.619441 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 180940us
I20260709 13:09:52.575723 30556 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:52.582190 30437 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:52.613858 30397 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:52.639067 30478 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:52.627665 30556 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:52.581393 30038 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.042s
I20260709 13:09:52.645236 30142 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:52.645684 30555 consensus_queue.cc:1059] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:09:52.605983 30075 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:52.646562 30075 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:52.647161 30551 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260709 13:09:52.648159 30545 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
I20260709 13:09:52.654064 25939 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 12 index: 12. (term mismatch)
I20260709 13:09:52.654445 25939 pending_rounds.cc:85] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 11
I20260709 13:09:52.654678 25939 pending_rounds.cc:107] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 11.12
I20260709 13:09:52.654786 30301 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:52.655345 30301 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:52.657843 30514 log.cc:927] Time spent T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.911s	user 0.005s	sys 0.000s
I20260709 13:09:52.648906 30075 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:52.597201 30553 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:52.667768 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:52.596223 30563 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:52.654922 25939 raft_consensus.cc:2889] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:52.616202 30511 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:52.673728 25908 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:52.677026 25908 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:52.677054 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:52.678236 25391 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:52.681249 25321 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:52.683328 25321 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:09:52.652881 30242 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:52.686012 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:52.688476 29928 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:52.689056 29928 raft_consensus.cc:697] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:52.690203 30525 raft_consensus.cc:3055] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:52.690027 29928 consensus_queue.cc:237] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 13, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:52.690761 30525 raft_consensus.cc:740] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:09:52.692875 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 224999us
I20260709 13:09:52.695317 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:52.695988 25241 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:09:52.698132 25908 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:52.701728 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:52.702553 25318 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:52.623287 30494 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.685878 30462 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:09:52.677408 29964 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:52.704869 30038 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:52.708086 30075 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:52.716642 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:52.710578 30397 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:52.711331 30549 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:52.718931 25317 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:09:52.720687 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:52.714028 30301 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:52.726329 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:52.692044 30525 consensus_queue.cc:260] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.694164 30469 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.685786 30553 consensus_queue.cc:1059] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.032s
W20260709 13:09:52.744166 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:52.746980 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:52.727922 25317 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:52.749002 25237 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:52.728742 30397 raft_consensus.cc:697] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:52.716861 29964 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.718120 30438 consensus_queue.cc:1059] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 13:09:52.734452 30494 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:52.757896 30438 raft_consensus.cc:3055] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:52.758308 30438 raft_consensus.cc:740] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:52.759264 30438 consensus_queue.cc:260] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:52.760556 30438 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:52.719377 30075 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:52.763024 30075 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:52.770234 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2328) took 1122 ms. Trace:
I20260709 13:09:52.778626 30547 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:09:52.780920 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:52.781814 25245 raft_consensus.cc:2393] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:52.781868 30547 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:09:52.782753 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:52.783475 25245 raft_consensus.cc:2393] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:09:52.794597 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:52.798478 25942 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:09:52.802680 29964 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:52.811684 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:52.812690 25393 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:52.770731 25243 rpcz_store.cc:276] 0709 13:09:51.647380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:51.647511 (+   131us) service_pool.cc:224] Handling call
0709 13:09:51.647872 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:51.889851 (+241979us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:52.769934 (+880083us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b5400362754. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:52.770139 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":554919168}
I20260709 13:09:52.844175 25392 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:09:52.849004 25391 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:52.726907 30555 consensus_queue.cc:1059] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.120s
I20260709 13:09:52.738364 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:52.711911 30501 consensus_queue.cc:1059] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.010s
I20260709 13:09:52.741927 25321 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:52.751003 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:52.751281 25285 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:52.873657 25315 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:09:52.873960 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 122782us
I20260709 13:09:52.874379 25243 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:09:52.753487 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:52.740850 30559 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:09:52.754433 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:52.755040 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:09:52.755546 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:52.718741 30038 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.725415 30549 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.756916 30142 consensus_queue.cc:1059] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.007s
W20260709 13:09:52.737722 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:52.878239 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:52.872038 25212 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:52.855897 30495 raft_consensus.cc:3055] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:52.856518 30473 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:52.847314 30554 consensus_queue.cc:1059] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260709 13:09:52.906394 25315 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
W20260709 13:09:52.873721 30512 consensus_meta.cc:287] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.539s	user 0.011s	sys 0.000s
I20260709 13:09:52.898720 25389 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 13:09:52.902809 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:52.882155 30547 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:52.909055 25394 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:52.863727 30466 consensus_queue.cc:1059] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.032s
I20260709 13:09:52.908812 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:52.918085 25388 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 13:09:52.918601 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:52.886704 30431 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:52.918190 30466 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:09:52.919375 30431 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:52.918130 25390 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
W20260709 13:09:52.928959 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 151087us
I20260709 13:09:52.893004 30495 raft_consensus.cc:740] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:52.940359 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:52.940876 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:52.876111 30486 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 13:09:52.946339 25243 raft_consensus.cc:2393] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 15: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:52.892629 29964 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:52.941066 25938 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:52.948309 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:52.913312 30142 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:09:52.959787 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:52.962459 25945 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260709 13:09:52.943082 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 219764us
I20260709 13:09:52.944976 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:52.948949 25321 raft_consensus.cc:2376] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12. Re-sending same reply.
W20260709 13:09:52.933840 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 185484us
I20260709 13:09:52.921789 30547 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:52.926528 30501 raft_consensus.cc:3055] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:52.968029 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:52.910272 30473 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:52.970149 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:52.947600 30553 raft_consensus.cc:3055] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:09:52.979647 25241 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 15. Current term is 16. Ops: []
I20260709 13:09:52.982112 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:52.982375 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:52.983981 25321 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 13:09:52.989933 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 47794.1us
I20260709 13:09:52.991281 25358 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:52.994105 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:52.983036 25313 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:52.942879 30478 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 13:09:52.946182 25393 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:09:52.970813 30547 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:53.002192 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:53.007848 25314 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:09:53.008478 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:52.927803 30512 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:52.971024 30501 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:09:53.011826 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:52.977593 30431 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:53.015146 25320 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 13:09:53.005056 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:53.030256 30469 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:53.030356 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:52.927296 30397 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.034742 30572 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 13:09:53.037972 30566 log.cc:927] Time spent T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 13:09:52.995523 30305 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:52.997758 30421 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:52.966408 29964 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:53.045908 30473 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:52.998324 30486 consensus_queue.cc:1059] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 13:09:53.047245 30549 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:53.001098 30495 consensus_queue.cc:260] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:52.980448 30553 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:53.048648 30038 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:53.013329 30554 consensus_queue.cc:1059] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 13:09:53.058245 25938 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:53.114440 25317 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:09:53.115774 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 121752us
I20260709 13:09:53.164870 25316 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:53.169531 30564 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:53.174639 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:53.178260 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:53.315176 25315 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:53.179064 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:53.040211 30556 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.179361 25239 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:09:53.182989 25908 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:53.180595 30421 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:09:53.298566 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:53.326408 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:53.299285 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:53.299919 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:53.301529 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 406260us
I20260709 13:09:53.328239 25244 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:53.164769 30376 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:09:53.302055 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:53.307844 25358 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:53.330660 25358 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:53.331835 25358 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:53.170552 25237 raft_consensus.cc:2376] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14. Re-sending same reply.
I20260709 13:09:53.163108 30565 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:53.314271 25314 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:09:53.313302 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 330346us
I20260709 13:09:53.179797 30502 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:53.322973 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 140092us
I20260709 13:09:53.184286 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:53.337200 25243 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 15: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:53.322073 25316 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
W20260709 13:09:53.338581 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:53.307807 30456 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:53.342741 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:53.344125 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:53.344372 25319 raft_consensus.cc:2393] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:53.338358 30486 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:53.346836 30486 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.349826 30486 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:53.312808 30419 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:53.358021 25316 raft_consensus.cc:1275] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:09:53.295327 30305 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:53.330180 30421 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:53.361657 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 891) took 1389 ms. Trace:
I20260709 13:09:53.332742 30495 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:53.362030 25246 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 13:09:53.334606 29966 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:53.400204 25240 rpcz_store.cc:276] 0709 13:09:51.972131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:52.028418 (+ 56287us) service_pool.cc:224] Handling call
0709 13:09:52.028752 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:52.037682 (+  8930us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:53.361417 (+1323735us) spinlock_profiling.cc:243] Waited 8.22 ms on lock 0x7b54003eb054. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:53.361599 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18916224}
I20260709 13:09:53.375883 25314 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:09:53.329772 30564 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.387519 29964 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:53.388123 30570 log.cc:927] Time spent T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.381s	user 0.001s	sys 0.000s
I20260709 13:09:53.390640 30512 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:53.380793 30477 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:53.345522 30431 raft_consensus.cc:2764] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 13:09:53.416294 25317 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:53.417821 29966 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:53.354677 30376 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:53.393814 30585 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:09:53.463107 30585 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
W20260709 13:09:53.461716 30438 consensus_meta.cc:287] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.701s	user 0.005s	sys 0.007s
I20260709 13:09:53.462280 29966 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:09:53.476848 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:53.344795 30565 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.354307 30502 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:53.481010 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 144443us
I20260709 13:09:53.359050 30456 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:53.482599 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 146843us
I20260709 13:09:53.398970 30419 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.488735 25908 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:53.498521 25908 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:53.475917 30563 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:53.362706 30553 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:53.503135 25909 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.008s	sys 0.006s
W20260709 13:09:53.508284 25909 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.008s	sys 0.006s
I20260709 13:09:53.520627 25939 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:53.522171 30553 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:53.509197 30431 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:53.493762 25911 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:53.505963 30075 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:53.534711 30431 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:53.544185 25908 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:53.545681 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:53.528877 30525 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:09:53.547468 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:53.510259 30583 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 13:09:53.573096 30376 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:53.538451 30438 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.035s
I20260709 13:09:53.546927 25284 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:53.548280 30075 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:53.581835 30075 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:53.590795 30559 raft_consensus.cc:3055] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:53.591257 30559 raft_consensus.cc:740] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:53.592137 30559 consensus_queue.cc:260] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.597711 30494 raft_consensus.cc:3055] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:53.598162 30494 raft_consensus.cc:740] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:53.598970 30494 consensus_queue.cc:260] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.600149 30494 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:53.542161 30397 consensus_queue.cc:1059] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.060s
I20260709 13:09:53.604307 30458 raft_consensus.cc:3055] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:53.604809 30458 raft_consensus.cc:740] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:53.605717 30458 consensus_queue.cc:260] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.606882 30458 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:53.609793 30387 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:53.610383 30387 raft_consensus.cc:697] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:53.611367 30387 consensus_queue.cc:237] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.613868 30591 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:53.614535 30591 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.558004 30565 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:53.558525 30554 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:53.559267 30431 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:53.559731 30502 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:53.564404 30564 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:53.537900 30529 raft_consensus.cc:3055] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:53.629287 30529 raft_consensus.cc:740] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:53.569175 30595 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:53.508734 30563 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:53.585847 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:53.587066 30456 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:53.637802 30594 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:09:53.638088 30599 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:53.638532 30594 raft_consensus.cc:697] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:53.638747 30599 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.639605 30594 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.641170 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 893) took 1156 ms. Trace:
I20260709 13:09:53.641417 30555 raft_consensus.cc:3055] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:53.641570 25239 rpcz_store.cc:276] 0709 13:09:52.484965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:52.485173 (+   208us) service_pool.cc:224] Handling call
0709 13:09:53.640937 (+1155764us) spinlock_profiling.cc:243] Waited 672 ms on lock 0x7b54002d0fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:53.641064 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1545722624}
I20260709 13:09:53.588356 29966 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:53.588591 30549 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:09:53.618671 25909 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:53.622375 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:53.540550 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:09:53.629601 30556 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:53.622418 30065 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:53.636719 30469 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:53.641963 30555 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:09:53.618541 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 74453.4us
I20260709 13:09:53.650483 25318 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 13. Current term is 14. Ops: []
W20260709 13:09:53.650432 25319 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2462) took 1703 ms (client timeout 1533 ms). Trace:
W20260709 13:09:53.664041 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 170366us
I20260709 13:09:53.653318 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:53.636236 30142 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:53.656988 25241 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:09:53.661564 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:53.631059 30565 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:53.667869 30498 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:09:53.671000 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 52419.3us
W20260709 13:09:53.664475 25319 rpcz_store.cc:269] 0709 13:09:51.947158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:52.667343 (+720185us) service_pool.cc:224] Handling call
0709 13:09:53.650101 (+982758us) spinlock_profiling.cc:243] Waited 676 ms on lock 0x7b54000631d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:53.650320 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1553772288}
W20260709 13:09:53.675971 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:53.677649 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:53.678473 25239 raft_consensus.cc:2393] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:53.672792 25909 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:53.651854 30369 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:53.683171 25209 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:53.683550 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:53.687341 25240 raft_consensus.cc:2410] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 13:09:53.668452 30579 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:53.663290 30599 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:53.691972 25209 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:53.630606 30554 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:53.677518 30142 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.678628 30456 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:09:53.709614 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:53.668519 30498 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.670809 30469 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:53.663105 30305 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:53.695322 25909 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:53.658555 30559 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:53.683810 30591 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:53.669849 30596 consensus_queue.cc:1059] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 13:09:53.688844 25241 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 13. Current term is 14. Ops: [13.12-13.12]
W20260709 13:09:53.690485 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:53.718425 30590 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:53.691298 30369 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:53.707453 30599 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:53.729262 30604 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:53.696689 30574 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:09:53.694947 30579 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:53.674412 30565 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:53.692610 30564 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
I20260709 13:09:53.685457 30483 consensus_queue.cc:260] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:53.713160 30456 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:53.737569 30518 raft_consensus.cc:3055] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:53.808882 30518 raft_consensus.cc:740] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:53.809865 30518 consensus_queue.cc:260] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:53.810935 30518 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:53.705454 29928 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:53.813112 29928 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:53.813570 29928 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:53.671155 30065 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.681429 30438 consensus_queue.cc:1059] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 13:09:53.817524 25242 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
W20260709 13:09:53.756156 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 60908.2us
I20260709 13:09:53.687534 30437 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.027s
I20260709 13:09:53.761694 25391 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:09:53.820535 30595 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:53.821069 30595 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.823146 30595 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:53.824565 30438 raft_consensus.cc:3055] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:53.825040 30438 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:53.826108 30438 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.827677 30438 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:09:53.763958 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 217097us
I20260709 13:09:53.769927 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2267) took 1723 ms. Trace:
I20260709 13:09:53.829941 25246 rpcz_store.cc:276] 0709 13:09:52.046978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:52.047151 (+   173us) service_pool.cc:224] Handling call
0709 13:09:53.361623 (+1314472us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:53.401431 (+ 39808us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:53.769638 (+368207us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54003eb050. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:53.769841 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2796521344}
I20260709 13:09:53.778640 30501 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.673939 30529 consensus_queue.cc:260] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.785888 30601 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:53.793365 30590 raft_consensus.cc:697] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:09:53.795310 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 463521us
I20260709 13:09:53.846688 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:53.787753 30466 raft_consensus.cc:3055] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:53.848206 30431 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:09:53.848950 30466 raft_consensus.cc:740] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:53.806875 30369 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:53.808012 30142 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:53.808467 30456 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:53.850193 30466 consensus_queue.cc:260] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:53.851909 30466 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:53.853924 30516 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:53.854920 30516 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:09:53.837965 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:53.857733 25312 raft_consensus.cc:2368] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16.
I20260709 13:09:53.858927 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:53.860354 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:53.839926 25321 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:09:53.841279 25316 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:09:53.816833 30065 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:53.862535 25316 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:53.861083 25312 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:53.862190 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:53.864639 30603 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:53.880755 25209 leader_election.cc:400] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16.
I20260709 13:09:53.881289 25209 leader_election.cc:403] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 13:09:53.882829 25358 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:53.886708 30305 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:53.894867 30305 consensus_queue.cc:1048] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 13:09:53.913763 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:53.876544 30456 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.915988 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:53.916805 25315 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:53.697067 30464 consensus_queue.cc:260] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.798125 30579 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:53.915239 25358 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:53.930639 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
W20260709 13:09:53.922250 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 94162.2us
I20260709 13:09:53.923218 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:53.800755 30534 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:53.932478 25313 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
W20260709 13:09:53.933492 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:53.940145 25358 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:53.942399 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 61854.8us
I20260709 13:09:53.950018 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:53.951026 25943 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:53.951424 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:53.844908 30473 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:53.953192 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:53.917840 30501 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:53.959936 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 1070) took 1076 ms. Trace:
I20260709 13:09:53.714160 30555 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:53.967896 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:54.048014 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:54.027719 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:54.048961 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:53.934871 30547 consensus_meta.cc:287] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.759s	user 0.006s	sys 0.006s
I20260709 13:09:54.037091 25392 rpcz_store.cc:276] 0709 13:09:52.883632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:52.886905 (+  3273us) service_pool.cc:224] Handling call
0709 13:09:52.909483 (+ 22578us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:52.926790 (+ 17307us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:09:52.926820 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:52.938474 (+ 11654us) log.cc:844] Serialized 26 byte log entry
0709 13:09:53.172872 (+234398us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:53.172917 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:53.942330 (+769413us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b5400234e54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:53.942404 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:53.942492 (+    88us) raft_consensus.cc:1685] finished
0709 13:09:53.942513 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:53.959865 (+ 17352us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":110,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":10038,"spinlock_wait_cycles":30280960,"thread_start_us":193145,"threads_started":1,"wal-append.queue_time_us":517175}
I20260709 13:09:53.918673 30483 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:53.913662 30574 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:54.041656 25943 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: []
I20260709 13:09:53.845539 30462 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.110s
I20260709 13:09:53.798619 30599 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:53.960181 30502 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:54.061655 25940 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
W20260709 13:09:54.062563 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:53.955669 30376 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:53.934618 30456 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:53.937347 30419 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:53.935365 30601 raft_consensus.cc:697] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:54.072247 25942 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
W20260709 13:09:54.074082 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:53.954277 29966 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:53.928406 30529 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:54.083343 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:54.052011 30494 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:54.036088 30065 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.055s
I20260709 13:09:54.112864 25320 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:09:54.112957 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:54.097265 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 157105us
I20260709 13:09:54.098249 25939 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:09:54.106427 25209 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:54.106962 25241 raft_consensus.cc:1275] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:09:54.116952 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:54.094328 30431 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:54.121582 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:54.121652 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:54.126052 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:54.126582 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:54.088338 30376 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:54.126797 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:54.127035 30376 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:54.127454 25241 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:54.126802 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:54.130868 30596 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:54.132769 30596 consensus_queue.cc:1059] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:54.088092 30465 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:54.122409 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:54.046012 30473 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:54.080294 30563 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:54.133781 30596 raft_consensus.cc:3055] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:54.134191 30596 raft_consensus.cc:740] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:54.135373 30473 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:54.134938 30596 consensus_queue.cc:260] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.133788 30465 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.136318 30596 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:54.137858 30376 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:54.138461 30622 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:53.955906 30534 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:54.143108 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:54.099560 30555 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:54.067734 30547 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.149024 25239 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:09:54.150480 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:54.121376 30603 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:54.154949 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:54.121652 30397 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:54.161610 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:54.142078 30525 consensus_meta.cc:287] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.593s	user 0.009s	sys 0.002s
I20260709 13:09:54.124439 30369 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:54.148691 30512 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:54.165537 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:54.155514 30495 consensus_meta.cc:287] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.736s	user 0.011s	sys 0.000s
I20260709 13:09:54.170234 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 845) took 1447 ms. Trace:
I20260709 13:09:54.179009 25318 rpcz_store.cc:276] 0709 13:09:52.714971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:52.715115 (+   144us) service_pool.cc:224] Handling call
0709 13:09:53.650291 (+935176us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:53.674653 (+ 24362us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:54.161883 (+487230us) spinlock_profiling.cc:243] Waited 935 ms on lock 0x7b54000631d0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:54.162006 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2149871360}
I20260709 13:09:54.166711 25387 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:54.171022 25238 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:09:54.181097 25938 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:09:54.122658 30494 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.166133 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:54.195422 25243 raft_consensus.cc:2410] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 13:09:54.198151 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:54.198778 25239 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:54.158234 30554 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:54.155714 30498 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:54.167677 30369 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:54.206476 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:54.166774 30469 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:54.205868 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:54.174342 30431 raft_consensus.cc:697] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:54.088867 30590 consensus_queue.cc:237] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.164753 30600 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:09:54.216032 30421 consensus_meta.cc:287] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.797s	user 0.004s	sys 0.008s
I20260709 13:09:54.241775 25239 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:09:54.209112 30621 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:54.194203 30462 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:54.211619 30565 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:54.248040 30603 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:54.311324 30603 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.315258 30603 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:54.262784 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:54.265699 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:54.291431 30547 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:54.317030 25937 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:09:54.293315 30581 log.cc:927] Time spent T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 13:09:54.275946 30494 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:54.208034 30619 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:54.316220 25318 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:54.319960 30619 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.323014 30619 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:54.170641 30397 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:54.231598 30369 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:54.195204 30512 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.199826 30601 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:54.254076 30615 log.cc:927] Time spent T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 13:09:54.166263 30574 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.280298 30376 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:54.277104 30431 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:54.285674 30462 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:54.286514 30465 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:54.289671 30621 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:54.309857 30534 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:54.156088 30622 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:54.326256 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:54.334444 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:54.336772 25284 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:54.325687 30397 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:54.342041 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:54.303650 30611 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.151s	user 0.000s	sys 0.001s
W20260709 13:09:54.343441 30438 consensus_meta.cc:287] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.515s	user 0.011s	sys 0.003s
I20260709 13:09:54.385183 25318 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:09:54.349247 30498 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:54.286274 30600 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:54.391047 25315 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:54.305555 30599 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:54.340997 30494 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:54.347882 30512 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:54.409454 25936 raft_consensus.cc:2410] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
W20260709 13:09:54.400018 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 63327.9us
I20260709 13:09:54.257537 30421 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.421471 30387 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:54.422071 30387 raft_consensus.cc:697] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:54.423004 30387 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:54.426971 30621 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:54.427140 30421 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:54.389359 30553 consensus_meta.cc:287] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.863s	user 0.009s	sys 0.004s
W20260709 13:09:54.418794 29928 consensus_meta.cc:287] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.605s	user 0.008s	sys 0.003s
I20260709 13:09:54.397579 30534 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:54.429582 30534 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:54.429502 29928 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:54.430579 30518 consensus_meta.cc:287] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.619s	user 0.007s	sys 0.006s
I20260709 13:09:54.432173 30534 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:54.432296 25237 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
I20260709 13:09:54.432693 25237 pending_rounds.cc:85] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 9
I20260709 13:09:54.419332 30494 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.433228 25237 pending_rounds.cc:107] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260709 13:09:54.433593 25237 raft_consensus.cc:2889] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:54.436096 30600 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.444308 25284 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:54.432565 29928 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:54.420544 30369 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:54.484112 25392 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:09:54.457561 25389 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:09:54.485093 25394 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:09:54.480618 25387 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:54.485399 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:54.452811 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:54.455876 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:54.455096 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:54.457497 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:54.504802 25386 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
W20260709 13:09:54.447110 30559 consensus_meta.cc:287] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.690s	user 0.011s	sys 0.002s
I20260709 13:09:54.505966 25391 raft_consensus.cc:2393] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:54.443603 30438 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:54.513716 25244 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:09:54.450855 30473 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:54.451202 30301 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:54.452486 30606 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:54.514542 25212 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:54.482394 30494 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:54.440748 30498 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.520054 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:54.518391 25241 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:09:54.496614 30477 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:54.526047 30534 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:54.526731 30501 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:54.526903 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:54.515753 30486 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:54.537012 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 438393us
I20260709 13:09:54.538635 25209 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:54.539930 25209 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:54.541698 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:54.543113 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:54.544100 25391 raft_consensus.cc:2393] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:54.545094 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:54.546540 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:54.548348 25209 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:54.527185 30501 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:54.519829 30590 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20260709 13:09:54.521826 30473 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.521080 30438 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:54.523526 30301 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:54.526724 30534 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.534209 30477 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:54.659327 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:54.650197 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:54.648357 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:54.724295 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:54.649297 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:54.541286 30547 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:54.583642 25390 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 13:09:54.647598 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:54.550882 25209 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:54.548979 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:54.877298 25320 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:09:54.876834 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:54.868726 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 909) took 1187 ms. Trace:
I20260709 13:09:54.721338 25945 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16. Ops: []
I20260709 13:09:54.583448 25315 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:09:54.882217 25318 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:09:54.882627 25314 raft_consensus.cc:2393] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 16: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:54.881778 25245 rpcz_store.cc:276] 0709 13:09:53.681041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:53.681363 (+   322us) service_pool.cc:224] Handling call
0709 13:09:53.681719 (+   356us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:53.721543 (+ 39824us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:53.721599 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:53.722034 (+   435us) log.cc:844] Serialized 26 byte log entry
0709 13:09:53.929548 (+207514us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:53.929610 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:54.811156 (+881546us) spinlock_profiling.cc:243] Waited 39.4 ms on lock 0x7b54002d19d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:54.811241 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:54.811308 (+    67us) raft_consensus.cc:1685] finished
0709 13:09:54.811329 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:54.868612 (+ 57283us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":922,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"spinlock_wait_cycles":90734336,"thread_start_us":207048,"threads_started":1,"wal-append.queue_time_us":373861}
I20260709 13:09:54.883436 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:54.883229 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:54.884152 25314 raft_consensus.cc:2393] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:54.884289 25315 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:54.593037 25943 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:45428
I20260709 13:09:54.551054 30483 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:54.541198 30494 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:54.863489 30501 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:54.873713 30574 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:54.810376 30498 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:54.541290 30242 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:09:54.642345 30625 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 13:09:54.643705 30622 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:54.651970 30502 consensus_meta.cc:287] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.557s	user 0.009s	sys 0.000s
I20260709 13:09:54.879206 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:54.896600 30596 consensus_meta.cc:287] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.760s	user 0.008s	sys 0.004s
W20260709 13:09:54.871796 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:54.588369 30579 consensus_meta.cc:287] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.632s	user 0.003s	sys 0.006s
W20260709 13:09:54.642855 30629 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:09:54.560322 25359 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:54.579912 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:54.597962 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:54.581489 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:54.638456 25287 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:54.876178 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:54.551599 30529 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:09:54.582461 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:54.930318 25316 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:54.549139 30559 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:54.886121 25314 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:09:54.592988 30630 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:54.556232 30599 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:54.585947 30369 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:54.540607 30486 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.886336 25943 raft_consensus.cc:606] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:54.608933 30555 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:54.576925 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:54.930200 30622 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:54.583058 30591 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:54.938683 30559 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:09:54.803967 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:54.608704 30553 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:54.635465 30462 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:54.557837 30626 consensus_queue.cc:1059] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.336s
W20260709 13:09:54.620503 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:54.575584 30376 consensus_queue.cc:1059] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.315s
I20260709 13:09:54.575124 30621 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.315s
I20260709 13:09:54.616073 30518 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:54.900383 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:54.887617 30483 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:09:54.889024 30601 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.018s	sys 0.012s
I20260709 13:09:54.881314 30547 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:54.921351 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 361098us
I20260709 13:09:54.982656 30559 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:54.928757 25315 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:54.892524 30038 consensus_queue.cc:1059] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.027s
W20260709 13:09:55.022161 30601 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.018s	sys 0.012s
I20260709 13:09:54.982091 30065 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:55.029628 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 2278) took 1974 ms. Trace:
I20260709 13:09:55.030218 25243 rpcz_store.cc:276] 0709 13:09:53.055309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:53.337961 (+282652us) service_pool.cc:224] Handling call
0709 13:09:55.029377 (+1691416us) spinlock_profiling.cc:243] Waited 850 ms on lock 0x7b54003a9dd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:55.029551 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1955710208}
W20260709 13:09:54.934219 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 295878us
I20260709 13:09:55.031721 25287 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:54.918946 30502 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16.
I20260709 13:09:55.039283 30498 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:55.040064 30154 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:09:55.039875 30498 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:55.041059 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 2279) took 1985 ms. Trace:
I20260709 13:09:55.041481 25238 rpcz_store.cc:276] 0709 13:09:53.055519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:53.326127 (+270608us) service_pool.cc:224] Handling call
0709 13:09:55.040898 (+1714771us) spinlock_profiling.cc:243] Waited 844 ms on lock 0x7b540028afd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:55.041003 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1940844416}
I20260709 13:09:54.893986 30494 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:54.898420 30574 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.956367 25943 raft_consensus.cc:614] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:09:54.607084 30465 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:54.938100 30596 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:54.957726 25392 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:09:54.939599 30529 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:54.615839 30512 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:09:54.963135 30636 log.cc:927] Time spent T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 13:09:54.965205 30599 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:54.961325 30555 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.920393 30579 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.068645 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:54.947927 30630 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:54.968935 30591 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:54.989297 25240 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:09:54.959491 30305 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:54.909397 30242 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:54.928157 25312 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:54.987597 30606 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:09:55.025892 30610 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.879s	user 0.005s	sys 0.000s
I20260709 13:09:55.045928 30431 consensus_queue.cc:1059] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 13:09:55.047029 30065 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.025805 30534 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:55.081758 25395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 1079) took 1374 ms. Trace:
I20260709 13:09:55.038029 25359 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:55.021838 30483 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:55.065099 25936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 1086) took 1102 ms. Trace:
I20260709 13:09:54.990464 30553 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.066063 30465 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:55.029505 30564 raft_consensus.cc:2764] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 13:09:55.023088 30438 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.072371 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:55.086674 25936 rpcz_store.cc:276] 0709 13:09:53.950747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:53.951107 (+   360us) service_pool.cc:224] Handling call
0709 13:09:55.052763 (+1101656us) spinlock_profiling.cc:243] Waited 457 ms on lock 0x7b54006748d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:55.052962 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1050711296}
I20260709 13:09:55.099747 25941 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:09:55.076843 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:55.054421 30301 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.019785 30518 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:55.084105 25395 rpcz_store.cc:276] 0709 13:09:53.707561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:53.707821 (+   260us) service_pool.cc:224] Handling call
0709 13:09:53.708925 (+  1104us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:53.712716 (+  3791us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:53.712746 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:53.722776 (+ 10030us) log.cc:844] Serialized 26 byte log entry
0709 13:09:53.914034 (+191258us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:53.914076 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:53.914150 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:55.025307 (+1111157us) spinlock_profiling.cc:243] Waited 3.77 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:09:55.081446 (+ 56139us) raft_consensus.cc:1685] finished
0709 13:09:55.081470 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:55.081694 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3381,"raft.queue_time_us":4419,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":18205,"spinlock_wait_cycles":8675072,"thread_start_us":190890,"threads_started":1,"wal-append.queue_time_us":360196}
I20260709 13:09:55.110255 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:55.069857 30641 log.cc:927] Time spent T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 13:09:55.061426 30477 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.066781 30529 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:09:55.062494 30596 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:09:55.085804 30642 log.cc:927] Time spent T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:09:55.098152 30376 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:09:55.063037 30626 consensus_queue.cc:1048] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
W20260709 13:09:55.158601 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 608413us
I20260709 13:09:55.087386 30630 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.082448 30065 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.101078 30473 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.082726 30606 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:09:55.142365 30634 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.244s	user 0.001s	sys 0.001s
I20260709 13:09:55.120148 30622 raft_consensus.cc:3055] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:55.118924 30534 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:55.098539 30600 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.064s
I20260709 13:09:55.145118 30596 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:55.074064 30512 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.170624 25909 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:55.139976 30477 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:55.119158 30438 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:55.206226 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.206923 25316 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:09:55.101217 30305 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.162488 30376 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:55.182948 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.219182 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.185556 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.188751 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:55.221560 25943 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:09:55.189657 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:55.190901 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:55.224161 30643 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:55.224670 30643 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:55.226902 30643 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.227380 30643 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:55.227803 30643 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.232882 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:55.233500 25942 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:55.234115 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.234673 25316 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:55.171670 30622 raft_consensus.cc:740] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:09:55.206451 30565 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.768s	user 0.004s	sys 0.029s
W20260709 13:09:55.193389 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:55.178134 25911 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:55.174393 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.213192 30305 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.216221 25285 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:55.231626 30376 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:55.208480 30534 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:09:55.206478 30617 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:09:55.268191 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 91935.6us
W20260709 13:09:55.269305 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 53223.6us
I20260709 13:09:55.238541 30644 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:09:55.274195 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 163974us
W20260709 13:09:55.242349 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 204452us
I20260709 13:09:55.273839 30644 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.176750 30606 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.238788 30473 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:55.289671 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:55.239661 30565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.004s	sys 0.029s
I20260709 13:09:55.217720 30438 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.286760 30305 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:55.208004 30477 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.253981 30465 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.284770 30494 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.297883 30579 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.270658 30301 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:55.270262 30599 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.303812 25359 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:55.306479 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:55.340560 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 44791.9us
I20260709 13:09:55.308727 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 910) took 1166 ms. Trace:
I20260709 13:09:55.341622 25358 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:55.293462 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:55.290674 30458 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:55.292362 30473 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.317766 30501 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:55.320827 30518 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.292570 30065 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:55.352005 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:55.351090 25321 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:09:55.308106 30617 raft_consensus.cc:697] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:55.324551 25942 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:55.354899 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:55.355723 25314 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:09:55.325038 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 101379us
I20260709 13:09:55.284364 30644 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.332677 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:55.360392 25243 raft_consensus.cc:2410] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 13:09:55.333345 25245 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 13:09:55.307056 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:55.289302 30553 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.343434 30655 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:09:55.343767 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:55.344226 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:55.367910 25938 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:09:55.342056 25241 rpcz_store.cc:276] 0709 13:09:54.142751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:54.142854 (+   103us) service_pool.cc:224] Handling call
0709 13:09:55.308558 (+1165704us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b5400193054. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:55.308670 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":861513088}
I20260709 13:09:55.347142 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:55.350374 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.369712 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.370332 25315 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:55.371547 25315 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 11. (term mismatch)
I20260709 13:09:55.371829 25315 pending_rounds.cc:85] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 10
I20260709 13:09:55.351066 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:55.351884 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.352095 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:55.338366 30559 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:55.357257 25209 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:55.321571 30305 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:55.357729 25936 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:55.359195 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:55.345360 30494 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:55.362881 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.352559 30555 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.384864 25320 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:09:55.372920 25244 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:55.372030 25315 pending_rounds.cc:107] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260709 13:09:55.289423 30622 consensus_queue.cc:260] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:55.346804 30465 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:55.386821 25315 raft_consensus.cc:2889] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:55.386873 30465 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.378280 25242 raft_consensus.cc:1240] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:55.346333 30458 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:55.389408 25315 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:09:55.389998 30458 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:55.366780 25316 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:09:55.390496 25315 raft_consensus.cc:606] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:55.381527 25314 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
W20260709 13:09:55.382100 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 65585us
I20260709 13:09:55.382668 25209 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:55.392645 30553 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:55.380606 25321 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:55.393769 30465 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.384073 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:55.393329 30553 raft_consensus.cc:697] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:55.362112 30518 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:55.399456 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 126634us
I20260709 13:09:55.396116 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:55.408740 25239 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:09:55.408878 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:55.410959 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:55.409834 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:55.412652 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:55.413439 25239 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:55.400329 25285 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:55.413686 25909 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:55.371994 30591 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.420506 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.382201 30305 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:55.427772 30501 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.385243 30643 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.429440 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:55.429654 30458 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.384418 30242 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:55.430971 30654 log.cc:927] Time spent T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 13:09:55.418959 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.431727 25314 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:09:55.431937 30458 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.411865 25241 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:09:55.402364 25245 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:55.433143 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:55.421412 25321 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
W20260709 13:09:55.434399 30650 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:09:55.434633 30473 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.435174 30473 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:55.435635 30473 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:55.437328 30512 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.438668 30464 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:55.443852 30574 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.446497 30458 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:55.447224 30458 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.448683 30516 raft_consensus.cc:3055] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:55.449110 30516 raft_consensus.cc:740] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:55.418164 25285 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:55.424391 30627 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:55.424031 30646 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:55.426077 30438 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.455772 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.425828 30666 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:55.395416 30494 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.495623 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:55.424954 30649 log.cc:927] Time spent T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 13:09:55.433038 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:55.497995 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:55.394824 30652 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:55.461580 25321 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:09:55.503986 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:55.492266 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 151366us
I20260709 13:09:55.367125 30657 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.091s
I20260709 13:09:55.451043 30599 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:09:55.497941 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:55.432711 30483 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:55.493981 30591 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:55.409255 30518 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:55.457615 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:55.508107 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.507452 25359 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:55.503856 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 162391us
I20260709 13:09:55.512317 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.531351 25315 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:09:55.496268 25317 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
W20260709 13:09:55.514318 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 128500us
I20260709 13:09:55.456244 30646 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:55.516173 25245 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:55.454638 30242 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:55.493865 30534 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:55.501603 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.537766 25908 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:55.520882 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.530313 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:55.535392 25212 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:55.395206 30658 consensus_queue.cc:1175] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "2329e428ebbf484fa3ea9452133df567" responder_term: 10 status { last_received { term: 10 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 10 index: 6 } }
I20260709 13:09:55.546630 25320 raft_consensus.cc:2410] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 15 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 13:09:55.547624 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.515394 30627 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:55.503682 30652 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:55.541236 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.541975 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.555035 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.555401 25317 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:55.543083 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:55.555701 25320 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 9.
I20260709 13:09:55.535504 25316 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:09:55.558255 25316 raft_consensus.cc:606] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:55.538197 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2669) took 1055 ms. Trace:
I20260709 13:09:55.559414 25389 rpcz_store.cc:276] 0709 13:09:54.481458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:54.904921 (+423463us) service_pool.cc:224] Handling call
0709 13:09:54.905294 (+   373us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:54.909989 (+  4695us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:54.910018 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:54.921880 (+ 11862us) log.cc:844] Serialized 52 byte log entry
0709 13:09:55.233949 (+312069us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:55.518201 (+284252us) log.cc:844] Serialized 14 byte log entry
0709 13:09:55.535334 (+ 17133us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:55.535683 (+   349us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:55.535847 (+   164us) raft_consensus.cc:1685] finished
0709 13:09:55.536112 (+   265us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:55.536610 (+   498us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3495,"prepare.queue_time_us":92844,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":13199,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":9392,"thread_start_us":500644,"threads_started":2,"wal-append.queue_time_us":348509}
I20260709 13:09:55.548339 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:55.564263 25244 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:09:55.551123 25319 raft_consensus.cc:2410] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:09:55.551342 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:55.552160 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.515640 30599 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:55.553807 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.451090 30376 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.524587 30591 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:55.545828 25239 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:55.555076 25315 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:09:55.569900 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:55.608482 25387 raft_consensus.cc:2410] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 13:09:55.572588 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.572741 30661 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:55.611706 25395 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:09:55.449998 30516 consensus_queue.cc:260] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:55.572853 30659 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
I20260709 13:09:55.546316 30242 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.620384 30678 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:55.621011 30678 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.622172 30242 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:55.623251 30678 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:55.624972 30376 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.625278 30656 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:55.625855 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:55.625802 30656 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:55.501549 30617 consensus_queue.cc:237] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.629499 25392 raft_consensus.cc:2393] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 14: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:55.629707 30601 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:09:55.630517 30516 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:55.630807 30483 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.631192 30601 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:09:55.536821 30477 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.576459 30599 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:09:55.550271 30529 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:55.629535 30438 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.004s
I20260709 13:09:55.634794 30494 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.634982 30529 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:55.636816 30662 log.cc:927] Time spent T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
I20260709 13:09:55.626600 25244 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:55.640338 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:55.540197 30646 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:55.640972 25244 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:55.641019 30478 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:55.553864 30553 consensus_queue.cc:237] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.611867 30591 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:55.554183 30652 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:55.641707 30478 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:55.571805 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 73953.4us
I20260709 13:09:55.618728 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:55.635180 25246 raft_consensus.cc:2393] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 17: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:55.570124 30616 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:09:55.647957 25239 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:09:55.652403 25239 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:09:55.653184 25386 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:55.650017 25909 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:55.607116 30606 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.657673 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.659942 30559 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.612447 30630 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:09:55.672344 30242 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:55.653151 30616 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:55.662101 30376 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:55.557843 30627 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:55.661295 30518 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.688009 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.689819 25316 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:09:55.690708 25316 raft_consensus.cc:606] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:55.691215 25316 raft_consensus.cc:614] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:09:55.668808 30630 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:55.676420 30242 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:55.664136 30673 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:55.696388 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.686672 30616 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:55.648516 30661 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.677042 30534 raft_consensus.cc:697] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:55.697798 30630 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:55.696161 30242 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:09:55.715318 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 159262us
I20260709 13:09:55.689144 30376 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.716467 25389 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260709 13:09:55.716976 25908 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:55.718034 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 1084) took 1768 ms. Trace:
I20260709 13:09:55.719113 30473 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.721799 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:55.724761 25312 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:09:55.744112 25312 raft_consensus.cc:606] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:55.744588 25312 raft_consensus.cc:614] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:09:55.744060 25246 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:55.741506 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:55.739615 25941 rpcz_store.cc:276] 0709 13:09:53.949864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:53.949993 (+   129us) service_pool.cc:224] Handling call
0709 13:09:54.044499 (+ 94506us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:55.103450 (+1058951us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:55.717755 (+614305us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400381b54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:55.717957 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2409488128}
I20260709 13:09:55.751325 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.756946 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:55.757797 25246 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:55.758035 30627 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:55.758669 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:55.759230 25246 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:55.759630 30376 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.760032 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:55.760746 25246 raft_consensus.cc:2393] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 14: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:09:55.762082 30596 consensus_meta.cc:287] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.555s	user 0.010s	sys 0.000s
I20260709 13:09:55.762458 30596 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:55.767295 30596 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.781397 30682 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:55.782006 30682 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:55.782369 30682 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:55.746423 30678 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:55.818436 30678 raft_consensus.cc:697] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:55.819499 30678 consensus_queue.cc:237] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.823174 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.823544 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2486) took 1243 ms. Trace:
I20260709 13:09:55.824030 25318 rpcz_store.cc:276] 0709 13:09:54.580171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:54.580510 (+   339us) service_pool.cc:224] Handling call
0709 13:09:54.580887 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:54.882603 (+301716us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:55.823292 (+940689us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b5400044a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:55.823482 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":692266880}
I20260709 13:09:55.800339 30652 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.829035 30652 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.830868 30594 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:55.831117 30534 consensus_queue.cc:237] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:55.831521 30594 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.831990 30547 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.835490 30594 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.838063 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:55.789201 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.839473 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:55.824049 25393 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:09:55.841347 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.794994 30486 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.798548 25390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2661) took 1342 ms. Trace:
I20260709 13:09:55.842154 25390 rpcz_store.cc:276] 0709 13:09:54.455818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:54.455931 (+   113us) service_pool.cc:224] Handling call
0709 13:09:54.484495 (+ 28564us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:54.876676 (+392181us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:55.798265 (+921589us) spinlock_profiling.cc:243] Waited 98.8 ms on lock 0x7b5400147b54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:55.798484 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":227264256}
I20260709 13:09:55.799799 25237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 911) took 1656 ms. Trace:
I20260709 13:09:55.802210 30498 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.807307 30242 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.816892 30661 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:55.817565 30656 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:55.844750 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:55.859124 25938 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:09:55.843268 25237 rpcz_store.cc:276] 0709 13:09:54.142955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:54.143079 (+   124us) service_pool.cc:224] Handling call
0709 13:09:54.143336 (+   257us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:54.434044 (+290708us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:55.799585 (+1365541us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b5400422554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:55.799743 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":663594240}
I20260709 13:09:55.845794 25940 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:55.847183 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.848500 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:55.860668 25936 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:09:55.861011 25392 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:55.849882 25238 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:09:55.851145 25937 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:09:55.863569 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:55.864775 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:55.796270 30305 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:55.867054 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.868425 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.868834 25284 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:55.872038 25284 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:55.873903 25393 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:55.875121 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.876451 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.877204 25393 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 9.
I20260709 13:09:55.831117 30458 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.880239 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.881960 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:55.891999 25390 raft_consensus.cc:2410] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 13:09:55.895532 25387 raft_consensus.cc:2368] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12.
I20260709 13:09:55.897815 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.898485 25393 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:55.879904 30606 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:55.899637 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.901336 25393 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:09:55.904274 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:55.881299 30663 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:55.906632 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.915194 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.915895 25393 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:09:55.916764 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:55.917474 25393 raft_consensus.cc:2393] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 14: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:55.918902 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.881494 30565 consensus_queue.cc:1059] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 13:09:55.922166 30242 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:55.907336 30663 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:55.928808 30663 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:55.901036 30606 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:55.934083 30606 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:55.934894 25941 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16. Ops: []
I20260709 13:09:55.936895 30547 consensus_queue.cc:1059] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:09:55.937780 30547 consensus_queue.cc:1243] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer f646dfd532974dceadfe00005d2bbf59 log is divergent from this leader: its last log entry 16.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:09:55.938509 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.940358 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:55.954514 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.881677 30473 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.025s
I20260709 13:09:55.956326 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.940511 30630 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:55.958788 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.970209 25386 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:09:55.971556 25390 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 15: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:55.970340 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.974047 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.975932 25386 raft_consensus.cc:2393] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:55.983527 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.984414 25386 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 16: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:55.986220 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.987128 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.989949 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.989027 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.996697 25944 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 10: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:55.997704 25386 raft_consensus.cc:2410] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 13:09:56.004666 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:55.967692 30616 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.008808 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.025000 25391 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 12: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:56.025455 30616 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:56.025898 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.027030 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.027659 25391 raft_consensus.cc:2368] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12.
I20260709 13:09:55.967253 30646 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.000994 30599 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:56.003785 30097 raft_consensus.cc:3055] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:09:55.975786 30591 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:56.005578 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 166491us
I20260709 13:09:56.010761 30478 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:56.028327 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.015420 30630 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:56.014700 25393 raft_consensus.cc:2410] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 13:09:56.031284 25317 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 13:09:56.033665 25358 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:56.031042 25287 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.189s (SENT) [suppressed 20 similar messages]
I20260709 13:09:56.035197 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2346) took 1929 ms. Trace:
I20260709 13:09:56.056774 25240 rpcz_store.cc:276] 0709 13:09:54.105659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:54.105856 (+   197us) service_pool.cc:224] Handling call
0709 13:09:54.588597 (+482741us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:55.070961 (+482364us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:56.034319 (+963358us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b5400457254. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:56.035005 (+   686us) spinlock_profiling.cc:243] Waited 482 ms on lock 0x7b5400457250. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:56.035140 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2029773056}
I20260709 13:09:56.038219 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:56.058923 25390 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:56.011627 25945 raft_consensus.cc:1217] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Deduplicated request from leader. Original: 14.11->[15.12-15.12]   Dedup: 15.12->[]
W20260709 13:09:56.043592 30690 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 13:09:56.055253 25391 raft_consensus.cc:2393] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 14: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:56.060461 25945 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 13:09:56.034963 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.070564 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:56.071242 25316 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:56.072688 25358 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:56.080179 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:56.071981 30473 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:56.081591 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.081669 30473 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:56.085310 30431 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:56.085610 30473 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:56.032349 30097 raft_consensus.cc:740] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:56.086762 30097 consensus_queue.cc:260] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.087895 30097 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:56.022254 30305 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:56.089659 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:56.020364 30682 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.092800 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:56.094079 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:56.094363 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:56.095983 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:56.103163 25238 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:09:56.104693 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:56.105259 25238 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:09:56.100239 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:56.107223 25237 raft_consensus.cc:2393] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:56.100955 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:56.107698 25245 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:56.108724 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:56.108922 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:56.109457 25314 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:09:56.111821 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 576492us
I20260709 13:09:56.112622 25212 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:56.113802 25390 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:56.031436 30599 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:56.030628 30646 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:56.033356 30621 raft_consensus.cc:3055] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:09:56.033951 30591 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:56.159493 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:56.116223 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:56.117242 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:56.117884 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:56.161309 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 45134.8us
I20260709 13:09:56.118321 25908 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:56.121235 25395 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:09:56.116218 25240 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:09:56.166666 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:56.121821 30678 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:56.127439 30483 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:09:56.149714 25242 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:56.158083 25285 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:56.179471 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:56.179956 25358 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:56.180619 25285 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:56.167287 25245 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:56.167425 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:56.182250 25246 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:09:56.115517 30555 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:56.126044 25314 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:56.169721 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:56.173834 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 55605us
I20260709 13:09:56.150703 30534 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:56.177794 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:56.178360 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 413045us
I20260709 13:09:56.320976 25395 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:09:56.181103 25243 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 13:09:56.183588 25285 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:56.218062 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.154611 30621 raft_consensus.cc:740] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:56.242869 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:56.173274 30667 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:56.172236 30682 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:56.163112 30565 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.081s
I20260709 13:09:56.155329 30305 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:09:56.223402 30674 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:56.164568 30606 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.384745 25390 raft_consensus.cc:2393] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 15: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:56.386003 30606 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:56.386332 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.386634 30674 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:56.387135 30674 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:56.387558 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.388657 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.219715 30599 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:56.184906 30678 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.389308 30599 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:56.389751 30599 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:56.389904 25320 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:09:56.390883 30678 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:56.184717 30646 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:56.391522 30646 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:56.243048 30652 consensus_queue.cc:1059] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.107s
I20260709 13:09:56.391901 30646 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:56.389757 25246 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:09:56.393682 25245 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:09:56.328547 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:56.329207 25139 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53264 (request call id 207) took 28125 ms (client timeout 15000 ms). Trace:
I20260709 13:09:56.329921 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:56.396663 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:56.398196 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 285647us
I20260709 13:09:56.400301 25212 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:56.402823 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.324014 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:56.343276 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.355646 30387 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:56.409400 30387 raft_consensus.cc:697] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:09:56.409842 30703 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 13:09:56.410537 30387 consensus_queue.cc:237] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:56.394621 25139 rpcz_store.cc:269] 0709 13:09:28.185756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:28.190510 (+  4754us) service_pool.cc:224] Handling call
0709 13:09:56.278466 (+28087956us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:09:50.597654 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:09:50.608140 (+ 10486us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:09:50.608194 (+    54us) write_op.cc:435] Acquired schema lock
0709 13:09:50.608213 (+    19us) tablet.cc:662] Decoding operations
0709 13:09:50.609594 (+  1381us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:09:50.609712 (+   118us) write_op.cc:641] Partition lock acquired for write op
0709 13:09:50.609743 (+    31us) tablet.cc:685] Acquiring locks for 14 operations
0709 13:09:50.611158 (+  1415us) tablet.cc:701] Row locks acquired
0709 13:09:50.611204 (+    46us) write_op.cc:260] PREPARE: finished
0709 13:09:50.611398 (+   194us) write_op.cc:270] Start()
0709 13:09:50.611555 (+   157us) write_op.cc:276] Timestamp: P: 1783602590611331 usec, L: 0
0709 13:09:50.611597 (+    42us) op_driver.cc:348] REPLICATION: starting
0709 13:09:50.731714 (+120117us) log.cc:844] Serialized 5759 byte log entry
0709 13:09:51.170883 (+439169us) op_driver.cc:464] REPLICATION: finished
0709 13:09:51.317609 (+146726us) write_op.cc:301] APPLY: starting
0709 13:09:51.329814 (+ 12205us) tablet.cc:1367] starting BulkCheckPresence
0709 13:09:51.890954 (+561140us) tablet.cc:1370] finished BulkCheckPresence
0709 13:09:51.891091 (+   137us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:09:52.330247 (+439156us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:09:52.330749 (+   502us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=28,mrs_lookups=0
0709 13:09:52.330821 (+    72us) write_op.cc:312] APPLY: finished
0709 13:09:52.331415 (+   594us) log.cc:844] Serialized 129 byte log entry
0709 13:09:52.489634 (+158219us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:09:52.507979 (+ 18345us) write_op.cc:454] Released schema lock
0709 13:09:52.519431 (+ 11452us) write_op.cc:341] FINISH: Updating metrics
0709 13:09:52.577031 (+ 57600us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:09:52.581101 (+  4070us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3088821,"reported_tablets":29,"child_traces":[["op",{"apply.queue_time_us":102661,"apply.run_cpu_time_us":66566,"apply.run_wall_time_us":1251099,"cfile_cache_hit":58,"cfile_cache_hit_bytes":62591,"num_ops":14,"prepare.queue_time_us":116114,"prepare.run_cpu_time_us":26352,"prepare.run_wall_time_us":310218,"raft.queue_time_us":119128,"raft.run_cpu_time_us":32215,"raft.run_wall_time_us":100723,"replication_time_us":542805,"spinlock_wait_cycles":59264,"thread_start_us":373259,"threads_started":5,"wal-append.queue_time_us":393570,"wal-append.run_cpu_time_us":16921,"wal-append.run_wall_time_us":2184448}]]}
I20260709 13:09:56.416452 25209 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:56.179276 30431 consensus_queue.cc:1059] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.102s
I20260709 13:09:56.359372 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:56.367447 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 55777.2us
I20260709 13:09:56.418105 30477 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:56.418655 30477 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.163048 30301 consensus_queue.cc:1059] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.081s
I20260709 13:09:56.419171 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.419934 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:56.424201 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:56.421365 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:56.425112 25386 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:56.426810 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:56.427002 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:56.427552 25941 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:56.427990 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:56.378225 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.417510 25209 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:56.424291 25395 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:09:56.429335 25944 raft_consensus.cc:2410] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 15 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 13:09:56.428550 25387 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:56.434403 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:56.435242 25243 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:56.435400 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:56.435961 25941 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:56.439711 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:56.444335 25388 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:09:56.444512 25239 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 13:09:56.445613 25209 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:56.438838 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 114882us
I20260709 13:09:56.446934 25358 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:56.406952 30494 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:56.449447 30652 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:56.452481 30529 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:56.452955 30529 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:56.427754 30502 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:09:56.453586 30529 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:56.453779 30502 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:56.454327 30502 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:09:56.455123 30038 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:56.455681 30038 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:56.448459 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:56.341254 30555 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:56.452872 30594 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:09:56.447422 25909 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:56.465291 25243 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:56.451297 25209 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:56.466754 25909 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:56.468003 25209 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:56.471148 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:56.473255 25245 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:56.343760 30663 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.460988 30553 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:56.490602 30622 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:56.507917 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:56.484225 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:56.391836 30621 consensus_queue.cc:260] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:56.449219 30606 raft_consensus.cc:3055] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:09:56.509289 30606 raft_consensus.cc:740] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:56.510246 30606 consensus_queue.cc:260] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.494349 30038 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:56.497113 25312 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:56.513046 30305 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:09:56.513581 30305 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:56.513964 30305 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:56.496578 30663 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:56.497556 30681 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:09:56.498756 30097 raft_consensus.cc:3055] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:56.453795 30599 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.504870 30616 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:56.507360 30645 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:09:56.508668 25940 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:09:56.510601 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:56.488857 30553 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.516144 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:56.522805 25944 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:56.519060 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.520824 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:56.522781 25386 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 10.
I20260709 13:09:56.525346 25209 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:56.476562 30617 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:56.476212 30600 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:56.476460 30559 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:09:56.521622 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:56.535218 25388 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:56.475535 30652 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:56.476696 30437 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:56.518477 25941 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:09:56.543237 25937 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:56.531826 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.544077 25392 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:56.519757 30097 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:56.548357 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:56.529322 30656 raft_consensus.cc:2764] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 13:09:56.537998 30674 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:56.534348 30661 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:09:56.570549 25390 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 12: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:56.570472 30661 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:56.571130 30661 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:56.571583 30674 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:56.545977 30600 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.541435 30559 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:56.575135 30477 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:56.531203 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:56.576022 30658 raft_consensus.cc:3055] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Stepping down as leader of term 12
W20260709 13:09:56.576303 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 45231.9us
I20260709 13:09:56.576226 30097 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.577647 30097 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:56.542433 30437 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:56.581041 30437 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:56.534484 30617 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:56.581574 30617 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:56.530069 30574 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:56.583317 30574 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:56.583786 30574 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:56.529529 30596 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.058s
I20260709 13:09:56.587378 30596 consensus_queue.cc:1243] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer 2329e428ebbf484fa3ea9452133df567 log is divergent from this leader: its last log entry 11.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:09:56.558911 30599 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:56.563464 30646 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.575410 30600 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:56.481437 25212 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:56.597870 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:56.598630 25243 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:56.545472 30681 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:09:56.597831 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 116468us
I20260709 13:09:56.592988 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:56.595129 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:09:56.590428 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 71864.8us
I20260709 13:09:56.600318 25387 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:56.589349 30559 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:09:56.607079 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:56.608204 25246 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:09:56.608789 25940 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:56.609709 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:56.609994 30502 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:56.611075 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:56.626660 25938 raft_consensus.cc:2368] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
W20260709 13:09:56.612144 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 458692us
I20260709 13:09:56.612347 30553 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:56.613276 30591 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:56.604274 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:56.602639 25359 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:56.600126 30486 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:56.646385 25359 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:56.601855 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 53578.8us
I20260709 13:09:56.648554 25287 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:56.622704 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:56.650667 25239 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:56.621289 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:56.627625 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:56.590659 30658 raft_consensus.cc:740] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:56.632962 25243 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:56.649474 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:56.649731 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.653592 25386 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:09:56.654655 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.655951 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.626564 25936 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:56.657342 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:56.660341 25940 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:56.661612 25941 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:09:56.662352 25938 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 15. Current term is 16. Ops: []
I20260709 13:09:56.606798 30518 raft_consensus.cc:3055] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:09:56.617551 30600 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:09:56.673019 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:56.664441 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:56.664701 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:56.665544 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:56.674649 30674 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:56.668850 30518 raft_consensus.cc:740] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:56.675086 30674 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:56.675499 30674 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:56.676024 30502 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:56.675848 30518 consensus_queue.cc:260] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.670661 30431 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:09:56.670771 30663 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:56.677709 30369 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:56.677779 30663 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:56.677042 30431 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:56.679075 30431 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:56.677177 30518 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:56.683231 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:56.684545 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:56.684441 30529 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:56.685689 25938 raft_consensus.cc:2393] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:56.667649 25359 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:56.668717 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:56.669823 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 65622.8us
I20260709 13:09:56.699765 25212 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:56.663357 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:56.597055 30656 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.045s
I20260709 13:09:56.666332 25287 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:56.672379 25936 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:09:56.677165 30547 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:56.671438 25386 raft_consensus.cc:1275] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:09:56.710326 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:56.711004 25359 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:56.716420 25318 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:56.711352 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:56.712191 25388 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:09:56.720279 25212 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:56.703620 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:56.721637 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:09:56.709913 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 526408us
I20260709 13:09:56.724707 25285 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:56.723150 25313 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
W20260709 13:09:56.720505 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 54380.4us
W20260709 13:09:56.728307 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 63924.5us
I20260709 13:09:56.730702 25209 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:56.732995 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:56.729925 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:56.735502 25312 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:56.735666 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:56.726018 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:56.734311 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:56.733625 25316 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:09:56.736891 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:56.736907 25936 raft_consensus.cc:2393] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 12: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:56.737690 25317 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 17: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:56.736416 25313 raft_consensus.cc:2393] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:56.616967 30681 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.740585 25285 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:56.741350 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:56.741427 25212 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:56.742313 25320 raft_consensus.cc:1275] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:09:56.747876 25314 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:56.673347 30600 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:56.719132 30655 raft_consensus.cc:2764] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 13:09:56.750602 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:56.728348 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:56.739020 25909 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:56.713085 30627 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:09:56.674168 30553 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:56.725256 30477 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:56.720908 30630 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:56.771710 30712 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:56.763840 25318 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:09:56.764966 25909 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:56.749249 30652 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:56.743088 30486 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:56.724983 30688 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:56.725548 30516 consensus_queue.cc:1059] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.049s
I20260709 13:09:56.765684 25285 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:56.760799 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:56.797235 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:56.775756 30712 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.750824 30547 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:56.745435 30529 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:56.771319 25316 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:09:56.772984 30553 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:56.743782 30369 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:09:56.783206 25312 raft_consensus.cc:2410] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:09:56.762727 30600 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:56.751593 30659 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:56.785991 25318 pending_rounds.cc:85] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 6
I20260709 13:09:56.814538 25318 pending_rounds.cc:107] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 13:09:56.814891 25318 raft_consensus.cc:2889] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:56.785737 30512 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:56.729775 30502 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:56.799739 25285 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:56.801347 30652 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:56.715435 25944 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:09:56.800743 25320 raft_consensus.cc:2410] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 13:09:56.751995 30596 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.037s
I20260709 13:09:56.775415 30630 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.714115 30574 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.772450 30599 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:56.751807 30464 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.037s
I20260709 13:09:56.773952 30678 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:56.860291 30678 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:56.860793 30678 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:56.776659 30711 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:56.861472 30711 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:56.861904 30711 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:56.791826 30616 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 13:09:56.789335 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 571528us
I20260709 13:09:56.837623 30565 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:56.865801 30616 raft_consensus.cc:697] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:56.866859 30616 consensus_queue.cc:237] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.868297 30630 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:56.865818 30565 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:56.868841 25911 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:56.870913 30565 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:56.872498 30682 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:09:56.776837 30687 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 13:09:56.873930 25911 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:56.877658 30617 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.879299 30678 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.765717 30686 consensus_queue.cc:1059] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.057s
I20260709 13:09:56.884711 30686 raft_consensus.cc:3055] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:56.885118 30686 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:56.886178 30686 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.887382 30686 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:56.888651 30678 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:56.768610 25287 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:56.834765 25285 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:56.771966 30534 consensus_queue.cc:1059] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 13:09:56.804252 30547 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:56.805284 30708 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:56.813699 30663 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:56.816331 30553 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:56.797590 30305 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:56.854269 30599 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:56.859508 25242 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 13:09:56.807256 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:56.858093 30437 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:56.851521 30154 consensus_queue.cc:1059] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20260709 13:09:56.822867 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:56.949503 25312 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:56.880048 30617 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:56.799557 30688 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:56.880242 30666 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:09:56.834081 25284 leader_election.cc:400] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:09:56.883450 25911 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:56.890049 25937 raft_consensus.cc:1240] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
W20260709 13:09:56.890098 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 55442.4us
W20260709 13:09:56.890859 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:56.835954 30369 raft_consensus.cc:697] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:56.781265 30658 consensus_queue.cc:260] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:56.925395 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 156804us
I20260709 13:09:56.926141 30547 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:56.906710 30646 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:56.926683 30075 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:09:56.926836 30564 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:09:56.945924 25314 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:09:56.860922 30574 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:56.943456 30708 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:56.947793 30599 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:56.959615 25284 leader_election.cc:403] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 13:09:56.960292 25285 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:56.960776 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 77403.5us
I20260709 13:09:57.007296 25911 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:09:57.008440 25911 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:56.962845 25945 raft_consensus.cc:2368] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15.
I20260709 13:09:57.008783 25312 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:56.963662 25287 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:57.000386 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 178030us
I20260709 13:09:57.010066 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:56.964419 30666 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:57.030385 25314 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:09:57.026057 25394 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
W20260709 13:09:56.953863 25240 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.003s	sys 0.015s
W20260709 13:09:57.035423 25240 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.003s	sys 0.016s
W20260709 13:09:57.045953 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 85741.1us
I20260709 13:09:57.041961 25284 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:57.001053 30710 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.147s	user 0.001s	sys 0.001s
W20260709 13:09:57.056205 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:57.056972 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:09:57.059449 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 95913.3us
I20260709 13:09:57.057466 25243 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:57.058195 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:57.060711 25287 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:57.062758 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:57.064028 25246 raft_consensus.cc:2393] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 14: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:56.979133 30681 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:56.999260 30708 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:57.035858 30645 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:57.072530 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:57.032120 30154 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:09:57.070890 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:57.019726 30075 raft_consensus.cc:697] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:57.032915 30617 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:57.035638 30682 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:09:57.079551 30645 raft_consensus.cc:697] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:57.082141 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:57.083823 25239 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 12: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:57.063016 30387 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:09:57.087776 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 344892us
I20260709 13:09:57.074275 30564 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:09:57.070518 25209 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:57.065471 30655 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:09:57.088349 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 342196us
I20260709 13:09:57.114826 25237 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:57.060019 30715 consensus_queue.cc:1059] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20260709 13:09:57.119109 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:57.120872 30477 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:57.121443 30477 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:09:57.120777 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 50369.9us
I20260709 13:09:57.121821 30477 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:09:57.121661 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 380294us
I20260709 13:09:57.124375 25212 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:57.123371 25209 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:57.083583 30682 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:57.119381 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:57.129498 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 884) took 1034 ms. Trace:
I20260709 13:09:57.062724 30369 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.090126 30712 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:57.119163 30655 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:57.137388 30437 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:57.130122 30667 consensus_queue.cc:1059] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.020s
I20260709 13:09:57.157856 25212 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:57.158375 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 116531us
W20260709 13:09:57.211305 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 53540.2us
W20260709 13:09:57.151614 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 118838us
I20260709 13:09:57.212607 25911 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:57.129433 30681 consensus_queue.cc:1059] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
I20260709 13:09:57.133104 30622 consensus_queue.cc:1059] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
I20260709 13:09:57.167014 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2367) took 1007 ms. Trace:
I20260709 13:09:57.215159 25241 rpcz_store.cc:276] 0709 13:09:56.159551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:56.159784 (+   233us) service_pool.cc:224] Handling call
0709 13:09:56.160150 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:56.169345 (+  9195us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:56.169391 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:56.169970 (+   579us) log.cc:844] Serialized 26 byte log entry
0709 13:09:56.784753 (+614783us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:56.784796 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:56.784888 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:57.166714 (+381826us) raft_consensus.cc:1685] finished
0709 13:09:57.166747 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:57.166929 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":191008,"raft.queue_time_us":18359,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":40261,"thread_start_us":423712,"threads_started":1,"wal-append.queue_time_us":736141}
I20260709 13:09:57.132601 30565 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:57.172610 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 111965us
I20260709 13:09:57.217084 25287 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:57.174350 25359 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:57.150631 30305 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:57.184665 30682 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:09:57.193498 30717 log.cc:927] Time spent T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.165s	user 0.001s	sys 0.000s
I20260709 13:09:57.159196 30075 consensus_queue.cc:237] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:57.195393 30724 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:09:57.195317 30656 raft_consensus.cc:3055] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Stepping down as leader of term 14
W20260709 13:09:57.196436 30722 log.cc:927] Time spent T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:09:57.208420 30655 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:09:57.180168 25315 rpcz_store.cc:276] 0709 13:09:56.095313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:56.098797 (+  3484us) service_pool.cc:224] Handling call
0709 13:09:56.099125 (+   328us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:56.099471 (+   346us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:56.099499 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:56.099924 (+   425us) log.cc:844] Serialized 26 byte log entry
0709 13:09:56.611744 (+511820us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:56.611792 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:56.611870 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:57.129281 (+517411us) raft_consensus.cc:1685] finished
0709 13:09:57.129303 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:57.129439 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11054,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":15781,"thread_start_us":433820,"threads_started":1,"wal-append.queue_time_us":639523}
W20260709 13:09:57.254462 25394 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 1974) took 1465 ms (client timeout 1748 ms). Trace:
W20260709 13:09:57.277976 25394 rpcz_store.cc:269] 0709 13:09:55.788717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:55.788885 (+   168us) service_pool.cc:224] Handling call
0709 13:09:57.254248 (+1465363us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54001e3a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:57.254389 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2624269568}
W20260709 13:09:57.252835 30674 consensus_meta.cc:287] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.577s	user 0.009s	sys 0.001s
I20260709 13:09:57.164166 30498 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:09:57.209103 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 90362us
I20260709 13:09:57.301966 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2678) took 1133 ms. Trace:
I20260709 13:09:57.302417 25391 rpcz_store.cc:276] 0709 13:09:56.168989 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:56.170325 (+  1336us) service_pool.cc:224] Handling call
0709 13:09:56.218268 (+ 47943us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:56.218623 (+   355us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:56.218650 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:56.329840 (+111190us) log.cc:844] Serialized 26 byte log entry
0709 13:09:56.832549 (+502709us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:56.832589 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:56.832660 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:57.277961 (+445301us) raft_consensus.cc:1685] finished
0709 13:09:57.277983 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:57.301899 (+ 23916us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64845,"raft.queue_time_us":325,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":35884,"thread_start_us":421533,"threads_started":1,"wal-append.queue_time_us":548789}
W20260709 13:09:57.218254 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 44080.3us
I20260709 13:09:57.220534 25911 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:57.306470 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 86024.2us
I20260709 13:09:57.307202 25911 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:09:57.228250 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:57.228765 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:57.309613 25944 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
W20260709 13:09:57.209180 30720 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 13:09:57.198084 30688 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:09:57.313856 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 96858.7us
W20260709 13:09:57.207609 30518 consensus_meta.cc:287] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.524s	user 0.016s	sys 0.000s
I20260709 13:09:57.229801 30727 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:09:57.243294 30621 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:57.229151 30652 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.015s
I20260709 13:09:57.244021 30387 raft_consensus.cc:697] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:57.337234 25212 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:57.337879 30727 raft_consensus.cc:697] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:57.337663 30387 consensus_queue.cc:237] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:57.338613 25212 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:57.338903 30727 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:57.244848 30661 consensus_meta.cc:287] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.673s	user 0.006s	sys 0.006s
I20260709 13:09:57.245496 30516 raft_consensus.cc:3055] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:09:57.351584 25237 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:09:57.231303 30673 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:57.219360 30729 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:57.146869 30645 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.253090 25140 catalog_manager.cc:5697] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 8 to 9, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 9 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:09:57.257411 30737 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:09:57.258697 30730 log.cc:927] Time spent T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260709 13:09:57.269776 30723 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
W20260709 13:09:57.208508 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 85048.7us
I20260709 13:09:57.310211 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:57.219014 30555 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:57.239603 30719 log.cc:927] Time spent T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
I20260709 13:09:57.240437 30667 consensus_queue.cc:1243] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 13.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:09:57.235780 30616 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:09:57.233681 30564 raft_consensus.cc:697] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:57.219822 30726 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:57.315865 25287 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:57.293314 30431 consensus_meta.cc:287] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.614s	user 0.011s	sys 0.007s
I20260709 13:09:57.276955 30656 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:09:57.342080 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:57.368597 30483 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:57.312147 30437 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:09:57.348750 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:57.304467 30591 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:57.352795 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:57.435393 25237 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:09:57.366035 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:57.437496 25239 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 12: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:57.328190 30736 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:57.437101 25359 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:57.371958 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:09:57.372958 30097 consensus_meta.cc:287] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.795s	user 0.013s	sys 0.000s
I20260709 13:09:57.372707 30728 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:09:57.373456 30673 raft_consensus.cc:697] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:57.376513 30477 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:57.377409 25140 catalog_manager.cc:5697] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 9 to 10, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 10 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:09:57.378939 30715 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:57.381534 30731 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:09:57.382620 30749 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:57.337670 30305 raft_consensus.cc:3055] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:57.384325 30553 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.327312 30674 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.387550 30154 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:09:57.350976 30661 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:57.390566 30737 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:57.392863 30599 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:57.394484 30565 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:09:57.352123 30600 consensus_meta.cc:287] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.520s	user 0.008s	sys 0.003s
I20260709 13:09:57.342137 30686 consensus_queue.cc:1059] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 13:09:57.402086 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:57.400667 25209 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:57.399296 25394 raft_consensus.cc:2376] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14. Re-sending same reply.
I20260709 13:09:57.409667 30498 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.002s
I20260709 13:09:57.354383 30745 raft_consensus.cc:3055] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:57.411891 30301 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:09:57.485553 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 84980.2us
I20260709 13:09:57.355582 30729 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:57.351364 30516 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:57.422019 25284 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:57.346396 25212 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:57.423347 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:57.426383 30656 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.427441 30431 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:57.431613 30547 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.358225 30622 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:57.447268 30746 consensus_queue.cc:1059] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 13:09:57.253451 30711 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:57.478052 30744 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:09:57.439972 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 124201us
W20260709 13:09:57.416322 30559 consensus_meta.cc:287] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.774s	user 0.011s	sys 0.002s
I20260709 13:09:57.443480 25237 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:57.461499 25140 catalog_manager.cc:5697] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 11 to 14, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 14 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:09:57.483362 30667 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:57.446070 30688 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:57.444944 30591 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.478940 25238 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 16: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:57.447451 30437 raft_consensus.cc:697] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:57.449553 30736 raft_consensus.cc:697] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:57.481433 30737 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:57.463130 30305 raft_consensus.cc:740] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:09:57.539908 30740 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
W20260709 13:09:57.540982 30750 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 13:09:57.540762 30305 consensus_queue.cc:260] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:57.533546 30656 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:57.542043 30305 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:09:57.542454 30741 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 13:09:57.490258 30301 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.539829 30736 consensus_queue.cc:237] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.547497 30301 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:57.483510 30600 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.549911 30600 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:57.549738 30437 consensus_queue.cc:237] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.528721 30622 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.552754 30622 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:57.435308 25241 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:57.479169 30681 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:57.557286 30681 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:57.486539 25209 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:57.558619 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 72139.5us
I20260709 13:09:57.559334 25209 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:09:57.492789 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 146457us
I20260709 13:09:57.560426 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:57.473915 25239 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 13:09:57.485424 30745 raft_consensus.cc:740] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:57.561993 30555 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:09:57.488154 30731 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:57.489183 30729 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:57.484263 30564 consensus_queue.cc:237] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.555763 30477 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:57.555927 30553 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:57.535627 30559 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:57.556694 30606 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:57.559487 30681 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:57.466501 30749 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:57.534145 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 111231us
I20260709 13:09:57.563549 25209 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:09:57.564407 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:57.560935 30713 log.cc:927] Time spent T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.533s	user 0.001s	sys 0.003s
I20260709 13:09:57.564523 25287 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:57.553529 25140 catalog_manager.cc:5697] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 7 to 10, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 10 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:09:57.536938 30688 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:57.602227 25140 catalog_manager.cc:5697] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:09:57.562532 30555 raft_consensus.cc:697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 17 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:57.531698 30652 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 13:09:57.573508 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:57.532459 30728 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:57.562242 30745 consensus_queue.cc:260] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.566869 30498 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:09:57.567623 30518 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:57.566192 30673 consensus_queue.cc:237] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:57.613610 25246 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 15. Current term is 16. Ops: []
I20260709 13:09:57.616889 25140 catalog_manager.cc:5697] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 6 to 9, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 9 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:09:57.605669 25287 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:57.563246 30075 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:57.626135 25287 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:57.598155 30534 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:57.604585 30688 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:57.563548 30667 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.628978 25287 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:57.599376 30731 raft_consensus.cc:697] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:57.567497 30726 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.056s
I20260709 13:09:57.636641 25287 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:57.638022 30547 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:57.631877 30565 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:57.643715 30711 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:57.599701 30516 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.615320 30744 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:57.625157 30477 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:09:57.646400 30301 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:57.627601 30387 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:09:57.722743 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 161494us
I20260709 13:09:57.657939 30661 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:57.660240 25239 raft_consensus.cc:1240] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:57.660118 30682 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.661583 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:57.648295 30746 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:57.666486 25238 raft_consensus.cc:1240] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
I20260709 13:09:57.677217 30674 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:57.680419 30663 raft_consensus.cc:3055] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:57.686357 30599 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:57.686437 30559 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:57.689916 30555 consensus_queue.cc:237] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:57.691893 30727 raft_consensus.cc:3055] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:57.692333 30667 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:57.693463 25394 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:57.684886 30737 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:57.699477 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 975) took 1091 ms. Trace:
I20260709 13:09:57.697660 25140 catalog_manager.cc:5697] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 10 to 13, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 13 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:09:57.700826 30708 consensus_meta.cc:287] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.607s	user 0.006s	sys 0.004s
I20260709 13:09:57.701496 30731 consensus_queue.cc:237] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:57.647257 30534 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.624387 30728 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:57.650303 25287 leader_election.cc:400] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12.
I20260709 13:09:57.706724 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2303) took 1541 ms. Trace:
W20260709 13:09:57.764545 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 114448us
I20260709 13:09:57.764349 25240 rpcz_store.cc:276] 0709 13:09:56.165552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:56.165688 (+   136us) service_pool.cc:224] Handling call
0709 13:09:56.166441 (+   753us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:56.407623 (+241182us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:56.407654 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:56.408252 (+   598us) log.cc:844] Serialized 52 byte log entry
0709 13:09:57.056757 (+648505us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:09:57.064568 (+  7811us) log.cc:844] Serialized 14 byte log entry
0709 13:09:57.653749 (+589181us) spinlock_profiling.cc:243] Waited 3.36 ms on lock 0x7b1800278620. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:09:57.700984 (+ 47235us) spinlock_profiling.cc:243] Waited 9.76 ms on lock 0x7b1800278620. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b3fc0a 00007f297d27f219 00007f2973bc7a40 00007f2973bc24e4 00007f2973bc2455 00007f297a6aa3c7 00007f2976cbc9a7 00007f2976ccc0bf 00007f2976ccc038
0709 13:09:57.705984 (+  5000us) log.cc:844] Serialized 14 byte log entry
0709 13:09:57.706280 (+   296us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:57.706409 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:57.706464 (+    55us) raft_consensus.cc:1685] finished
0709 13:09:57.706481 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:57.706653 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":883356,"prepare.queue_time_us":666796,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":75,"raft.queue_time_us":246414,"raft.run_cpu_time_us":368,"raft.run_wall_time_us":12688,"spinlock_wait_cycles":30157440,"thread_start_us":612677,"threads_started":1,"wal-append.queue_time_us":535293}
I20260709 13:09:57.766238 25287 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:57.767007 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:57.767352 25287 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:57.767712 25240 raft_consensus.cc:2368] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16.
I20260709 13:09:57.768489 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:57.711231 25359 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:57.714115 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:57.730795 30483 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:57.720677 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:57.782963 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:57.783547 25945 raft_consensus.cc:2368] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16.
I20260709 13:09:57.783757 25238 raft_consensus.cc:2393] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 15: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:57.720829 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:57.784432 25240 raft_consensus.cc:2368] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:09:57.784638 25938 raft_consensus.cc:2393] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 14: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:57.721781 25284 leader_election.cc:400] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15.
I20260709 13:09:57.631469 30075 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:57.795104 25287 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:57.739837 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:57.704782 30749 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:57.801514 30742 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:09:57.802137 30075 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:57.794275 30674 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:57.740689 30663 raft_consensus.cc:740] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:57.799772 25140 catalog_manager.cc:5697] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 8 to 12. New cstate: current_term: 12 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:09:57.723560 30744 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:57.758069 25937 rpcz_store.cc:276] 0709 13:09:56.607646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:56.616817 (+  9171us) service_pool.cc:224] Handling call
0709 13:09:56.617179 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:56.964678 (+347499us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:57.699238 (+734560us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b54004ad8d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:57.699415 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":627090304}
I20260709 13:09:57.630771 30655 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:57.710291 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 146814us
I20260709 13:09:57.770386 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:57.771298 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:57.771910 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:57.728009 30477 raft_consensus.cc:697] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:57.720235 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:57.784989 25284 leader_election.cc:403] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
W20260709 13:09:57.822791 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 101158us
W20260709 13:09:57.823817 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:57.824517 25284 leader_election.cc:400] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16.
I20260709 13:09:57.731086 30729 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:57.786191 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 75263.9us
I20260709 13:09:57.793171 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:57.798219 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:57.829490 25241 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:09:57.830590 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:57.799842 30681 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:09:57.762526 30387 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:57.800760 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:57.800616 30483 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:57.721204 30494 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:57.750952 30727 raft_consensus.cc:740] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:57.809216 25287 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:57.752471 30516 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:57.764668 30708 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:57.818372 25394 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:57.864223 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:57.818771 25388 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 12: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:57.818331 25209 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:57.824795 25284 leader_election.cc:403] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:09:57.782822 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:57.867094 25395 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:57.869315 30565 consensus_queue.cc:1059] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:09:57.870474 30565 raft_consensus.cc:3055] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:09:57.871071 30565 raft_consensus.cc:740] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:57.805647 30674 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:57.826725 25944 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 13:09:57.817066 25140 catalog_manager.cc:5697] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 9 to 10, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 10 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:09:57.877875 25140 catalog_manager.cc:5697] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 10 to 11, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 11 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:57.880085 25140 catalog_manager.cc:5697] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 12 to 13, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 13 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
W20260709 13:09:57.827751 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:57.794562 30599 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:57.833878 25241 raft_consensus.cc:2410] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 15 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 13:09:57.813962 30737 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:57.849584 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:09:57.854091 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 44994.3us
I20260709 13:09:57.883492 25238 raft_consensus.cc:2368] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17.
I20260709 13:09:57.884847 25315 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:09:57.818346 25393 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:09:57.831826 30711 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:09:57.858984 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 168626us
I20260709 13:09:57.872242 30565 consensus_queue.cc:260] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:57.861483 25245 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:57.810832 30746 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.032s
I20260709 13:09:57.866535 30712 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 16
I20260709 13:09:57.896132 30565 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:57.897275 30711 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15.
I20260709 13:09:57.897435 25140 catalog_manager.cc:5697] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 8 to 10, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 10 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:09:57.900068 25394 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 16: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:09:57.900590 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 82367.7us
I20260709 13:09:57.900367 25395 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:09:57.909116 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:57.909384 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:57.834276 30681 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:09:57.913255 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:57.914685 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:57.956416 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:57.957332 25246 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:57.915309 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 919) took 2501 ms. Trace:
I20260709 13:09:57.857543 30431 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:57.959455 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:57.978801 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:57.979300 25388 raft_consensus.cc:2368] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16.
I20260709 13:09:57.961815 25393 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:09:57.963073 25312 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 13:09:57.980140 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:57.964311 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:57.966811 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:57.981248 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:57.968667 25938 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: [12.11-14.12]
I20260709 13:09:57.981981 25388 raft_consensus.cc:2368] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:09:57.969511 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:57.971799 25319 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 13:09:57.984501 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:57.985548 25393 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 12: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:57.986949 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:57.987775 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:57.988777 25316 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:09:58.013540 25314 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 12: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:57.995388 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:57.995433 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:58.015466 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:58.016322 25388 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:09:58.016491 25393 raft_consensus.cc:2368] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16.
I20260709 13:09:57.849678 30622 consensus_queue.cc:1059] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.108s
I20260709 13:09:57.849929 30305 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.107s
I20260709 13:09:57.914738 30711 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:57.998139 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:57.860179 30547 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:58.004437 25140 catalog_manager.cc:5697] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:09:57.917532 30590 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:57.953908 30726 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:09:58.007776 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:57.849620 30387 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:57.960855 30716 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 13:09:57.959115 30681 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:09:58.010260 25321 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12. Ops: []
W20260709 13:09:58.011201 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 124162us
I20260709 13:09:58.012508 25313 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:57.958159 25244 rpcz_store.cc:276] 0709 13:09:55.399474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:55.641721 (+242247us) service_pool.cc:224] Handling call
0709 13:09:55.642026 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:56.156202 (+514176us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:09:56.223470 (+ 67268us) log.cc:844] Serialized 14 byte log entry
0709 13:09:57.560386 (+1336916us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:09:57.860266 (+299880us) log.cc:844] Serialized 14 byte log entry
0709 13:09:57.900522 (+ 40256us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:57.900585 (+    63us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:09:57.900655 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:57.900743 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:57.900886 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1404331,"prepare.queue_time_us":762272,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":3844,"raft.queue_time_us":614357,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":17462,"spinlock_wait_cycles":9403264,"thread_start_us":935398,"threads_started":3,"wal-append.queue_time_us":626080}
I20260709 13:09:57.823307 30744 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:58.018472 25314 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:09:58.018406 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:58.045934 30547 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:58.062429 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:58.032866 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:58.033403 25945 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 16: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:58.039188 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:58.063946 25316 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:58.041575 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:57.977555 30477 consensus_queue.cc:237] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:58.070818 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:58.071798 25244 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:58.043244 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:09:58.010152 30713 log.cc:927] Time spent T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:09:58.008653 30301 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:58.049369 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:57.915529 30437 consensus_queue.cc:1059] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.117s
I20260709 13:09:58.054668 30534 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:58.048638 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:58.021943 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 261961us
I20260709 13:09:58.055593 30726 raft_consensus.cc:697] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 17 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:58.056455 25212 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:58.025907 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:58.066423 25246 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:09:58.042099 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:58.050529 30305 consensus_queue.cc:1243] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 14.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:09:58.072927 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:58.084136 25212 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:58.084640 25314 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:58.034017 30673 consensus_queue.cc:1059] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.026s
W20260709 13:09:58.061977 30725 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.748s	user 0.000s	sys 0.004s
I20260709 13:09:58.067034 30547 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:58.098431 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:58.050230 25140 catalog_manager.cc:5697] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 8 to 10, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 10 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:09:57.913203 30663 consensus_queue.cc:260] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:58.079147 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:58.072024 30621 consensus_meta.cc:287] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.733s	user 0.010s	sys 0.002s
I20260709 13:09:58.081660 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:58.045181 30711 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:58.074236 25319 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:58.111267 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:58.111842 25244 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:58.114794 25246 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:58.115639 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:58.116123 25246 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:58.088766 25316 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:09:58.093528 25392 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 13:09:58.110793 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:58.112011 30154 consensus_queue.cc:1059] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.013s
I20260709 13:09:58.129678 25313 raft_consensus.cc:2368] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17.
I20260709 13:09:58.130333 25392 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:58.110550 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:58.095381 30559 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:58.097424 30518 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:09:58.146548 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:58.159909 25936 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:58.124825 30666 consensus_queue.cc:1059] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.021s
I20260709 13:09:58.111910 30731 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:58.131605 30667 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:58.149876 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:58.154623 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:58.090495 30512 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:09:58.171614 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:58.107352 30728 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:58.146373 30655 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:58.147539 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:58.234475 25313 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:58.126132 30652 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:09:58.149499 25246 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:09:58.165884 25945 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:09:58.166592 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:58.169518 25312 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 16: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:58.240839 25321 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:58.164799 30657 raft_consensus.cc:3055] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Stepping down as leader of term 16
W20260709 13:09:58.236454 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:58.157981 30729 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:58.125110 30688 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:58.176007 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:09:58.222553 30756 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.268s	user 0.001s	sys 0.000s
I20260709 13:09:58.165095 30559 raft_consensus.cc:697] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:58.254894 25316 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:58.222321 30599 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:58.258214 30745 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:09:58.259275 30761 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.190s	user 0.000s	sys 0.001s
W20260709 13:09:58.259948 30768 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 13:09:58.262002 25239 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:09:58.262584 25390 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
W20260709 13:09:58.263355 30606 consensus_meta.cc:287] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.666s	user 0.009s	sys 0.001s
I20260709 13:09:58.302278 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:58.252709 25244 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:58.303112 25312 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:09:58.267477 25392 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:09:58.259964 30688 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:58.245265 30483 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:58.310149 25244 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:09:58.294911 25321 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:09:58.313179 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:58.171792 30708 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:58.221863 30731 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:58.316519 25321 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:09:58.318377 30731 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:58.319355 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:58.320025 25321 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:58.285674 30591 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:58.320927 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:58.321704 25321 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:09:58.322446 30773 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:09:58.329692 30744 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:58.265851 30075 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:58.331487 30711 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:58.232452 30726 consensus_queue.cc:237] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:58.337301 30473 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:58.338148 30473 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:58.249869 30657 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:58.296442 30512 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:58.282541 30667 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:58.342417 30512 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:58.342061 30559 consensus_queue.cc:237] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:58.346318 30387 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:58.348086 30681 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:58.342756 25389 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
W20260709 13:09:58.350579 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:58.350749 25387 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 10. Current term is 11. Ops: []
W20260709 13:09:58.351822 30656 consensus_meta.cc:287] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.809s	user 0.009s	sys 0.000s
I20260709 13:09:58.352917 25393 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:09:58.353036 30681 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:58.349555 25358 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:58.342136 30657 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:58.349848 25321 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:58.355125 30657 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:09:58.355567 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:58.356472 25358 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:58.356694 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:58.357271 30688 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:09:58.358963 30728 consensus_queue.cc:1059] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:09:58.362982 30728 raft_consensus.cc:3055] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:09:58.363422 30728 raft_consensus.cc:740] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:58.364392 30728 consensus_queue.cc:260] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:58.359814 30687 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:58.366503 30728 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:58.344990 30512 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:58.367616 30512 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:58.368309 30512 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:58.359933 30656 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:09:58.361513 30075 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:58.372762 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:58.379276 25321 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:58.381793 30737 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:09:58.382109 30767 log.cc:927] Time spent T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.112s	user 0.000s	sys 0.004s
I20260709 13:09:58.373668 25358 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:58.389724 30727 consensus_queue.cc:260] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:58.394354 30681 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:58.394659 30727 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:58.396219 30547 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:58.401717 25319 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: [11.6-13.7]
I20260709 13:09:58.401973 25321 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:09:58.387933 30712 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16.
I20260709 13:09:58.402874 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:58.411628 30674 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:58.385628 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:58.399597 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:58.400665 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32994 (request call id 2384) took 1053 ms. Trace:
I20260709 13:09:58.397903 25358 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:58.414252 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:58.414644 25245 rpcz_store.cc:276] 0709 13:09:57.347342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:57.401740 (+ 54398us) service_pool.cc:224] Handling call
0709 13:09:58.400446 (+998706us) spinlock_profiling.cc:243] Waited 371 ms on lock 0x7b5400362c54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:58.400598 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":853777280}
W20260709 13:09:58.417320 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:58.397842 30656 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.014s
I20260709 13:09:58.415467 25358 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:58.418708 25313 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
W20260709 13:09:58.421221 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:58.421501 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 265122us
I20260709 13:09:58.420251 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:58.424113 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:58.424261 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 408778us
I20260709 13:09:58.426720 25358 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:58.427513 25312 raft_consensus.cc:2393] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 15: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:09:58.435365 25358 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:58.438673 25318 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:09:58.445570 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:58.450480 25908 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:58.453136 25209 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:58.454843 25209 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:58.415753 30674 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:58.461506 25209 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:58.451102 25358 leader_election.cc:400] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17.
I20260709 13:09:58.465415 25358 leader_election.cc:403] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 13:09:58.307163 30599 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:58.421763 30687 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:58.471154 25287 leader_election.cc:400] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16.
I20260709 13:09:58.466466 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:58.477550 25287 leader_election.cc:403] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 13:09:58.478088 25358 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:09:58.499291 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:58.528003 25287 leader_election.cc:400] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:09:58.530622 25287 leader_election.cc:403] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 13:09:58.452351 30516 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:09:58.452113 30688 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:09:58.462052 30547 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:58.459224 30305 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:09:58.509728 30652 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:58.514183 30590 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:58.466612 30760 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:58.460014 30731 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:09:58.476024 30565 consensus_meta.cc:287] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.579s	user 0.013s	sys 0.000s
I20260709 13:09:58.468488 30674 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:58.467834 30483 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:09:58.531576 25287 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:58.470009 30621 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17.
W20260709 13:09:58.533012 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 54991.3us
W20260709 13:09:58.561796 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 459729us
I20260709 13:09:58.479440 30534 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:09:58.565461 25315 raft_consensus.cc:606] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:58.540725 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 307413us
I20260709 13:09:58.545042 25313 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:09:58.473659 30781 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:09:58.501181 30606 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16.
W20260709 13:09:58.559355 25944 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.055s	sys 0.108s
I20260709 13:09:58.520879 30786 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:09:58.564597 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 104216us
I20260709 13:09:58.537420 30688 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:58.565286 25285 leader_election.cc:400] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16.
I20260709 13:09:58.535179 25316 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:09:58.573199 25285 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:58.549319 30727 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:09:58.567090 25237 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 17 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 13:09:58.566666 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:58.574424 25313 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:09:58.565908 25315 raft_consensus.cc:614] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:09:58.555738 30590 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:09:58.569849 25944 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.514s	user 0.055s	sys 0.108s
I20260709 13:09:58.551174 30516 raft_consensus.cc:697] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:09:58.576030 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 44566.8us
I20260709 13:09:58.576817 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:09:58.579880 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:58.581267 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:09:58.588284 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:58.588223 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:09:58.589681 25313 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:09:58.591212 25313 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:09:58.566161 30674 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:58.596087 25287 leader_election.cc:400] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16.
I20260709 13:09:58.600551 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:58.582866 25285 leader_election.cc:400] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:09:58.569756 30305 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:58.556654 30652 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:58.606113 25285 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:58.612908 30154 raft_consensus.cc:2764] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 13:09:58.565761 30534 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:58.575603 30688 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:58.624543 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:58.575031 30781 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 16 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:58.619737 30564 raft_consensus.cc:2764] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 13:09:58.656193 25313 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20260709 13:09:58.578055 30727 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:09:58.621119 30783 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 13:09:58.663362 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:58.623462 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:58.574921 30038 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:09:58.563733 30731 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:58.635594 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:58.629808 25314 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
W20260709 13:09:58.646772 25389 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2689) took 2468 ms (client timeout 1867 ms). Trace:
W20260709 13:09:58.685245 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 114793us
W20260709 13:09:58.685463 25389 rpcz_store.cc:269] 0709 13:09:56.178480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:56.359059 (+180579us) service_pool.cc:224] Handling call
0709 13:09:58.646570 (+2287511us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b54003e3d54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:58.646679 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4427494144}
I20260709 13:09:58.686844 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:58.687595 25389 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:58.624717 30534 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:09:58.647953 30781 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:58.586513 30565 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:09:58.650321 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:58.693818 25387 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:09:58.651307 30652 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:58.651795 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:58.652592 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:58.653165 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:58.703372 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:58.653239 30483 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:09:58.593014 30777 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:09:58.619922 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:58.656602 25313 pending_rounds.cc:85] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 9
I20260709 13:09:58.658187 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:58.659921 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:58.706506 25313 pending_rounds.cc:107] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 13:09:58.706837 25313 raft_consensus.cc:2889] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:09:58.707062 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 47225.7us
I20260709 13:09:58.707962 25287 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:58.662213 30727 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:58.586062 30786 raft_consensus.cc:697] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:58.680684 25241 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: []
I20260709 13:09:58.680450 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:58.681099 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:58.671916 25392 raft_consensus.cc:2410] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
W20260709 13:09:58.668416 30757 log.cc:927] Time spent T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.624s	user 0.003s	sys 0.000s
I20260709 13:09:58.646298 30516 consensus_queue.cc:237] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:58.689008 25389 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:09:58.708402 25909 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:58.689807 25212 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:58.716203 25909 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:58.716871 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
W20260709 13:09:58.717788 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:58.718030 30652 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:09:58.718451 30652 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:58.718889 30652 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:58.721683 30759 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:09:58.722235 30759 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:58.722659 30759 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:58.692297 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:58.621747 30746 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:58.694123 25908 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:58.691177 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 124449us
I20260709 13:09:58.708366 25941 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16. Ops: []
I20260709 13:09:58.726689 30746 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:58.711045 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:58.732386 30746 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:58.706153 30483 consensus_queue.cc:1243] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer a2a77ecf2c0941618b5bc79d4c04db61 log is divergent from this leader: its last log entry 14.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:09:58.748382 30523 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:58.748134 30786 consensus_queue.cc:237] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:58.704841 30564 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:58.749503 30564 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:58.709396 30656 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:58.750401 30656 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:58.752348 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 44472.2us
W20260709 13:09:58.704609 30788 log.cc:927] Time spent T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 13:09:58.624225 30305 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:58.655911 30793 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.050s
W20260709 13:09:58.771422 30794 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 13:09:58.748908 30523 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:58.762830 30789 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 13:09:58.763216 30656 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:58.763757 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:58.779201 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:09:58.779814 30765 log.cc:927] Time spent T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.704s	user 0.005s	sys 0.003s
I20260709 13:09:58.780297 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:58.760675 30778 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:09:58.766497 30658 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:09:58.782027 30760 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:58.782801 30760 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:58.783669 30760 consensus_queue.cc:1059] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:09:58.786820 30674 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:58.789482 30674 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:58.762002 25320 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
W20260709 13:09:58.770529 30780 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.199s	user 0.001s	sys 0.000s
I20260709 13:09:58.780920 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2387) took 1065 ms. Trace:
I20260709 13:09:58.789681 30622 raft_consensus.cc:3055] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:09:58.790287 30795 raft_consensus.cc:3055] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:09:58.790625 30523 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:58.790710 30797 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:09:58.792650 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:58.837217 25246 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 12: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:58.793313 30798 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.013s
I20260709 13:09:58.793972 30793 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:09:58.794627 30483 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:58.798979 30305 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:58.799471 30564 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:58.800819 30731 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:58.767238 30746 consensus_queue.cc:1059] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 13:09:58.809085 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:58.808617 25243 rpcz_store.cc:276] 0709 13:09:57.715432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:57.716269 (+   837us) service_pool.cc:224] Handling call
0709 13:09:57.734455 (+ 18186us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:57.738134 (+  3679us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:57.738166 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:57.738517 (+   351us) log.cc:844] Serialized 26 byte log entry
0709 13:09:57.959694 (+221177us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:57.959742 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:57.959821 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:58.779412 (+819591us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b2400120638. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973a4b9c4 00007f2973a49acb 00007f2973c04dc9 00007f2973c04a55 00007f2973c0409d 00007f297683610e 00007f297684d88c 00007f297684cbb6 00007f297684c7d8 00007f2976856221
0709 13:09:58.780626 (+  1214us) raft_consensus.cc:1685] finished
0709 13:09:58.780648 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:58.780861 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3200,"raft.queue_time_us":3345,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":10248,"spinlock_wait_cycles":32264576,"thread_start_us":165715,"threads_started":1,"wal-append.queue_time_us":286838}
I20260709 13:09:58.802349 30745 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:09:58.758899 30369 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 13:09:58.841094 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:58.837703 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:58.911794 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:58.913174 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:58.913865 25320 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:09:58.916312 30555 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:58.893711 30305 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:58.916939 30555 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:58.834828 30795 raft_consensus.cc:740] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:58.832880 30599 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:58.832480 30673 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:58.929742 25287 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:09:58.930608 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 211315us
W20260709 13:09:58.894166 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:58.918378 30746 consensus_queue.cc:1243] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer a2a77ecf2c0941618b5bc79d4c04db61 log is divergent from this leader: its last log entry 12.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 13:09:58.922093 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 228021us
I20260709 13:09:58.941833 25908 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:09:58.926676 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 199116us
I20260709 13:09:58.928360 25320 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:09:58.946740 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:58.947758 25387 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:09:58.949653 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:58.832463 30622 raft_consensus.cc:740] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:58.931457 25909 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:58.925022 25243 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:09:58.917879 30305 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:58.894713 30667 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:58.940126 25287 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:58.847949 30483 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:58.953923 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:58.965873 25243 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:09:58.954975 25387 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 14: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:58.839576 30652 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:58.926371 30590 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:58.961102 25287 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:58.969311 25320 raft_consensus.cc:3055] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:09:58.969897 25320 raft_consensus.cc:740] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:58.849092 30797 raft_consensus.cc:697] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:58.967828 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:58.970789 25387 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:09:58.972404 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2711) took 1199 ms. Trace:
I20260709 13:09:58.972978 25388 rpcz_store.cc:276] 0709 13:09:57.772553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:58.058917 (+286364us) service_pool.cc:224] Handling call
0709 13:09:58.059251 (+   334us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:58.059599 (+   348us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:58.059629 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:58.073605 (+ 13976us) log.cc:844] Serialized 52 byte log entry
0709 13:09:58.449728 (+376123us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:09:58.449772 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:58.451163 (+  1391us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:58.972086 (+520923us) raft_consensus.cc:1685] finished
0709 13:09:58.972118 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:58.972319 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4548,"raft.queue_time_us":254,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":9835,"thread_start_us":324044,"threads_started":1,"wal-append.queue_time_us":418617}
I20260709 13:09:58.974647 25287 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:58.976630 30038 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:09:58.976277 25320 consensus_queue.cc:260] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:58.977065 30038 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:58.977579 30038 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:58.958464 30793 raft_consensus.cc:697] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:58.985510 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:58.986161 30795 consensus_queue.cc:260] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:58.955754 30599 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:09:59.013237 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:59.018260 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 43758.7us
W20260709 13:09:59.023542 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:59.018644 25212 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:59.019981 25243 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:59.019454 25320 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:58.928769 30688 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:59.030956 25239 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 15: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:58.964260 30744 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:09:58.936170 30673 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:59.044028 25320 raft_consensus.cc:1275] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
I20260709 13:09:59.044391 25320 pending_rounds.cc:85] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 6
I20260709 13:09:59.044638 25320 pending_rounds.cc:107] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 13:09:59.044840 25320 raft_consensus.cc:2889] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:09:59.031067 30767 log.cc:927] Time spent T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 13:09:59.021443 30565 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:09:59.032820 30808 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:59.051465 25390 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:09:59.019822 30617 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:09:59.038091 30622 consensus_queue.cc:260] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:59.077462 30797 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:59.053632 25243 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:09:59.079872 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:59.080526 25243 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:09:59.080633 30555 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:59.081560 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:59.052327 30807 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.006s
I20260709 13:09:59.057457 30483 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:59.086329 25238 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14. Ops: []
I20260709 13:09:59.058208 30688 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:59.056361 30673 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:59.060393 30534 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:59.051549 30565 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:09:59.061231 30811 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 13:09:59.103893 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 90770.3us
I20260709 13:09:59.065893 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:59.067081 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:09:59.068169 30659 raft_consensus.cc:3055] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 13:09:59.069613 30803 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
I20260709 13:09:59.093734 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:59.070815 30590 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:59.071305 30793 consensus_queue.cc:237] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:59.071831 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:59.073617 30305 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:09:59.073770 30657 consensus_meta.cc:287] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.718s	user 0.009s	sys 0.004s
I20260709 13:09:59.074616 30813 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:59.076642 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:59.036401 30795 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:59.080236 30759 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:59.060241 30574 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:09:59.088899 25238 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 13:09:59.205025 25240 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16. Ops: []
I20260709 13:09:59.108310 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:59.107658 25394 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:09:59.121438 25319 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:09:59.128172 25314 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260709 13:09:59.104615 30565 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:09:59.207512 30800 log.cc:927] Time spent T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.172s	user 0.000s	sys 0.000s
I20260709 13:09:59.208344 25394 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 15 index: 7. (index mismatch)
I20260709 13:09:59.209827 25394 pending_rounds.cc:85] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 11
I20260709 13:09:59.210721 25394 pending_rounds.cc:107] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260709 13:09:59.211007 25394 raft_consensus.cc:2889] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:09:59.206175 25313 raft_consensus.cc:1813] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Handling vote request from an unknown peer a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:59.215510 25313 raft_consensus.cc:2410] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 13:09:59.056396 30727 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:59.219334 30727 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:59.218825 30038 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:59.220386 25320 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:09:59.221290 25391 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:09:59.221675 25242 raft_consensus.cc:1275] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:09:59.197413 25389 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16. Ops: []
W20260709 13:09:59.222949 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 271427us
I20260709 13:09:59.199766 30534 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:59.200253 30813 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:59.110061 30659 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:59.114938 30811 raft_consensus.cc:697] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:59.221863 30681 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:59.087790 30808 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:59.253048 25313 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:09:59.257120 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:59.206116 30759 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:59.263274 30812 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:59.263911 30812 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:59.265230 30806 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:09:59.266446 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49638 (request call id 35) took 1382 ms. Trace:
I20260709 13:09:59.266927 25315 rpcz_store.cc:276] 0709 13:09:57.884561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:57.884699 (+   138us) service_pool.cc:224] Handling call
0709 13:09:59.266185 (+1381486us) spinlock_profiling.cc:243] Waited 680 ms on lock 0x7b5400276354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cac853 00007f297bc7a60a 00007f29752007d4 00007f2975200762 00007f2975200691 00007f297520060c 00007f29751ff8c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:59.266356 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1563770240}
I20260709 13:09:59.268903 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:09:59.270027 30749 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:59.270749 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:09:59.270551 30749 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:59.270805 25317 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:09:59.270064 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2509) took 1361 ms. Trace:
I20260709 13:09:59.110774 30483 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.130s
I20260709 13:09:59.273940 25318 rpcz_store.cc:276] 0709 13:09:57.908577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:57.908745 (+   168us) service_pool.cc:224] Handling call
0709 13:09:59.269857 (+1361112us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b5400088ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:59.269986 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1133087104}
I20260709 13:09:59.275506 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:59.276125 25318 raft_consensus.cc:2376] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15. Re-sending same reply.
I20260709 13:09:59.278003 25318 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:09:59.273063 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2532) took 1128 ms. Trace:
I20260709 13:09:59.280726 25317 rpcz_store.cc:276] 0709 13:09:58.144345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:58.451046 (+306701us) service_pool.cc:224] Handling call
0709 13:09:59.270361 (+819315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:59.271370 (+  1009us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:59.272822 (+  1452us) spinlock_profiling.cc:243] Waited 818 ms on lock 0x7b5400088ed0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:59.272991 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1881393024}
I20260709 13:09:59.278975 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:59.239761 30617 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.049s
I20260709 13:09:59.300129 25313 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
W20260709 13:09:59.299377 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 367974us
I20260709 13:09:59.306314 25243 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 16 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260709 13:09:59.273344 30681 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:59.310365 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:59.316120 30038 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:59.317261 30815 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:59.330592 30808 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:59.320276 30811 consensus_queue.cc:237] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:59.311367 30646 raft_consensus.cc:3055] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:09:59.335445 30646 raft_consensus.cc:740] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:09:59.339299 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:59.341399 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:59.342425 25938 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 16: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:59.350116 30813 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:59.296343 30659 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:59.353817 30659 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:59.356511 30645 raft_consensus.cc:3055] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:09:59.356024 30812 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:59.364848 30645 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:09:59.427436 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:59.454840 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:59.458777 30483 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:59.460662 30483 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:59.466529 30483 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:59.369431 30565 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:59.471117 30494 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:09:59.472222 30494 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:59.472805 30565 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:59.479035 30645 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:59.499156 25938 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:09:59.502351 30512 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:59.489070 30645 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:09:59.507990 30564 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 12 is higher.
W20260709 13:09:59.514410 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:59.517503 30559 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:59.518447 30559 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:59.518882 30494 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:59.496804 30473 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:59.532116 25285 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:59.533690 30473 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:09:59.535018 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:59.535138 30652 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:59.537030 30559 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:59.545650 30473 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:09:59.547771 30663 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:09:59.549571 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:59.552578 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:59.553544 30682 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:59.559847 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:59.553840 25240 raft_consensus.cc:2393] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 16: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:59.562350 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2703) took 1791 ms. Trace:
I20260709 13:09:59.563019 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:09:59.563956 25242 raft_consensus.cc:2393] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 16: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:59.562862 25391 rpcz_store.cc:276] 0709 13:09:57.770347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:57.770459 (+   112us) service_pool.cc:224] Handling call
0709 13:09:57.770744 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:59.221675 (+1450931us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:59.561858 (+340183us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b54001216d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:59.562155 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3334865664}
I20260709 13:09:59.577014 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:59.577630 30599 consensus_queue.cc:1059] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:09:59.579197 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:59.579293 30599 raft_consensus.cc:3055] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:09:59.579996 30599 raft_consensus.cc:740] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:09:59.578135 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:59.581272 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:59.593714 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:59.600009 30559 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:59.600850 30559 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:59.603479 25237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2386) took 1890 ms. Trace:
I20260709 13:09:59.604081 25237 rpcz_store.cc:276] 0709 13:09:57.714912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:57.715543 (+   631us) service_pool.cc:224] Handling call
0709 13:09:57.716000 (+   457us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:58.574272 (+858272us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:59.602697 (+1028425us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b5400363b54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:59.603223 (+   526us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1956577024}
I20260709 13:09:59.607075 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:59.608335 25242 raft_consensus.cc:2410] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 13:09:59.609485 25942 raft_consensus.cc:2393] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:09:59.609678 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:59.611322 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:59.614143 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:09:59.611694 30646 consensus_queue.cc:260] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:59.621696 30746 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:59.624313 30746 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:59.629544 30746 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:59.632768 30736 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:59.633675 30736 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:59.616856 25941 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:09:59.612773 30559 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:59.637621 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:59.617103 30646 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:59.681303 30726 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:09:59.682502 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 950) took 1258 ms. Trace:
I20260709 13:09:59.681310 30559 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:59.682965 25245 rpcz_store.cc:276] 0709 13:09:58.423997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:58.425686 (+  1689us) service_pool.cc:224] Handling call
0709 13:09:58.426111 (+   425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:58.426335 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:09:58.457564 (+ 31229us) log.cc:844] Serialized 14 byte log entry
0709 13:09:59.541571 (+1084007us) spinlock_profiling.cc:243] Waited 2.95 ms on lock 0x7b18001b2bc0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f297685223a 00007f2976cbca2a
0709 13:09:59.680554 (+138983us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:59.680646 (+    92us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:09:59.680816 (+   170us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:59.682157 (+  1341us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:59.682341 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":31035,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":3214,"raft.queue_time_us":5519,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":209,"spinlock_wait_cycles":6795008,"thread_start_us":112384,"threads_started":1}
W20260709 13:09:59.685067 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:09:59.685729 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:59.691757 25317 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:09:59.692555 30726 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:59.693070 30726 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:09:59.698243 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 2326) took 1074 ms. Trace:
I20260709 13:09:59.698900 25239 rpcz_store.cc:276] 0709 13:09:58.623338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:58.625218 (+  1880us) service_pool.cc:224] Handling call
0709 13:09:59.697716 (+1072498us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b54003a98d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:59.698011 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":753923328}
I20260709 13:09:59.684644 30559 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:59.677241 25937 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:09:59.701066 25244 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 13:09:59.701272 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:59.709230 25938 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
W20260709 13:09:59.711239 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 580152us
I20260709 13:09:59.712080 25237 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:09:59.712780 25359 leader_election.cc:400] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17.
I20260709 13:09:59.715512 30305 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:09:59.716332 30559 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:59.717108 30559 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:59.751770 30516 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:59.717926 30656 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:59.753765 30622 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:09:59.767501 30656 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:59.769443 30736 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:59.770043 25313 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
W20260709 13:09:59.754338 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 694495us
W20260709 13:09:59.758883 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 53676.4us
I20260709 13:09:59.760355 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 893) took 1436 ms. Trace:
I20260709 13:09:59.761276 25239 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 13:09:59.761945 30559 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:59.766422 25321 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:09:59.754243 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2714) took 1142 ms. Trace:
I20260709 13:09:59.747578 30599 consensus_queue.cc:260] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 15.7, Last appended by leader: 7, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:59.802095 30785 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:09:59.804514 30760 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:09:59.794097 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 44336.2us
W20260709 13:09:59.808187 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:59.805325 30681 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:59.777072 30512 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:59.752687 30516 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:59.772856 25386 rpcz_store.cc:276] 0709 13:09:58.590928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:58.591052 (+   124us) service_pool.cc:224] Handling call
0709 13:09:58.591288 (+   236us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:58.591491 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:09:58.804766 (+213275us) log.cc:844] Serialized 14 byte log entry
0709 13:09:59.696500 (+891734us) spinlock_profiling.cc:243] Waited 9.87 ms on lock 0x7b180045d120. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b3fc0a 00007f297d27f219 00007f2973bc7a40 00007f2973bc24e4 00007f2973bc2455 00007f2976caa060 00007f2976cbc716 00007f2976ccd287 00007f2976ccd208
0709 13:09:59.696903 (+   403us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:59.697018 (+   115us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:09:59.701769 (+  4751us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:59.708368 (+  6599us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:59.733000 (+ 24632us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7001,"prepare.queue_time_us":384763,"prepare.run_cpu_time_us":1909,"prepare.run_wall_time_us":30238,"raft.queue_time_us":699062,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":332,"spinlock_wait_cycles":22698880,"thread_start_us":422611,"threads_started":2,"wal-append.queue_time_us":280165}
I20260709 13:09:59.822685 30681 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:59.823695 30681 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:59.826851 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:59.827181 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:59.827976 25319 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:09:59.828104 25320 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:09:59.805182 30760 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:59.830696 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 47145.6us
I20260709 13:09:59.831234 30760 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:59.831775 25285 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:59.833369 30516 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:59.833877 30599 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:59.772389 25312 rpcz_store.cc:276] 0709 13:09:58.323483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:58.464709 (+141226us) service_pool.cc:224] Handling call
0709 13:09:58.465333 (+   624us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:09:58.465592 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:09:58.483847 (+ 18255us) log.cc:844] Serialized 14 byte log entry
0709 13:09:59.557864 (+1074017us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b1800435a60. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f297685223a 00007f2976cbca2a
0709 13:09:59.707621 (+149757us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:09:59.728492 (+ 20871us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:09:59.733792 (+  5300us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:59.746697 (+ 12905us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:59.760170 (+ 13473us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":32756,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":17142,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":123,"spinlock_wait_cycles":34028800,"thread_start_us":111702,"threads_started":1,"wal-append.queue_time_us":1223260}
I20260709 13:09:59.827232 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:59.813436 30656 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:59.834056 30746 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:09:59.859020 25212 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:59.859719 30749 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:59.856200 25359 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:09:59.867125 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:09:59.868650 30652 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:09:59.869205 30652 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:59.869599 30652 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:09:59.857055 25320 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:09:59.882211 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 952) took 1223 ms. Trace:
I20260709 13:09:59.851218 30512 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:09:59.889181 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 50674.4us
I20260709 13:09:59.883646 25244 rpcz_store.cc:276] 0709 13:09:58.659260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:58.764380 (+105120us) service_pool.cc:224] Handling call
0709 13:09:59.700536 (+936156us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:59.709237 (+  8701us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:59.879388 (+170151us) spinlock_profiling.cc:243] Waited 936 ms on lock 0x7b54003a98d0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:59.882118 (+  2730us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2151872768}
W20260709 13:09:59.912341 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 73517.8us
I20260709 13:09:59.916824 30534 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:09:59.917598 30534 raft_consensus.cc:697] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:09:59.918793 30534 consensus_queue.cc:237] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:09:59.923208 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:09:59.923157 30843 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:59.903910 30681 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:59.926684 30600 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:59.927131 25212 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:09:59.927507 30600 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:59.928305 30369 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:09:59.932726 25314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2537) took 1307 ms. Trace:
I20260709 13:09:59.933104 30600 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:59.934504 25909 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:09:59.925757 30749 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:09:59.943342 30599 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:09:59.943257 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:09:59.924656 25321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2534) took 1746 ms (client timeout 2228 ms). Trace:
I20260709 13:09:59.950110 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:09:59.953011 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:09:59.955786 30749 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:59.956483 30786 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:09:59.957192 30786 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:59.958128 30749 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:09:59.924031 30843 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:09:59.959334 30786 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:09:59.961122 30843 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:09:59.969483 25284 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:59.971339 25284 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:59.971683 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:09:59.972679 25312 raft_consensus.cc:2410] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 13:09:59.972574 30663 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:09:59.973593 30387 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 13:09:59.950392 25321 rpcz_store.cc:269] 0709 13:09:58.175451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:58.402636 (+227185us) service_pool.cc:224] Handling call
0709 13:09:59.899914 (+1497278us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54000870d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:09:59.921784 (+ 21870us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2998674304}
W20260709 13:09:59.974447 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 76773.8us
I20260709 13:09:59.975718 30477 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:09:59.977460 30477 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:09:59.978632 25387 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:09:59.979357 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:09:59.979647 25945 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:09:59.980789 30494 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:09:59.981398 30494 raft_consensus.cc:697] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:09:59.981498 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:09:59.982650 30494 consensus_queue.cc:237] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:09:59.985306 25387 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 13:09:59.987869 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 53517.8us
I20260709 13:09:59.982650 25386 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 16: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:09:59.991835 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:00.001978 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 184492us
W20260709 13:10:00.005734 30837 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
I20260709 13:10:00.011335 30512 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:00.013273 30646 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:00.013831 30646 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:00.014205 30646 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:00.019177 25391 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
I20260709 13:09:59.944600 25314 rpcz_store.cc:276] 0709 13:09:58.604942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:59.207105 (+602163us) service_pool.cc:224] Handling call
0709 13:09:59.207515 (+   410us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:09:59.615992 (+408477us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:59.616045 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:09:59.616693 (+   648us) log.cc:844] Serialized 26 byte log entry
0709 13:09:59.677500 (+ 60807us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:09:59.677553 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:59.893572 (+216019us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b54000f9954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:09:59.893620 (+    48us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:09:59.893660 (+    40us) raft_consensus.cc:1685] finished
0709 13:09:59.893674 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:09:59.912232 (+ 18558us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1681,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":7461,"spinlock_wait_cycles":922934656,"thread_start_us":60318,"threads_started":1,"wal-append.queue_time_us":112198}
I20260709 13:09:59.951673 30559 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:00.020386 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:00.022130 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:00.022979 25392 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:10:00.023904 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.024560 25243 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:00.026428 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:00.026993 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 47883.2us
I20260709 13:10:00.028056 30746 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:10:00.029265 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:00.030542 25284 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:00.031376 30746 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:00.031952 25284 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:00.032778 30387 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:00.034160 25359 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:00.035542 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 88405.6us
I20260709 13:10:00.037536 30843 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:00.040748 25394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2722) took 1014 ms. Trace:
I20260709 13:10:00.041602 30599 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:00.041733 30726 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.042814 30555 consensus_queue.cc:1059] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.134s
I20260709 13:10:00.043233 25393 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 10. (term mismatch)
I20260709 13:10:00.045641 25285 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:00.024829 30369 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:00.056829 25393 pending_rounds.cc:85] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 9
W20260709 13:10:00.052603 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 82621.2us
I20260709 13:10:00.053431 25320 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:00.064081 25393 pending_rounds.cc:107] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 16.10
I20260709 13:10:00.071486 25393 raft_consensus.cc:2889] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:00.072539 30785 consensus_queue.cc:1059] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:10:00.073071 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 2022) took 1369 ms. Trace:
I20260709 13:10:00.073504 25393 rpcz_store.cc:276] 0709 13:09:58.704146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:58.781110 (+ 76964us) service_pool.cc:224] Handling call
0709 13:09:58.839054 (+ 57944us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:00.072159 (+1233105us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:00.072815 (+   656us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54003d45d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:00.073019 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2768085888}
I20260709 13:10:00.048432 30843 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:00.075910 30843 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:00.076190 30387 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:00.076275 30749 consensus_queue.cc:1059] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:00.077335 30807 raft_consensus.cc:3055] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:00.077458 30749 raft_consensus.cc:3055] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:00.077786 30807 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:00.077993 30749 raft_consensus.cc:740] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:00.078999 30599 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:00.079241 30749 consensus_queue.cc:260] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.045766 25394 rpcz_store.cc:276] 0709 13:09:59.026431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:59.209239 (+182808us) service_pool.cc:224] Handling call
0709 13:09:59.209590 (+   351us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:09:59.698796 (+489206us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:09:59.698856 (+    60us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:09:59.699849 (+   993us) log.cc:844] Serialized 52 byte log entry
0709 13:09:59.747879 (+ 48030us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:09:59.747906 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:09:59.751596 (+  3690us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:00.040360 (+288764us) raft_consensus.cc:1685] finished
0709 13:10:00.040386 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:00.040588 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":486704,"raft.queue_time_us":503169,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":153,"thread_start_us":47503,"threads_started":1,"wal-append.queue_time_us":338029}
I20260709 13:10:00.080735 30749 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:00.083175 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.083910 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:00.084751 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:00.076743 30387 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:00.071194 25359 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:00.072983 30726 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:00.101550 30600 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:00.102451 30600 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:00.105106 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:00.108265 25321 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:00.093262 30387 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:00.118577 25243 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:10:00.107005 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:00.120416 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 63760.1us
I20260709 13:10:00.117552 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.097165 30785 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:00.123566 25319 raft_consensus.cc:2410] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 17 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:10:00.123981 25312 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:10:00.132539 25284 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:00.133669 30599 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:00.133754 25318 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:10:00.092664 30645 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 13:10:00.112520 30663 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:00.112151 30600 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:00.109666 30656 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:00.140621 25359 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:00.145875 30785 consensus_queue.cc:1059] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:10:00.125538 30622 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:00.151702 25237 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:10:00.142477 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:00.079031 30807 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.149019 25394 raft_consensus.cc:1240] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:00.122861 30811 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.047s
I20260709 13:10:00.151165 30622 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:00.171486 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 66918us
I20260709 13:10:00.179237 25911 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:00.184531 25319 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20260709 13:10:00.190249 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:00.186659 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:00.191671 25321 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:10:00.192966 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:00.193459 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:00.195129 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:00.171696 30600 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:00.196768 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:00.197240 25315 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10. Ops: []
I20260709 13:10:00.196009 30858 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:10:00.198205 25143 catalog_manager.cc:5697] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 10 to 14. New cstate: current_term: 14 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:00.162930 30760 raft_consensus.cc:3055] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:00.205482 30760 raft_consensus.cc:740] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:00.206617 30760 consensus_queue.cc:260] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.207828 30760 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:00.209348 30616 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:10:00.167173 30483 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:00.196815 30659 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:10:00.214213 30850 log.cc:927] Time spent T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 13:10:00.181859 30622 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:00.166895 30656 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:00.224614 30659 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:00.232913 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:00.233261 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:00.232668 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:00.232883 30659 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:00.185810 30663 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:00.284526 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:00.226490 30807 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:00.293869 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:00.299043 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 167717us
I20260709 13:10:00.297529 25944 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:00.311632 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:00.317435 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:00.317994 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:00.318125 25937 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:00.318524 30778 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:00.319082 30778 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:00.321244 30559 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:00.321494 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:00.321933 30559 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:00.324299 30559 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:00.324501 30600 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:10:00.325301 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:00.296893 30483 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.328675 30483 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:00.330606 25312 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:00.347155 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:00.348304 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:00.238819 30477 raft_consensus.cc:3055] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:10:00.332690 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:00.333770 25313 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 13:10:00.334138 25319 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:10:00.334481 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:00.418385 25316 raft_consensus.cc:2410] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 16 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 6.
I20260709 13:10:00.305733 30645 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:00.342635 30656 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:00.344933 30652 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.361232 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:00.382978 30555 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:00.414426 25320 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:00.348874 25312 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:10:00.415483 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:00.416610 25317 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
W20260709 13:10:00.417037 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 79207.2us
I20260709 13:10:00.353722 30600 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:00.389457 30305 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:00.426578 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:00.428980 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:00.406697 30483 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:00.429821 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:00.430590 25937 raft_consensus.cc:2393] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 15: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:00.406412 30778 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:00.438409 25321 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:10:00.441625 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:00.441079 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:00.449745 25936 raft_consensus.cc:2410] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 13:10:00.417479 30477 raft_consensus.cc:740] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:00.454901 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:00.456020 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.452013 25239 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:00.428031 30656 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:00.423959 30858 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.033s
I20260709 13:10:00.464886 30599 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:00.474869 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:00.466814 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.440483 30512 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:00.470158 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:00.485684 30857 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.059s
I20260709 13:10:00.462559 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:00.459210 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 280095us
I20260709 13:10:00.461812 30600 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:00.484401 25313 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 13:10:00.482684 30652 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:00.471266 30746 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:00.483991 30483 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.522959 30599 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:00.524184 30483 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:00.427292 30494 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.062s
W20260709 13:10:00.526217 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 63800.4us
I20260709 13:10:00.526503 30494 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:00.527985 30726 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:10:00.528235 30559 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:10:00.530287 30559 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:10:00.468714 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.530742 30600 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:00.531246 25245 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:10:00.467773 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.532394 25246 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:00.470860 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:00.468583 30749 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:00.533869 25318 raft_consensus.cc:2410] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 15 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 13:10:00.466535 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.497459 25243 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:00.508543 25316 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:10:00.508796 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:00.479202 30622 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:00.513062 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:00.446133 30516 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:00.455001 30555 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.518498 30387 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:00.563378 30387 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:00.566067 25320 raft_consensus.cc:2393] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 15: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:10:00.521404 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 106504us
I20260709 13:10:00.527441 30494 consensus_queue.cc:1059] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.003s
W20260709 13:10:00.461874 30853 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.125s	user 0.000s	sys 0.005s
I20260709 13:10:00.496984 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:00.449846 30645 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.521880 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:10:00.533617 30749 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.450222 30305 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:00.542095 30599 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:00.551777 25143 catalog_manager.cc:5697] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 9 to 12, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 12 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:00.552707 30622 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:00.556766 25241 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:10:00.556686 30856 consensus_queue.cc:1059] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:10:00.558647 30555 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:00.528599 25209 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:10:00.572718 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 70714.5us
I20260709 13:10:00.547845 30857 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:00.582706 25317 raft_consensus.cc:2410] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:10:00.577100 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.551259 30656 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:00.590744 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 62235us
I20260709 13:10:00.591262 25911 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:00.591573 30483 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:10:00.591912 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:00.602738 25209 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:00.591903 30857 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.591995 30811 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:00.594367 30646 consensus_meta.cc:287] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.580s	user 0.005s	sys 0.010s
I20260709 13:10:00.599620 30749 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:00.510579 30858 consensus_queue.cc:1059] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.070s
I20260709 13:10:00.609434 30483 raft_consensus.cc:697] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:10:00.602733 30843 consensus_meta.cc:287] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.526s	user 0.004s	sys 0.006s
I20260709 13:10:00.605463 30599 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:00.613240 30555 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:00.609575 30811 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.666227 25945 raft_consensus.cc:2410] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:10:00.663808 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:10:00.670351 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 431161us
I20260709 13:10:00.618276 30387 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:00.674639 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:00.606037 30736 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.056s
W20260709 13:10:00.675869 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:00.618685 30516 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:00.665797 30555 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:00.690564 25243 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:00.675473 25392 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:00.624269 30785 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.052s
I20260709 13:10:00.619069 30646 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.698851 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:00.693229 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.674737 30387 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.701506 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.701711 30483 consensus_queue.cc:237] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:00.688329 30616 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:10:00.704246 30681 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:00.708613 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:00.657665 30843 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:00.730971 30843 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:00.732523 25393 raft_consensus.cc:2368] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17.
W20260709 13:10:00.715384 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 126485us
I20260709 13:10:00.733853 30811 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:00.734268 25911 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:00.737735 30616 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:00.738566 30616 consensus_queue.cc:1059] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:10:00.739594 30616 raft_consensus.cc:3055] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:10:00.739991 30616 raft_consensus.cc:740] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:00.741310 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:00.719336 30534 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:00.742033 25392 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:00.742452 30534 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:00.744196 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:00.744635 30534 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:00.744957 25388 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:00.745189 30512 raft_consensus.cc:3055] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:10:00.745606 30512 raft_consensus.cc:740] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:00.746534 30512 consensus_queue.cc:260] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.747042 30305 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:00.748044 30512 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:00.748248 25389 raft_consensus.cc:2410] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 13:10:00.749195 30659 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.750581 25390 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 17: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:00.751663 30659 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:00.740885 30616 consensus_queue.cc:260] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.752676 30857 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:00.687400 30599 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:00.706282 30555 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:00.710019 30516 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:00.720770 25237 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:00.688762 30494 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:00.727891 30861 log.cc:927] Time spent T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 13:10:00.694139 25143 catalog_manager.cc:5697] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 10 to 13. New cstate: current_term: 13 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:10:00.728897 30855 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 13:10:00.766718 30387 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:00.762687 30646 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:00.767210 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:00.769672 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:00.767565 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:00.753600 30616 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:00.753624 25911 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:00.775408 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.753072 30645 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:10:00.787415 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:00.799180 25246 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 14
I20260709 13:10:00.753551 25358 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:00.783268 25389 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:10:00.792145 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:00.790287 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.790374 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.782918 25388 raft_consensus.cc:2410] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 13 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 13:10:00.784940 25243 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:00.778575 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:00.789610 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:00.778935 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:00.802452 25386 raft_consensus.cc:2393] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 14: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:00.759088 30599 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:00.765527 30663 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:00.763454 30516 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:00.804390 25359 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:00.777936 30534 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:00.786435 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.810106 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.817422 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:00.818037 25237 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:00.789016 30646 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:00.813762 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.779707 30746 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:00.821816 25284 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:00.757675 30656 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:00.809788 25143 catalog_manager.cc:5697] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 4 to 10, term changed from 12 to 15, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to <none>, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted, no longer has a pending config: VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65), VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66), VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 15 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:00.813781 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:00.827167 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 73715.2us
I20260709 13:10:00.832525 25285 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:00.845054 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:00.846177 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:00.833165 25394 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:00.849907 25359 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:00.850230 25317 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: [11.6-13.7]
W20260709 13:10:00.805512 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 219066us
I20260709 13:10:00.838021 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:00.854631 25243 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:00.855120 25312 raft_consensus.cc:2410] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 16 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 13:10:00.791602 30369 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.041s
I20260709 13:10:00.855916 25313 raft_consensus.cc:2410] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 16 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 13:10:00.815037 30843 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:00.818601 30726 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:00.861479 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.787448 30494 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.797610 30652 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.041s
I20260709 13:10:00.865918 25358 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:00.833303 30646 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:00.760316 30477 consensus_queue.cc:260] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:00.856287 30516 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:10:00.867882 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 46152.6us
I20260709 13:10:00.871824 25359 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:00.845424 30659 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:00.875254 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.877949 25143 catalog_manager.cc:5697] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 9 to 11, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 11 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:00.882511 25393 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:10:00.885620 25246 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:00.858255 30599 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:00.893718 25242 raft_consensus.cc:2393] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 16: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:00.897959 25911 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:00.866055 30786 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:00.923712 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 60706.6us
I20260709 13:10:00.923044 25239 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:10:00.868539 30864 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:00.832068 30534 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:00.871222 30862 raft_consensus.cc:3055] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:00.876773 30646 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:00.869206 30681 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:00.939076 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:00.869889 30843 raft_consensus.cc:697] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:00.861672 30746 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:00.883963 30516 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:00.883467 30749 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:10:00.891026 30760 consensus_meta.cc:287] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.683s	user 0.013s	sys 0.001s
I20260709 13:10:00.871301 30622 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.074s
I20260709 13:10:00.895730 30305 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:10:00.976403 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 88552us
I20260709 13:10:00.979647 30516 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:00.943048 30600 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:00.971251 30681 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:00.986565 25358 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:00.988668 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 49664.3us
I20260709 13:10:00.898207 30659 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:00.962844 30785 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:00.963936 30862 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:10:00.995152 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 390624us
I20260709 13:10:00.924150 30726 consensus_queue.cc:1059] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
W20260709 13:10:00.923614 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 144423us
I20260709 13:10:00.989876 25143 catalog_manager.cc:5697] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 12 to 13, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 13 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:10:01.038599 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 52136.7us
I20260709 13:10:00.979959 30749 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:00.924336 30599 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:01.056252 25320 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
W20260709 13:10:01.026674 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 140231us
I20260709 13:10:01.057197 30866 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:01.057351 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:01.057760 30866 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:01.058205 30866 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:01.058950 25143 catalog_manager.cc:5697] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 7 to 12. New cstate: current_term: 12 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:10:01.060238 30876 log.cc:927] Time spent T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 13:10:01.043183 30785 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:10:01.061673 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:01.062887 25393 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260709 13:10:01.063081 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 138662us
I20260709 13:10:00.989195 30858 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:01.054034 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:01.054497 30867 log.cc:927] Time spent T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.188s	user 0.000s	sys 0.001s
I20260709 13:10:01.061077 25143 catalog_manager.cc:5697] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 9 to 12, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 12 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:01.067929 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:01.068274 25287 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:01.047284 25243 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:01.028357 30681 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:10:01.062391 30872 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
W20260709 13:10:01.114851 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 46651.4us
W20260709 13:10:01.047849 25212 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.524s (SENT) [suppressed 7 similar messages]
I20260709 13:10:01.108081 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:01.110158 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:00.980995 30786 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:01.048106 30600 raft_consensus.cc:697] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:01.112838 25359 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:01.124395 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:01.107167 25143 catalog_manager.cc:5697] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 9 to 12, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 12 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:01.125257 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:01.126760 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:01.130632 25944 raft_consensus.cc:2410] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 16 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 13:10:01.027132 30645 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:01.061213 30749 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:00.970903 30864 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:01.093729 30663 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:01.099318 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:01.137961 25287 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:01.142969 25395 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:10:01.052129 30305 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:10:01.061645 30785 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:01.150384 25937 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:01.164176 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:01.050526 30736 consensus_queue.cc:1059] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.086s
I20260709 13:10:01.157539 25941 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
W20260709 13:10:01.158232 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 45456.9us
I20260709 13:10:01.164817 25390 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:01.144493 30645 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:01.167840 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 234310us
I20260709 13:10:01.178726 25244 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 13:10:01.179725 25143 catalog_manager.cc:5697] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 10 to 12, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 12 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:01.187655 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:01.188759 25243 raft_consensus.cc:2393] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 14: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:01.175644 30645 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:01.220786 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 172944us
I20260709 13:10:01.144748 30858 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:01.179157 30534 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:01.188665 30622 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 13:10:01.232060 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 166212us
W20260709 13:10:01.230170 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 117400us
I20260709 13:10:01.167608 30682 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260709 13:10:01.167392 30663 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:01.233187 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:01.212375 30494 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:01.230333 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:01.253222 25389 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 13:10:01.255342 25359 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:01.261945 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:01.264564 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 126787us
I20260709 13:10:01.265688 25287 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:01.264710 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:01.272131 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:01.268342 25143 catalog_manager.cc:5697] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 8 to 10, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 10 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:01.277565 25938 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20260709 13:10:01.277994 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:01.165220 30843 consensus_queue.cc:237] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:01.277613 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:01.253348 30369 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:01.291337 25143 catalog_manager.cc:5697] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 8 to 12. New cstate: current_term: 12 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:10:01.290144 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 116248us
I20260709 13:10:01.267293 30534 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:01.261270 30622 raft_consensus.cc:697] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:01.302048 25143 catalog_manager.cc:5697] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 7 to 12, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 12 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:01.306514 25909 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:01.274240 30746 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:01.311932 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:01.315879 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:01.314293 25140 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39596 (request call id 286) took 23173 ms (client timeout 15000 ms). Trace:
I20260709 13:10:01.174811 30862 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:01.320279 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:01.308761 25242 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:10:01.320240 25143 catalog_manager.cc:5697] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 8 to 10, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 10 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:01.318387 25391 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:10:01.343612 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:01.346489 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:01.336941 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:01.315879 30659 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:01.261884 30600 consensus_queue.cc:237] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:01.357736 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:01.328533 30652 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:01.329231 30483 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:01.359886 25317 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:01.366580 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:01.332489 30534 raft_consensus.cc:697] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:01.373320 30878 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.183s
I20260709 13:10:01.375368 30881 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:01.376055 30884 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:10:01.376314 30865 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.305s	user 0.000s	sys 0.001s
I20260709 13:10:01.376590 30884 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:01.377032 30884 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:01.377787 25319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2621) took 1042 ms. Trace:
I20260709 13:10:01.375562 30896 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:01.392987 30896 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:01.393524 30663 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:01.395661 30896 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:01.383481 30900 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:01.396778 30900 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:01.383632 30887 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 13:10:01.384620 30880 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:01.399679 30880 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:01.401974 30880 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:01.383700 25143 catalog_manager.cc:5697] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 9 to 11, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 11 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:10:01.385453 30879 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
I20260709 13:10:01.389199 30898 consensus_queue.cc:1059] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.044s
W20260709 13:10:01.389727 30890 log.cc:927] Time spent T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 13:10:01.375396 30886 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:01.391695 30622 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:01.352532 30736 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:10:01.414706 25390 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
W20260709 13:10:01.424834 25237 consensus_meta.cc:287] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.587s	user 0.010s	sys 0.000s
I20260709 13:10:01.425158 25395 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:10:01.378185 25319 rpcz_store.cc:276] 0709 13:10:00.335055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:00.489674 (+154619us) service_pool.cc:224] Handling call
0709 13:10:00.495742 (+  6068us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:00.496165 (+   423us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:00.496201 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:00.514418 (+ 18217us) log.cc:844] Serialized 26 byte log entry
0709 13:10:00.791978 (+277560us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:10:00.792017 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:00.792105 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:01.377540 (+585435us) raft_consensus.cc:1685] finished
0709 13:10:01.377566 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:01.377721 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":26472,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":113,"thread_start_us":277085,"threads_started":1,"wal-append.queue_time_us":373568}
W20260709 13:10:01.405074 30555 consensus_meta.cc:287] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.645s	user 0.003s	sys 0.004s
W20260709 13:10:01.440845 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:01.441219 25237 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:01.443712 25390 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 13:10:01.443532 30516 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:01.444092 25391 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:10:01.451148 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:01.405604 30864 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:01.436889 30896 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:01.447381 30760 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:01.492472 25390 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 9. (term mismatch)
W20260709 13:10:01.466207 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 200599us
I20260709 13:10:01.494160 25390 pending_rounds.cc:85] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 8
I20260709 13:10:01.500603 25390 pending_rounds.cc:107] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 16.9
I20260709 13:10:01.456647 30659 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:01.491101 30896 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:01.500912 25390 raft_consensus.cc:2889] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:01.504393 30807 raft_consensus.cc:3055] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:10:01.512006 30494 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:10:01.519479 30874 log.cc:927] Time spent T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.306s	user 0.002s	sys 0.000s
I20260709 13:10:01.530107 30862 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:01.536243 25287 leader_election.cc:400] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17.
I20260709 13:10:01.547781 25287 leader_election.cc:403] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 13:10:01.423938 30483 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:01.543372 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:01.450960 30555 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:01.496819 30905 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:01.317375 25140 rpcz_store.cc:269] 0709 13:09:37.976113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:37.981476 (+  5363us) service_pool.cc:224] Handling call
0709 13:09:38.137542 (+156066us) spinlock_profiling.cc:243] Waited 2.65 ms on lock 0x7b5400300110. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8e640 00007f297ca65b76 00007f297cb6fc3e 00007f297b0302a4 00007f297b030232 00007f297b030161 00007f297b0300dc 00007f297b02f392 00007f2974fed48c 00007f2974fec8c6
0709 13:10:01.143045 (+23005503us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:09:58.315482 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:09:58.316074 (+   592us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:09:58.316108 (+    34us) write_op.cc:435] Acquired schema lock
0709 13:09:58.316126 (+    18us) tablet.cc:662] Decoding operations
0709 13:09:58.317821 (+  1695us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:09:58.317896 (+    75us) write_op.cc:641] Partition lock acquired for write op
0709 13:09:58.317936 (+    40us) tablet.cc:685] Acquiring locks for 14 operations
0709 13:09:58.319373 (+  1437us) tablet.cc:701] Row locks acquired
0709 13:09:58.319400 (+    27us) write_op.cc:260] PREPARE: finished
0709 13:09:58.319602 (+   202us) write_op.cc:270] Start()
0709 13:09:58.319757 (+   155us) write_op.cc:276] Timestamp: P: 1783602598319541 usec, L: 0
0709 13:09:58.319793 (+    36us) op_driver.cc:348] REPLICATION: starting
0709 13:09:58.320675 (+   882us) log.cc:844] Serialized 5624 byte log entry
0709 13:09:59.075363 (+754688us) op_driver.cc:464] REPLICATION: finished
0709 13:09:59.360435 (+285072us) write_op.cc:301] APPLY: starting
0709 13:09:59.362613 (+  2178us) tablet.cc:1367] starting BulkCheckPresence
0709 13:09:59.539971 (+177358us) tablet.cc:1370] finished BulkCheckPresence
0709 13:09:59.540161 (+   190us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:09:59.563265 (+ 23104us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:09:59.563920 (+   655us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=28,mrs_lookups=0
0709 13:09:59.563981 (+    61us) write_op.cc:312] APPLY: finished
0709 13:09:59.564575 (+   594us) log.cc:844] Serialized 129 byte log entry
0709 13:09:59.564907 (+   332us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:09:59.565443 (+   536us) write_op.cc:454] Released schema lock
0709 13:09:59.565550 (+   107us) write_op.cc:341] FINISH: Updating metrics
0709 13:09:59.623558 (+ 58008us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:09:59.623608 (+    50us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":48,"spinlock_wait_cycles":6106368,"child_traces":[["op",{"apply.queue_time_us":280859,"apply.run_cpu_time_us":26559,"apply.run_wall_time_us":264259,"cfile_cache_hit":58,"cfile_cache_hit_bytes":62591,"num_ops":14,"prepare.queue_time_us":83478,"prepare.run_cpu_time_us":44413,"prepare.run_wall_time_us":249302,"raft.queue_time_us":425389,"raft.run_cpu_time_us":35794,"raft.run_wall_time_us":246014,"replication_time_us":755422,"thread_start_us":686544,"threads_started":4,"wal-append.queue_time_us":277388,"wal-append.run_cpu_time_us":75565,"wal-append.run_wall_time_us":1960115}]]}
W20260709 13:10:01.534616 30876 log.cc:927] Time spent T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 13:10:01.549161 25391 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:10:01.554720 25359 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:01.440377 30886 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:01.499967 30760 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:01.565147 30858 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:01.565289 30908 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:01.565883 30908 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:01.568202 30908 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:01.568704 30908 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:01.569320 30897 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.165s	user 0.000s	sys 0.002s
I20260709 13:10:01.569698 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:01.570339 25390 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
W20260709 13:10:01.571679 30914 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260709 13:10:01.572878 30882 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:01.573812 30901 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:01.574452 30901 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:01.575958 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 81304.2us
I20260709 13:10:01.547061 30494 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:01.577116 25287 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:01.577561 30494 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:01.578277 30616 consensus_meta.cc:287] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.795s	user 0.010s	sys 0.002s
I20260709 13:10:01.578925 30904 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:01.546418 30534 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:01.579830 30899 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260709 13:10:01.560058 30905 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:01.581929 30663 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:10:01.583031 25314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 919) took 1499 ms. Trace:
I20260709 13:10:01.583316 30905 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:01.583678 30477 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:01.566180 30559 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:01.585563 30786 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:01.585466 30559 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:01.586992 30760 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:01.587735 30559 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:01.589457 30859 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 13:10:01.591306 30305 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:01.578353 25287 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:01.593449 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:01.593747 25287 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:01.583400 25314 rpcz_store.cc:276] 0709 13:10:00.083463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:00.085187 (+  1724us) service_pool.cc:224] Handling call
0709 13:10:00.118071 (+ 32884us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:00.612613 (+494542us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:00.612658 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:00.659871 (+ 47213us) log.cc:844] Serialized 26 byte log entry
0709 13:10:00.875630 (+215759us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:10:00.875657 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:00.899297 (+ 23640us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54000f9e54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:00.899359 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:01.518513 (+619154us) spinlock_profiling.cc:243] Waited 7.71 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:01.582845 (+ 64332us) raft_consensus.cc:1685] finished
0709 13:10:01.582866 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:01.582979 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":155826,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"spinlock_wait_cycles":441881088,"thread_start_us":486560,"threads_started":2,"wal-append.queue_time_us":371913}
W20260709 13:10:01.578430 30895 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.193s	user 0.000s	sys 0.001s
W20260709 13:10:01.564446 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 197926us
I20260709 13:10:01.543220 30807 raft_consensus.cc:740] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:01.606105 25287 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:01.657749 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:01.659010 25316 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:10:01.658174 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:01.662039 25315 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:01.662683 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:01.542979 30659 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:01.646435 30646 consensus_meta.cc:287] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.661s	user 0.006s	sys 0.005s
I20260709 13:10:01.661080 25940 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
I20260709 13:10:01.659324 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:01.663383 25239 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:10:01.608572 30387 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:01.671947 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:01.672417 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:01.725478 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:01.648123 30878 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:01.663254 30555 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:01.729236 30512 consensus_meta.cc:287] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.975s	user 0.013s	sys 0.000s
I20260709 13:10:01.604661 30908 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:01.673250 30387 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:01.742544 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:01.667377 30559 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.072s
W20260709 13:10:01.771800 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:01.755759 30807 consensus_queue.cc:260] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:01.783654 30807 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:10:01.786298 30918 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 13:10:01.719600 30646 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:01.796280 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:01.797583 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:01.811201 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:01.822156 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:01.824231 25936 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:01.770128 30901 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:01.808032 30516 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:01.848587 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:01.862725 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:01.745527 30387 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:01.878492 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:01.862176 30736 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:01.896821 30926 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
W20260709 13:10:01.899029 30681 consensus_meta.cc:287] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.762s	user 0.009s	sys 0.000s
I20260709 13:10:01.899497 30681 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:01.902117 30599 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:01.903684 25387 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:01.905995 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 1039) took 1613 ms. Trace:
I20260709 13:10:01.906399 25941 rpcz_store.cc:276] 0709 13:10:00.292866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:00.302406 (+  9540us) service_pool.cc:224] Handling call
0709 13:10:01.905780 (+1603374us) spinlock_profiling.cc:243] Waited 685 ms on lock 0x7b54006a31d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:01.905903 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1575912704}
W20260709 13:10:01.913981 30921 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 13:10:01.874094 30896 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:01.924211 25390 raft_consensus.cc:1275] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:10:01.925166 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:01.925392 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:01.925899 25389 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:01.927054 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:01.928292 25358 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:01.888916 30915 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260709 13:10:01.929993 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:01.930650 25393 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:10:01.931738 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:01.869827 30901 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:10:01.928469 30785 consensus_meta.cc:287] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.758s	user 0.004s	sys 0.006s
I20260709 13:10:01.738694 30305 raft_consensus.cc:697] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:01.938470 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:01.942811 25395 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 11.
W20260709 13:10:01.930998 30872 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:10:01.950114 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:01.953794 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:01.954495 25318 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 9. Current term is 10. Ops: []
I20260709 13:10:01.952251 25911 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:01.957921 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:10:01.944278 30866 consensus_meta.cc:287] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.886s	user 0.009s	sys 0.005s
I20260709 13:10:01.958765 25388 raft_consensus.cc:2376] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16. Re-sending same reply.
I20260709 13:10:01.953828 25240 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
W20260709 13:10:01.951485 30749 consensus_meta.cc:287] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.786s	user 0.003s	sys 0.008s
W20260709 13:10:01.985417 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 57196.9us
I20260709 13:10:01.992063 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:01.993435 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:01.992774 25318 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:10:01.963251 30905 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:10:01.940805 30736 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:01.970988 30857 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:01.950789 30901 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:01.948549 30785 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.017866 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:02.020670 25319 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:10:01.984863 30886 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:01.960399 30494 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.029840 30646 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:02.092197 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 96832.2us
I20260709 13:10:02.031957 30811 raft_consensus.cc:3055] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:02.032591 30908 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:02.035674 30736 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:02.038679 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:02.039243 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:02.041291 30477 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:02.043823 30922 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.198s	user 0.000s	sys 0.001s
W20260709 13:10:02.045411 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:02.048216 25285 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:02.157737 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:02.052330 30864 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:02.175755 30659 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:02.053097 30599 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:02.176738 30599 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.179953 30599 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:02.020226 30645 consensus_meta.cc:287] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.772s	user 0.011s	sys 0.000s
I20260709 13:10:02.182736 30645 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.184075 25942 raft_consensus.cc:2368] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17.
I20260709 13:10:02.184413 30900 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:02.077916 25944 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:10:02.185667 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:02.068235 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:02.186524 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 1131) took 1119 ms. Trace:
I20260709 13:10:02.186899 25942 rpcz_store.cc:276] 0709 13:10:01.067577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:01.067696 (+   119us) service_pool.cc:224] Handling call
0709 13:10:02.186298 (+1118602us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54005d0054. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:02.186454 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2504138624}
I20260709 13:10:02.187920 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:02.188562 25241 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:02.066169 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:02.088886 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:02.189545 25238 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:02.190528 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:02.191892 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:02.186547 25940 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:02.066560 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:02.193441 25240 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32994 (request call id 2429) took 1389 ms (client timeout 1567 ms). Trace:
I20260709 13:10:02.067772 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:02.085536 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:02.003191 30927 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:01.986335 30749 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.137226 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:02.139814 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:02.140967 25237 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:02.023815 30881 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:02.212581 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:02.214103 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:02.096928 30746 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:02.216536 25389 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:02.107093 30483 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:02.117889 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:02.148011 25314 raft_consensus.cc:1275] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:10:02.052758 30778 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:02.066920 30923 log.cc:927] Time spent T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.195s	user 0.001s	sys 0.000s
I20260709 13:10:02.066870 30919 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:02.153239 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:02.160854 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:02.161017 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:02.076506 30885 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.861s	user 0.002s	sys 0.000s
I20260709 13:10:02.232932 25284 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:02.161927 30933 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 13:10:02.162550 25242 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:10:02.163318 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:02.163688 30931 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.073s	user 0.000s	sys 0.003s
I20260709 13:10:02.135144 30811 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:02.166946 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:02.167620 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:02.174913 30901 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:02.175618 30864 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.052384 30878 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:02.073680 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:02.014739 30857 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:02.067701 30928 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 13:10:01.967904 30866 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:02.068980 30925 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.207s	user 0.000s	sys 0.001s
I20260709 13:10:02.193446 25938 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:10:02.195595 30477 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:02.193876 25240 rpcz_store.cc:269] 0709 13:10:00.804212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:00.809720 (+  5508us) service_pool.cc:224] Handling call
0709 13:10:02.193290 (+1383570us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540048f654. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:02.193386 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2615669504}
I20260709 13:10:02.194252 25945 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
W20260709 13:10:02.197224 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 45832.2us
I20260709 13:10:02.208312 25313 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:02.069240 30681 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:02.223981 25319 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:02.231925 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:02.270850 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2431) took 1012 ms. Trace:
I20260709 13:10:02.210203 30645 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:02.206431 30927 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:02.245214 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:02.247375 25237 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:02.250051 25393 raft_consensus.cc:2410] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:10:02.252388 25287 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:02.253943 25209 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:02.412374 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 158515us
W20260709 13:10:02.412580 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 160258us
I20260709 13:10:02.413269 25287 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:02.413894 25209 leader_election.cc:400] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17.
I20260709 13:10:02.414201 25209 leader_election.cc:403] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 13:10:02.414543 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:02.255123 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:02.255472 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 45979.8us
I20260709 13:10:02.269737 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:02.276369 25394 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:02.198334 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:02.231395 30900 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:02.232223 30599 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:02.079424 30785 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:02.327725 25389 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:02.246906 30778 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:02.326453 30843 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.900s	user 0.008s	sys 0.019s
I20260709 13:10:02.267294 30856 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.151s
I20260709 13:10:02.265156 30811 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.326316 30901 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:02.374029 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:02.410329 30927 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:02.326049 30878 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:02.375412 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 182106us
I20260709 13:10:02.420176 30786 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:10:02.200516 30305 consensus_queue.cc:237] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:02.325819 30892 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:02.247264 30919 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:02.470077 30901 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.474287 30878 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:02.474956 30811 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:02.475036 30919 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:02.269359 30871 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:02.475688 30901 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:02.475963 30871 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.329007 30896 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:02.477702 25940 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:10:02.478075 30896 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:02.478520 30896 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:02.255303 30682 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:02.272099 30908 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:02.257330 30483 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:02.416163 25908 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:02.416790 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:02.484369 25395 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:02.272205 30555 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:02.416989 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 368797us
I20260709 13:10:02.410454 25387 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
W20260709 13:10:02.379843 30933 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 13:10:02.421640 25237 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 13:10:02.442116 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2631) took 1623 ms. Trace:
W20260709 13:10:02.443121 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 210235us
I20260709 13:10:02.278988 30857 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:02.498211 30871 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:02.500727 25238 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:10:02.501323 30891 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:02.501874 30891 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.502274 30477 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:02.503346 30952 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:02.504045 30891 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:02.504249 30555 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:02.420894 30807 consensus_meta.cc:287] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.637s	user 0.012s	sys 0.000s
I20260709 13:10:02.328191 25245 rpcz_store.cc:276] 0709 13:10:01.258376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:01.259067 (+   691us) service_pool.cc:224] Handling call
0709 13:10:01.273923 (+ 14856us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:01.453177 (+179254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:01.453207 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:01.463246 (+ 10039us) log.cc:844] Serialized 26 byte log entry
0709 13:10:01.719264 (+256018us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:01.719335 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:01.719400 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:02.270634 (+551234us) raft_consensus.cc:1685] finished
0709 13:10:02.270653 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:02.270783 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6363,"raft.queue_time_us":22251,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":9163,"thread_start_us":419713,"threads_started":2,"wal-append.queue_time_us":336488}
I20260709 13:10:02.468622 25392 raft_consensus.cc:2393] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 16: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:02.249289 30646 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:02.506999 30646 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.472743 30786 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:02.509389 30786 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:10:02.470009 30387 consensus_meta.cc:287] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.599s	user 0.010s	sys 0.002s
I20260709 13:10:02.511950 30387 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:02.513137 25317 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:10:02.513644 30807 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:02.491240 30908 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:02.516180 30908 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:02.516225 25285 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:02.518188 25285 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:02.519546 25285 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:02.520690 30919 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:10:02.520830 30884 consensus_meta.cc:287] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.083s	user 0.014s	sys 0.006s
I20260709 13:10:02.521361 30884 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:02.420331 30559 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.063s
I20260709 13:10:02.523296 30559 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:02.523748 30884 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:02.523844 30559 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.492491 25321 rpcz_store.cc:276] 0709 13:10:00.818953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:00.819081 (+   128us) service_pool.cc:224] Handling call
0709 13:10:00.819512 (+   431us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:00.845503 (+ 25991us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:00.845532 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:00.872719 (+ 27187us) log.cc:844] Serialized 26 byte log entry
0709 13:10:02.075530 (+1202811us) spinlock_profiling.cc:243] Waited 4.77 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:02.080021 (+  4491us) spinlock_profiling.cc:243] Waited 2.38 ms on lock 0x7b1800377a40. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f297685a4bc 00007f2973ba848b 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f29768520d1
0709 13:10:02.080249 (+   228us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:10:02.080304 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:02.080423 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:02.441814 (+361391us) raft_consensus.cc:1685] finished
0709 13:10:02.441842 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:02.442048 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47257,"raft.queue_time_us":26045,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":5948,"spinlock_wait_cycles":16444544,"thread_start_us":225424,"threads_started":1,"wal-append.queue_time_us":260365}
I20260709 13:10:02.525580 30866 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:02.525978 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:02.526633 25321 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:02.526677 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:02.494351 30483 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.479852 30682 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.529315 30483 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:02.530331 30682 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:02.531050 25942 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:02.471668 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:02.495656 30934 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 13:10:02.496762 30951 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:10:02.497303 30937 log.cc:927] Time spent T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
I20260709 13:10:02.473409 30892 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:02.498096 30942 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 13:10:02.446300 30369 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17.
W20260709 13:10:02.507457 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 91374.9us
I20260709 13:10:02.514243 30387 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:02.419621 30900 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:02.521732 30919 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:10:02.496486 30945 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
W20260709 13:10:02.448060 30843 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.057s	user 0.009s	sys 0.019s
I20260709 13:10:02.533520 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:02.584399 25238 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:02.445875 30681 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.114s
I20260709 13:10:02.532919 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:02.585521 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:02.549100 25284 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:02.442868 30904 consensus_meta.cc:287] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.863s	user 0.012s	sys 0.000s
I20260709 13:10:02.534999 25394 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:02.565140 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:02.587427 25938 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:02.463168 30599 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.595997 30599 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:02.598881 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:02.600101 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:02.569790 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:02.604344 25240 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:10:02.564643 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:02.605298 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:02.606046 25240 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:10:02.606249 25911 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:02.610821 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:02.611687 25240 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:02.525074 30884 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:02.531728 30881 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:02.581406 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:02.582887 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 92127.2us
I20260709 13:10:02.504086 30952 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:02.564510 30656 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:10:02.588295 30659 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:02.591454 30892 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:02.592338 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:02.593801 30948 log.cc:927] Time spent T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 13:10:02.559267 30494 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:02.594058 25936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1005) took 1530 ms. Trace:
I20260709 13:10:02.570562 30749 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:02.579594 30900 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:02.575086 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:02.590170 25908 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:02.632975 25943 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:02.615217 25285 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:02.590117 30864 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:02.617448 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 202809us
I20260709 13:10:02.617740 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:02.612751 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:02.621321 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 72298.4us
I20260709 13:10:02.636623 25238 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:02.613641 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:02.600334 30646 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:02.629482 25936 rpcz_store.cc:276] 0709 13:10:01.063921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:01.093962 (+ 30041us) service_pool.cc:224] Handling call
0709 13:10:01.107581 (+ 13619us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:01.832814 (+725233us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:02.593777 (+760963us) spinlock_profiling.cc:243] Waited 716 ms on lock 0x7b5400014d54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:02.593968 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1647523584}
I20260709 13:10:02.582026 30919 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:02.612711 30786 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.648839 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:02.617072 30904 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17.
I20260709 13:10:02.636289 25240 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:10:02.640702 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:02.650687 25395 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:02.641696 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:02.644975 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 54891.3us
I20260709 13:10:02.651448 25387 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:10:02.652786 25246 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:02.653309 30871 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:02.653698 30871 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:02.653820 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:02.654090 30871 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:02.654495 25392 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:02.658022 30892 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:02.658564 30892 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:02.658933 30892 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:02.661312 30866 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:02.661918 30866 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:02.664067 30866 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:02.665526 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:02.665850 30778 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:02.632922 30682 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:02.667266 30682 raft_consensus.cc:697] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:02.621236 30881 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:02.672325 25320 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:02.646587 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:02.639385 25287 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:02.648609 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:02.648772 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:02.676064 25317 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:02.676540 25943 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:02.679455 25239 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 17: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:02.671453 25909 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:02.678051 25284 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:02.681118 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 63476.8us
I20260709 13:10:02.681457 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:02.682188 25245 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:02.682343 25284 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:02.680521 25287 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:02.691861 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:02.651114 30919 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:02.650501 30843 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:10:02.628592 30659 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.649168 30856 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:02.635281 30726 consensus_queue.cc:1048] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
W20260709 13:10:02.716332 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 44954.4us
I20260709 13:10:02.723908 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:02.725054 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:02.694890 30864 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:02.668381 30682 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.693322 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:02.683772 30387 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:02.688941 30778 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:02.710210 30952 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:10:02.727725 30942 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 13:10:02.709740 30866 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:10:02.728273 30945 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 13:10:02.741925 25240 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:10:02.739106 25245 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:02.743283 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:02.745942 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 63894.2us
I20260709 13:10:02.752506 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:02.741464 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:02.732141 30760 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:02.689811 30898 consensus_queue.cc:1059] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.049s
I20260709 13:10:02.768761 30559 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:02.764590 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:02.770263 25312 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:02.770437 30786 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:02.765318 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:02.765467 30555 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:02.766746 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:02.691526 30896 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:02.777343 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 137613us
I20260709 13:10:02.780074 25313 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:10:02.794219 25908 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:02.741428 30864 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:02.802161 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:02.819932 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 931) took 1082 ms. Trace:
I20260709 13:10:02.820560 25315 rpcz_store.cc:276] 0709 13:10:01.737535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:01.738518 (+   983us) service_pool.cc:224] Handling call
0709 13:10:01.755110 (+ 16592us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:01.807300 (+ 52190us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:01.807349 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:01.840788 (+ 33439us) log.cc:844] Serialized 52 byte log entry
0709 13:10:01.999731 (+158943us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:02.097411 (+ 97680us) log.cc:844] Serialized 14 byte log entry
0709 13:10:02.097648 (+   237us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:02.097733 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:02.644280 (+546547us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b18001a1940. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:10:02.763593 (+119313us) spinlock_profiling.cc:243] Waited 47.4 ms on lock 0x7b18001a1940. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc5522 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:10:02.819565 (+ 55972us) raft_consensus.cc:1685] finished
0709 13:10:02.819599 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:02.819842 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":94054,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":46,"raft.queue_time_us":24888,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":17804,"spinlock_wait_cycles":162930304,"thread_start_us":104150,"threads_started":1,"wal-append.queue_time_us":339079}
I20260709 13:10:02.773834 30681 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 13:10:02.825323 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 83920.4us
I20260709 13:10:02.744609 30952 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:02.872191 25909 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:02.872586 25212 leader_election.cc:336] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.524s (SENT)
I20260709 13:10:02.875620 25212 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:02.876736 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 82647.3us
I20260709 13:10:02.798144 30868 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:02.801420 30875 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:02.800876 30864 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:02.743040 30856 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:02.921882 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 126518us
I20260709 13:10:02.920984 30656 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:10:02.921325 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 127971us
I20260709 13:10:02.879009 30952 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:02.907315 30898 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:10:02.909180 30736 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:02.939706 25209 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:02.941672 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:02.948544 25313 raft_consensus.cc:2410] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:10:02.915066 30811 raft_consensus.cc:3055] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:02.899549 30512 consensus_queue.cc:1059] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 13:10:02.905656 30946 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:02.916205 30681 raft_consensus.cc:697] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:02.954977 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:02.960220 30857 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.962565 30896 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:02.963341 30387 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:02.963826 30652 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:10:02.922021 30875 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.932226 30656 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:02.964802 30656 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:10:02.966226 30875 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:02.937263 30965 consensus_queue.cc:1059] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 13:10:02.968266 25944 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 1138) took 1154 ms. Trace:
I20260709 13:10:02.969678 30512 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:10:02.970217 30512 raft_consensus.cc:697] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:02.971179 30512 consensus_queue.cc:237] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:02.973951 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:02.975315 30494 raft_consensus.cc:3055] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:10:02.975795 30494 raft_consensus.cc:740] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:02.950863 30811 raft_consensus.cc:740] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:02.978209 25287 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:02.979574 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:02.963915 30387 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:02.987866 25911 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:02.968669 25944 rpcz_store.cc:276] 0709 13:10:01.814335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:01.814502 (+   167us) service_pool.cc:224] Handling call
0709 13:10:01.864358 (+ 49856us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:02.186063 (+321705us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:02.968018 (+781955us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b5400168854. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:02.968208 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":490641664}
I20260709 13:10:02.992251 25287 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:03.119659 30681 consensus_queue.cc:237] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:03.125970 30927 consensus_meta.cc:287] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.643s	user 0.009s	sys 0.000s
I20260709 13:10:03.126533 30927 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:02.951865 30736 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:02.963374 30959 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 13:10:02.976862 30494 consensus_queue.cc:260] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:03.145629 30954 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.159s	user 0.000s	sys 0.000s
I20260709 13:10:03.146446 30927 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:03.149374 25390 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
W20260709 13:10:03.151954 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 164552us
I20260709 13:10:02.979252 30946 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:02.990854 30896 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:10:03.184875 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 358662us
I20260709 13:10:03.002624 30875 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:03.127444 30785 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:10:03.204833 25317 consensus_meta.cc:287] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.513s	user 0.007s	sys 0.000s
I20260709 13:10:03.141057 30736 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:03.210345 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 311348us
I20260709 13:10:03.141476 30884 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:03.213606 25317 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:03.218220 25312 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
W20260709 13:10:03.221798 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:03.227805 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:03.231101 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:03.190936 30652 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:03.194250 30898 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:03.208940 30875 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:03.200327 30946 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:03.212610 30785 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:03.230751 30900 consensus_meta.cc:287] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.561s	user 0.010s	sys 0.000s
W20260709 13:10:03.259424 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 292109us
I20260709 13:10:03.275758 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:03.284871 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:03.285646 25321 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:03.286528 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:03.285063 30905 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:03.257879 30875 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:03.242736 30969 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:03.267258 30785 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:03.324545 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:03.331664 25940 raft_consensus.cc:2410] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:10:03.205717 30811 consensus_queue.cc:260] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.267230 30483 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:03.329689 30659 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:03.362437 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:03.366905 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:03.367117 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:03.367552 25388 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:03.364104 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:03.368498 25389 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:03.365759 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:03.367754 25943 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:10:03.321527 30954 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 13:10:03.251184 30898 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.329779 30856 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:03.374707 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:03.380791 25938 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:03.256484 30652 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:03.382385 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:03.383278 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:03.383832 25395 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:03.383941 25937 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:03.384119 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:03.385329 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:03.364601 25358 connection.cc:256] RPC call timeout handler was delayed by -1.4095787140000766s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2198s for a response
I20260709 13:10:03.388484 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:03.389168 25388 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:03.388484 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:03.282191 30900 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:03.384941 25393 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:03.394392 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:03.394574 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:03.395097 25941 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:03.395193 25394 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:03.390605 25940 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:10:03.396577 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:03.362959 30909 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:03.413710 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:03.414538 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:03.418340 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:03.417023 25395 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:03.419111 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:03.423544 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:03.367745 30894 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:03.431991 25394 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:03.439036 25390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 1154) took 1350 ms. Trace:
I20260709 13:10:03.325237 30905 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.335114 30969 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.440945 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:03.446369 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:03.448609 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:03.449477 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:03.458135 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 2050) took 1320 ms. Trace:
I20260709 13:10:03.468779 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:03.451826 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:03.507359 25937 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:03.463845 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:03.508615 25395 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 14: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:03.467492 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:03.481460 25321 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:03.482555 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:03.463706 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 45496.9us
I20260709 13:10:03.483069 25393 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:03.408524 25943 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:10:03.512639 25909 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:03.432436 25284 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:03.514371 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 82053.3us
I20260709 13:10:03.474677 25390 rpcz_store.cc:276] 0709 13:10:02.088198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:02.118200 (+ 30002us) service_pool.cc:224] Handling call
0709 13:10:02.160997 (+ 42797us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:03.194127 (+1033130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:03.438631 (+244504us) spinlock_profiling.cc:243] Waited 988 ms on lock 0x7b5400277ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:03.438934 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2272513920}
W20260709 13:10:03.486546 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 106417us
W20260709 13:10:03.435760 30871 consensus_meta.cc:287] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.781s	user 0.008s	sys 0.002s
I20260709 13:10:03.532543 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:03.375380 30483 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.533474 25387 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 16
I20260709 13:10:03.450611 30857 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:03.535725 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:03.539006 25388 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:03.459219 30746 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:03.463935 30901 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:10:03.553977 25389 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:03.505612 25391 rpcz_store.cc:276] 0709 13:10:02.138149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:02.265600 (+127451us) service_pool.cc:224] Handling call
0709 13:10:02.265938 (+   338us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:02.277486 (+ 11548us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:02.277539 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:02.328478 (+ 50939us) log.cc:844] Serialized 26 byte log entry
0709 13:10:02.485091 (+156613us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:02.549818 (+ 64727us) log.cc:844] Serialized 14 byte log entry
0709 13:10:02.564843 (+ 15025us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:02.564922 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:02.928087 (+363165us) spinlock_profiling.cc:243] Waited 4.9 ms on lock 0x7b180009bc80. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:10:03.457718 (+529631us) raft_consensus.cc:1685] finished
0709 13:10:03.457849 (+   131us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:03.458052 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25821,"prepare.queue_time_us":65710,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":54863,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":30901,"spinlock_wait_cycles":11266944,"thread_start_us":75311,"threads_started":1,"wal-append.queue_time_us":171969}
I20260709 13:10:03.514112 30866 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:03.535535 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:03.556931 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:03.417058 30884 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:03.550963 25359 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:03.506184 30919 consensus_meta.cc:287] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.771s	user 0.011s	sys 0.000s
I20260709 13:10:03.531935 30856 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:03.555101 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:03.563915 25284 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:03.484336 30778 consensus_queue.cc:1059] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.070s
I20260709 13:10:03.559736 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:03.564432 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:03.569581 25241 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 13:10:03.570950 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:03.571345 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:03.569473 25359 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:03.548863 30555 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:03.571763 25389 raft_consensus.cc:2376] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12. Re-sending same reply.
I20260709 13:10:03.565754 25938 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
W20260709 13:10:03.565899 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 609404us
I20260709 13:10:03.505298 30894 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:03.559156 30881 consensus_meta.cc:287] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.848s	user 0.011s	sys 0.000s
I20260709 13:10:03.561264 25244 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:03.596907 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:03.629459 30900 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:03.606772 30881 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:03.632616 30881 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:03.611505 30555 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.635771 30555 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:03.607906 30516 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:03.637576 25243 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:10:03.637975 25313 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:03.638278 30969 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:03.639110 25386 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:10:03.553609 30646 consensus_queue.cc:1059] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.052s
I20260709 13:10:03.641652 30646 raft_consensus.cc:3055] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:03.642024 30646 raft_consensus.cc:740] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:03.642884 30646 consensus_queue.cc:260] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:03.643661 30864 consensus_meta.cc:287] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.715s	user 0.004s	sys 0.008s
I20260709 13:10:03.644039 30646 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:03.648159 30952 consensus_meta.cc:287] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.692s	user 0.006s	sys 0.004s
I20260709 13:10:03.648623 30952 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.649869 25318 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:10:03.647142 30875 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.650301 25387 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:03.651935 30483 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:03.652204 25911 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:03.557420 30681 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
I20260709 13:10:03.653159 30978 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:03.653728 30978 raft_consensus.cc:697] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:03.654537 30681 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:10:03.655196 30681 raft_consensus.cc:697] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:10:03.566938 30656 consensus_meta.cc:287] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.602s	user 0.010s	sys 0.000s
I20260709 13:10:03.655768 30656 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:03.656121 30681 consensus_queue.cc:237] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:03.657913 30656 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:03.658238 30652 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:03.538110 30871 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:03.596478 30892 consensus_meta.cc:287] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.937s	user 0.013s	sys 0.000s
I20260709 13:10:03.658959 30892 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.660256 30871 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:03.566802 30807 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:03.661032 30807 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.570835 30856 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:03.614852 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:03.606302 30878 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:03.560835 30977 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:03.654671 30978 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.578179 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:03.663035 30976 log.cc:927] Time spent T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:10:03.605814 25317 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:03.664472 25945 raft_consensus.cc:1240] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: [13.13-13.13]
W20260709 13:10:03.663905 30968 log.cc:927] Time spent T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.517s	user 0.000s	sys 0.004s
I20260709 13:10:03.680343 30786 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:03.666929 25391 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:03.676543 30878 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:03.664621 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:03.680945 30369 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:03.679436 25312 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:03.667235 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:03.696453 30387 consensus_meta.cc:287] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.565s	user 0.011s	sys 0.000s
W20260709 13:10:03.693183 30983 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 13:10:03.698427 25316 raft_consensus.cc:2393] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:03.698271 30786 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:03.699430 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:03.700588 30786 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:03.576862 30866 raft_consensus.cc:697] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:10:03.667302 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 54098.5us
I20260709 13:10:03.605170 30622 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:03.557956 30746 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.665392 25241 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:10:03.579412 25284 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:03.682054 25937 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:03.650244 30905 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:03.662465 30856 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:03.576086 30919 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:03.678013 25390 raft_consensus.cc:2393] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 16: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:03.670522 25320 raft_consensus.cc:2368] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 13. Current term is 14.
I20260709 13:10:03.661100 30892 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:03.663371 30807 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:03.702591 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:03.721380 25312 raft_consensus.cc:2393] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 10: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:03.700590 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:03.706750 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 127429us
I20260709 13:10:03.644093 30864 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.722949 25237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 986) took 1230 ms. Trace:
I20260709 13:10:03.723028 25284 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:03.703528 30878 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:03.724684 25237 rpcz_store.cc:276] 0709 13:10:02.492646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:02.492878 (+   232us) service_pool.cc:224] Handling call
0709 13:10:02.493258 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:02.493465 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:10:02.795702 (+302237us) log.cc:844] Serialized 14 byte log entry
0709 13:10:03.713779 (+918077us) log.cc:844] Serialized 14 byte log entry
0709 13:10:03.720576 (+  6797us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:03.720632 (+    56us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:03.720685 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:03.720751 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:03.722879 (+  2128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53628,"prepare.queue_time_us":439801,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":4783,"raft.queue_time_us":133562,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":9012,"thread_start_us":1103088,"threads_started":3,"wal-append.queue_time_us":614128}
I20260709 13:10:03.731832 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:03.709824 30559 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.065s	sys 0.068s
I20260709 13:10:03.734957 25209 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:03.766122 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:03.707549 30622 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:03.767581 25314 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 17: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:03.768364 30652 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:03.769763 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:03.771004 25209 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:03.736029 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:03.709357 30881 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:03.732563 30749 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:03.724260 30369 raft_consensus.cc:697] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:03.698292 30387 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.779947 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:03.737092 25319 raft_consensus.cc:2368] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17.
W20260709 13:10:03.733955 30726 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.015s	sys 0.012s
W20260709 13:10:03.805882 30726 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.015s	sys 0.013s
W20260709 13:10:03.734534 30559 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.066s	sys 0.069s
I20260709 13:10:03.796281 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:03.807166 25316 raft_consensus.cc:2393] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 15: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:03.772131 30972 consensus_queue.cc:1059] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.035s
I20260709 13:10:03.808001 30749 raft_consensus.cc:697] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:03.808112 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:03.809094 25316 raft_consensus.cc:2410] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 13:10:03.809012 30749 consensus_queue.cc:237] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.810103 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:03.811363 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:03.812541 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:03.813171 30369 consensus_queue.cc:237] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.813093 25316 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
W20260709 13:10:03.795475 30736 consensus_meta.cc:287] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.558s	user 0.007s	sys 0.006s
I20260709 13:10:03.815227 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:03.814985 30736 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.818252 30736 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:03.819006 25316 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:03.820261 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:03.821524 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:03.822086 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:03.822568 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:03.822729 25390 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:03.823169 25316 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:03.795826 30881 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:03.825598 30746 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:03.825948 30881 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:03.827286 30919 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:03.829347 30387 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:03.830130 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:03.830291 25390 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:03.831311 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:03.831837 25390 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:03.833675 30952 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:03.835613 25940 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 15: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:03.836584 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:03.768821 30652 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:03.837810 30652 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:03.837486 30866 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:03.842139 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 62260.4us
I20260709 13:10:03.845362 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:03.848665 25209 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:03.771642 30483 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:03.854596 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:03.851152 25391 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:10:03.856861 25312 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:10:03.825389 30875 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:03.735905 30973 raft_consensus.cc:3055] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:03.860663 25312 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:03.803747 30477 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:03.841195 30969 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:10:03.853533 30736 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:03.864684 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:03.852329 30986 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:03.854138 30952 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:03.872450 25390 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:03.889348 25209 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:03.865696 30555 raft_consensus.cc:3055] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Stepping down as leader of term 13
W20260709 13:10:03.870316 30946 consensus_meta.cc:287] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.588s	user 0.011s	sys 0.000s
I20260709 13:10:03.872449 30736 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.864004 30973 raft_consensus.cc:740] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:03.887100 30969 raft_consensus.cc:697] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:03.904368 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:03.907249 30736 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:03.898340 30786 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:03.907992 30786 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:03.908349 30786 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:03.894304 30726 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:10:03.903723 30785 consensus_meta.cc:287] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.534s	user 0.011s	sys 0.000s
I20260709 13:10:03.897539 30555 raft_consensus.cc:740] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:03.870507 30952 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:03.870636 30965 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:10:03.927600 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 394282us
W20260709 13:10:03.929972 25212 leader_election.cc:336] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.843s (SENT)
I20260709 13:10:03.930320 25212 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:03.931241 25212 leader_election.cc:336] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.985s (SENT)
I20260709 13:10:03.931566 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:03.928170 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 358760us
I20260709 13:10:03.922066 30807 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:03.922498 30726 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:03.904755 30946 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.934901 30965 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:03.934693 30952 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:10:04.070089 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 115683us
I20260709 13:10:04.071735 25287 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:04.080847 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:04.081631 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:04.083737 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:04.106160 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:04.115244 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 43783.2us
I20260709 13:10:04.121223 25287 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:04.123980 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:04.126228 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:04.127136 25943 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:04.128729 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:04.129534 25943 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:04.136166 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:04.140345 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:04.140530 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:04.141328 25390 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:04.141379 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:04.142338 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:04.143642 30746 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:04.144141 30746 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.950503 30726 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:04.150884 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:03.935411 30993 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:04.157743 25287 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:03.928670 30785 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:03.953142 30807 raft_consensus.cc:697] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:04.157619 25284 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:04.160627 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:04.161696 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:04.168776 25387 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:04.164287 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:04.106383 30965 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:04.064798 30896 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:04.191753 30896 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:04.193040 25942 raft_consensus.cc:2368] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:10:04.162084 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:04.110533 30387 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:04.152772 30919 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:04.182899 30946 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:04.183634 30987 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.231s	user 0.000s	sys 0.000s
I20260709 13:10:04.183921 30900 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:04.188867 30807 consensus_queue.cc:237] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:04.190760 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 728339us
I20260709 13:10:04.192191 30785 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:04.152428 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 218933us
I20260709 13:10:03.949425 30843 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.209s
I20260709 13:10:04.110069 30991 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:10:04.182163 30992 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 13:10:04.231241 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2654) took 1281 ms. Trace:
I20260709 13:10:04.231634 25315 rpcz_store.cc:276] 0709 13:10:02.950270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:03.152849 (+202579us) service_pool.cc:224] Handling call
0709 13:10:03.206666 (+ 53817us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:03.212705 (+  6039us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:03.212743 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:03.234615 (+ 21872us) log.cc:844] Serialized 26 byte log entry
0709 13:10:03.709645 (+475030us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:10:03.709691 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:03.709761 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:04.231028 (+521267us) raft_consensus.cc:1685] finished
0709 13:10:04.231054 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:04.231185 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9588,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":11096,"thread_start_us":420544,"threads_started":1,"wal-append.queue_time_us":661914}
I20260709 13:10:04.168597 30892 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:04.151680 30988 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:04.203258 25388 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:03.952404 30973 consensus_queue.cc:260] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:04.201844 30559 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:04.210388 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:04.222429 30919 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:04.226543 30896 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:04.201356 25390 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:04.250442 25285 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:04.252240 25285 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:04.161119 30878 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:04.244760 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:04.242627 25387 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:04.163209 30778 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:10:04.261708 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:04.245664 30516 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.100s	sys 0.175s
I20260709 13:10:04.152411 30969 consensus_queue.cc:237] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:04.273474 25315 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:10:04.247357 30892 raft_consensus.cc:697] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:04.204996 30387 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:04.253746 30900 raft_consensus.cc:697] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:04.259696 30919 raft_consensus.cc:697] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:04.263355 30973 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:04.272732 30516 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.100s	sys 0.175s
I20260709 13:10:04.284893 31008 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:04.285524 30749 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:04.285735 25315 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 13:10:04.272369 30652 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:04.287657 30994 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 13:10:04.290138 30746 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:04.204515 30555 consensus_queue.cc:260] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:04.291870 30856 consensus_meta.cc:287] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.561s	user 0.006s	sys 0.002s
I20260709 13:10:04.246886 30988 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:04.268908 30778 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:04.293678 30878 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:04.295050 30892 consensus_queue.cc:237] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:04.296083 30900 consensus_queue.cc:237] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:04.297292 30387 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:04.256471 30559 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:04.304437 30993 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:10:04.306505 30786 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:04.320494 25242 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:04.301070 30516 raft_consensus.cc:697] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:04.302488 30749 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:04.314152 30512 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:04.303252 31008 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:04.318986 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2655) took 1365 ms. Trace:
I20260709 13:10:04.238118 25909 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:04.336773 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 179145us
W20260709 13:10:04.334196 30881 consensus_meta.cc:287] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.508s	user 0.010s	sys 0.000s
W20260709 13:10:04.343106 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 90975.6us
I20260709 13:10:04.348399 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:04.340619 25316 rpcz_store.cc:276] 0709 13:10:02.953743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:03.823809 (+870066us) service_pool.cc:224] Handling call
0709 13:10:03.824063 (+   254us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:03.824245 (+   182us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:10:03.894469 (+ 70224us) log.cc:844] Serialized 14 byte log entry
0709 13:10:04.318652 (+424183us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:04.318703 (+    51us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:04.318745 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:04.318803 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:04.318928 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":69982,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":76,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":15539,"thread_start_us":423791,"threads_started":1}
I20260709 13:10:04.352190 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:04.352468 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 114476us
I20260709 13:10:04.335402 30387 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:04.339656 30555 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:04.339205 30749 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:04.414008 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:04.422576 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:04.423036 25943 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:04.423642 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:04.427847 25244 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:04.318526 30483 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.024s
I20260709 13:10:04.317359 30856 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:04.317164 30972 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 13:10:04.414690 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:04.424767 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:04.440129 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 123591us
I20260709 13:10:04.440979 25246 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:04.424237 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:04.438743 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:04.441917 25239 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:04.342506 31008 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:04.442687 25937 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
W20260709 13:10:04.445204 25318 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 943) took 1700 ms (client timeout 2012 ms). Trace:
I20260709 13:10:04.412226 30946 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:04.452924 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:04.446069 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2553) took 1234 ms. Trace:
W20260709 13:10:04.447510 25318 rpcz_store.cc:269] 0709 13:10:02.744950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:02.765033 (+ 20083us) service_pool.cc:224] Handling call
0709 13:10:04.444909 (+1679876us) spinlock_profiling.cc:243] Waited 878 ms on lock 0x7b540005df54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:04.445089 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2019279232}
W20260709 13:10:04.428700 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 85640.3us
I20260709 13:10:04.455459 25940 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:04.420507 30999 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:04.460557 30999 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:04.462893 30999 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:04.454360 25317 rpcz_store.cc:276] 0709 13:10:03.211318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:03.227501 (+ 16183us) service_pool.cc:224] Handling call
0709 13:10:04.445847 (+1218346us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b5400044a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:04.445989 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":846490880}
I20260709 13:10:04.465039 25317 raft_consensus.cc:1240] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
W20260709 13:10:04.460588 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 105549us
I20260709 13:10:04.467047 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:04.457657 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:04.344033 30881 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:04.469029 25940 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:04.343369 30512 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:04.344161 30965 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:04.355448 30878 consensus_queue.cc:1059] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.097s
I20260709 13:10:04.450883 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:04.479807 25943 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:04.429586 30866 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:04.441608 30483 consensus_queue.cc:1243] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer a2a77ecf2c0941618b5bc79d4c04db61 log is divergent from this leader: its last log entry 15.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
W20260709 13:10:04.494127 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 69233.2us
I20260709 13:10:04.421288 30369 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 13:10:04.487720 25287 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:04.422341 30387 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:04.526015 25244 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:04.531268 25246 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:04.539448 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:04.504360 31007 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:04.527803 31017 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:04.429554 30919 consensus_queue.cc:237] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:04.533970 31001 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 13:10:04.521876 30894 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:04.606796 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 64168.3us
I20260709 13:10:04.626432 25209 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:04.633265 30512 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:04.632954 31017 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:04.634739 30559 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:04.635237 30387 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:04.635694 30778 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:04.637079 31005 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.178s	user 0.000s	sys 0.000s
I20260709 13:10:04.640081 31021 consensus_queue.cc:1048] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:10:04.635604 31017 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:04.643362 30952 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:04.636860 30786 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:04.633373 31003 log.cc:927] Time spent T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.298s	user 0.001s	sys 0.000s
I20260709 13:10:04.503173 30866 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:04.666783 25242 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:04.636940 30856 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:04.518605 30516 consensus_queue.cc:237] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:04.685878 30952 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:04.698173 30881 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:04.703562 30726 consensus_meta.cc:287] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.538s	user 0.010s	sys 0.000s
I20260709 13:10:04.704126 30726 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:04.706780 30965 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:04.707746 30988 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:04.746397 30659 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:04.747233 30659 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:04.751593 30681 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:04.752178 30681 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:04.754317 30681 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:04.755527 25358 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:04.758520 31000 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:04.759434 31000 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:04.612030 31007 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:04.762316 31000 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:04.652467 25239 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:04.756826 30659 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:04.763355 31000 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:10:04.649857 30999 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:04.763772 31007 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:04.766507 30866 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:04.770169 25318 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:04.775002 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:04.777105 25358 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:04.779045 25358 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:04.779865 31000 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:04.781555 30999 raft_consensus.cc:697] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:04.782884 30999 consensus_queue.cc:237] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:04.811772 30494 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:04.811798 31007 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:04.813591 31007 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:04.814730 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2550) took 1604 ms. Trace:
I20260709 13:10:04.815153 25321 rpcz_store.cc:276] 0709 13:10:03.210343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:03.286278 (+ 75935us) service_pool.cc:224] Handling call
0709 13:10:04.814538 (+1528260us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b54000627d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:04.814671 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":446699776}
I20260709 13:10:04.818518 25285 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:04.819253 25945 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 1141) took 1603 ms. Trace:
I20260709 13:10:04.823190 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:04.824082 25943 raft_consensus.cc:2393] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:10:04.825958 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 338308us
I20260709 13:10:04.826967 25287 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:04.828619 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2653) took 1884 ms. Trace:
I20260709 13:10:04.819996 25945 rpcz_store.cc:276] 0709 13:10:03.215718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:03.218075 (+  2357us) service_pool.cc:224] Handling call
0709 13:10:03.255250 (+ 37175us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:03.688404 (+433154us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:04.818986 (+1130582us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x7b54004ad8d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:04.819180 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":940473600}
I20260709 13:10:04.829180 25313 rpcz_store.cc:276] 0709 13:10:02.944443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:02.979747 (+ 35304us) service_pool.cc:224] Handling call
0709 13:10:02.980041 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:03.638468 (+658427us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:04.828321 (+1189853us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x7b54000636d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:04.828550 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1512646528}
I20260709 13:10:04.837937 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:04.838145 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2764) took 1398 ms. Trace:
I20260709 13:10:04.838680 25938 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 17: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:04.838485 25386 rpcz_store.cc:276] 0709 13:10:03.423296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:03.552494 (+129198us) service_pool.cc:224] Handling call
0709 13:10:04.821029 (+1268535us) spinlock_profiling.cc:243] Waited 71.5 ms on lock 0x7b54003246d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:04.821859 (+   830us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":164552448}
I20260709 13:10:04.840184 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:04.841414 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 62491.2us
I20260709 13:10:04.843842 25358 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:04.845541 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 346985us
I20260709 13:10:04.839534 25285 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:04.848016 25320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 941) took 2107 ms (client timeout 2110 ms). Trace:
I20260709 13:10:04.848295 25941 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: [14.11-16.12]
I20260709 13:10:04.848980 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:04.844297 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2756) took 1423 ms. Trace:
I20260709 13:10:04.849594 25391 rpcz_store.cc:276] 0709 13:10:03.420757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:03.560669 (+139912us) service_pool.cc:224] Handling call
0709 13:10:04.844078 (+1283409us) spinlock_profiling.cc:243] Waited 39.7 ms on lock 0x7b54001f59d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:04.844213 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":91237120}
I20260709 13:10:04.850042 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:04.850184 25358 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:04.850406 25212 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:04.851159 25936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 1042) took 1628 ms (client timeout 1876 ms). Trace:
W20260709 13:10:04.851636 25936 rpcz_store.cc:269] 0709 13:10:03.223125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:04.837629 (+1614504us) service_pool.cc:224] Handling call
0709 13:10:04.851088 (+ 13459us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:04.857898 25285 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:04.859655 25240 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 18: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:10:04.848546 25320 rpcz_store.cc:269] 0709 13:10:02.742777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:02.743016 (+   239us) service_pool.cc:224] Handling call
0709 13:10:04.834819 (+2091803us) spinlock_profiling.cc:243] Waited 919 ms on lock 0x7b5400045954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:04.843186 (+  8367us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2112914304}
I20260709 13:10:04.866314 25358 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:04.866824 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:04.867163 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:04.869462 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:04.870142 25285 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:04.815780 31007 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:04.873986 25939 raft_consensus.cc:2368] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 15. Current term is 16.
I20260709 13:10:04.874081 25945 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 15: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:10:04.846496 30969 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.006s	sys 0.005s
I20260709 13:10:04.880376 25321 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:04.877426 25358 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:04.837725 30896 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 13:10:04.883510 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 257130us
I20260709 13:10:04.887130 25358 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:04.846131 31018 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:04.846395 31000 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:04.855003 30952 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:04.853492 30305 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:04.888882 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:04.892465 25285 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:04.867445 30659 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:04.867236 30892 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:10:04.879622 30843 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:10:04.926442 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 76095.9us
I20260709 13:10:04.893819 30927 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:10:04.885382 30969 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.006s	sys 0.005s
I20260709 13:10:04.891122 30681 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:04.879454 30919 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:10:04.926558 25939 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:10:04.929299 25212 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:04.894970 30905 raft_consensus.cc:2764] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 13:10:04.930022 25317 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:04.890563 31025 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:04.894894 30387 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:04.932407 25945 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:04.933424 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:04.934172 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:04.938580 25320 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:10:04.935847 25936 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 13:10:04.936517 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:04.939266 25313 raft_consensus.cc:2410] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 13:10:04.936697 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:04.945103 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:04.947971 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:04.900626 30786 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:10:04.940635 25940 raft_consensus.cc:2393] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:04.946924 25321 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:10:04.954252 25212 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:04.928766 30896 raft_consensus.cc:697] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:04.954770 25943 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:10:04.959798 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:04.963696 25318 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:10:04.958683 25212 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:04.937590 31025 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:04.937438 30986 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:04.969017 25941 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:04.911208 30952 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:04.940194 30555 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:10:04.970615 25938 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12. Ops: []
I20260709 13:10:04.939179 30969 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:04.974696 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:04.975484 25238 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:04.975422 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:04.955163 30726 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:04.928352 30998 consensus_queue.cc:1059] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.068s
I20260709 13:10:04.956590 31026 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:04.981238 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:04.962795 30786 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:04.966094 30387 raft_consensus.cc:697] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:04.985988 25938 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:04.985965 25244 raft_consensus.cc:2368] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15.
W20260709 13:10:04.985908 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:04.987056 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:04.987092 25212 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:04.970038 31025 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:04.974908 30993 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:04.996129 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:05.003579 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:05.008558 25238 raft_consensus.cc:1275] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:10:04.970831 30305 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.021s
I20260709 13:10:04.993960 30483 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:04.976366 31012 consensus_queue.cc:1059] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 13.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.033s
I20260709 13:10:05.013339 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:05.005403 31008 consensus_meta.cc:287] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.549s	user 0.011s	sys 0.001s
I20260709 13:10:04.991410 31026 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:05.007287 30896 consensus_queue.cc:237] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 15.7, Last appended by leader: 7, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:05.015426 30988 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:04.985791 30969 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:05.043720 25212 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:05.026151 31012 raft_consensus.cc:3055] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:10:05.024315 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:05.072649 30969 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:05.008289 30726 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:10:05.072858 30645 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:05.072865 25237 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
W20260709 13:10:05.029784 31028 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 13:10:05.073572 30645 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:05.074154 30645 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:05.074237 25237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 998) took 1229 ms. Trace:
I20260709 13:10:05.074635 25237 rpcz_store.cc:276] 0709 13:10:03.844895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:03.845036 (+   141us) service_pool.cc:224] Handling call
0709 13:10:05.074064 (+1229028us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54003a98d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:05.074172 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2821168896}
I20260709 13:10:05.076097 25237 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12. Ops: []
I20260709 13:10:05.018322 25938 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:05.017544 30483 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:05.068044 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 1163) took 1607 ms. Trace:
I20260709 13:10:05.023679 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:05.081015 25212 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:05.079135 25392 rpcz_store.cc:276] 0709 13:10:03.460722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:03.640788 (+180066us) service_pool.cc:224] Handling call
0709 13:10:04.821190 (+1180402us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:04.821435 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:04.821472 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:04.821530 (+    58us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:04.821573 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:04.821646 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:05.067839 (+246193us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54003246d0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:05.067968 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2713889152}
W20260709 13:10:05.077718 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 53507us
I20260709 13:10:05.087297 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:05.089499 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:05.022562 31029 log.cc:927] Time spent T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 13:10:05.085210 31019 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:05.064173 30749 consensus_meta.cc:287] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.620s	user 0.008s	sys 0.000s
I20260709 13:10:05.108306 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:05.117516 25313 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:05.038131 31030 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.111s	user 0.000s	sys 0.001s
I20260709 13:10:05.069680 30811 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:05.120689 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:05.121580 25318 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:05.122699 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:05.129720 25320 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:10:05.114542 25238 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:05.115978 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:05.132651 25321 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:05.022125 30952 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:05.078760 31023 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:05.087422 30483 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:05.077400 31012 raft_consensus.cc:740] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:05.001096 30993 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:05.040378 30898 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:05.142611 30726 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:05.164239 30726 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:05.147671 30973 consensus_meta.cc:287] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.847s	user 0.013s	sys 0.007s
I20260709 13:10:05.133708 30969 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:05.133895 30856 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:05.134506 30873 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:05.158933 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:05.101918 31008 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.210206 31008 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:05.206429 30873 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.211472 25318 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 15: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:05.214658 30873 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:05.205053 25313 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:05.217554 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:05.218446 25313 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:05.231475 25313 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:05.125473 31019 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.139596 30749 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:05.247900 30993 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:05.164726 30726 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:05.248597 31012 consensus_queue.cc:260] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.234483 30778 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:10:05.272847 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 206180us
I20260709 13:10:05.274909 25315 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:05.235438 30969 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:05.144044 30387 consensus_queue.cc:237] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.277804 30760 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:05.275830 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:05.250622 30477 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:05.293833 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2657) took 1567 ms. Trace:
I20260709 13:10:05.322119 25317 rpcz_store.cc:276] 0709 13:10:03.726068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:04.466111 (+740043us) service_pool.cc:224] Handling call
0709 13:10:04.466473 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:04.932384 (+465911us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:05.293581 (+361197us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b54004399d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:05.293769 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1065547008}
I20260709 13:10:05.323582 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:05.278731 25358 leader_election.cc:336] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Timed out: RequestConsensusVote RPC to 127.23.140.68:39211 timed out after 1.671s (SENT)
I20260709 13:10:05.297367 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2449) took 1745 ms. Trace:
W20260709 13:10:05.325614 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 46928.3us
I20260709 13:10:05.325753 25243 rpcz_store.cc:276] 0709 13:10:03.551707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:03.551854 (+   147us) service_pool.cc:224] Handling call
0709 13:10:03.552220 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:03.638017 (+ 85797us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:05.297093 (+1659076us) spinlock_profiling.cc:243] Waited 84.9 ms on lock 0x7b5400059454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:05.297298 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":195347328}
W20260709 13:10:05.301605 25314 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 944) took 2556 ms (client timeout 3134 ms). Trace:
W20260709 13:10:05.327666 25314 rpcz_store.cc:269] 0709 13:10:02.745242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:02.766474 (+ 21232us) service_pool.cc:224] Handling call
0709 13:10:05.301401 (+2534927us) spinlock_profiling.cc:243] Waited 995 ms on lock 0x7b5400087ad4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:05.301542 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2288030976}
I20260709 13:10:05.328759 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:05.273550 30916 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:05.292634 25944 raft_consensus.cc:2410] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
W20260709 13:10:05.272783 30807 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.995s	user 0.014s	sys 0.032s
I20260709 13:10:05.324939 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:05.260161 31019 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:05.272888 31012 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:05.237763 30856 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.289034 25359 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:05.303850 25908 leader_election.cc:400] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 13. Current term is 14.
I20260709 13:10:05.332142 25358 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:05.329316 25319 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2559) took 1935 ms (client timeout 1520 ms). Trace:
W20260709 13:10:05.340963 25319 rpcz_store.cc:269] 0709 13:10:03.393514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:03.418834 (+ 25320us) service_pool.cc:224] Handling call
0709 13:10:05.329051 (+1910217us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b54002f3fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:05.329232 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":624138112}
I20260709 13:10:05.332547 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:05.335968 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 47036.2us
I20260709 13:10:05.344182 25359 leader_election.cc:400] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15.
I20260709 13:10:05.344552 25359 leader_election.cc:403] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:10:05.316641 25209 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:05.319693 25313 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:05.329491 25314 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:05.356180 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:05.357318 30927 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:05.358566 30927 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:05.360373 30927 consensus_queue.cc:1059] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:10:05.331692 30843 consensus_queue.cc:1059] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.030s
I20260709 13:10:05.361709 30905 raft_consensus.cc:3055] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:05.361904 30843 raft_consensus.cc:3055] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:05.362159 30905 raft_consensus.cc:740] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:05.362360 30843 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:05.363080 30905 consensus_queue.cc:260] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.361290 30659 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:05.363168 30843 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:05.290946 31004 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 1.005s	user 0.001s	sys 0.009s
W20260709 13:10:05.331399 30807 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.056s	user 0.014s	sys 0.032s
I20260709 13:10:05.389950 25908 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:05.330396 30785 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:05.296476 30622 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:05.353029 30305 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:05.312927 31008 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:05.295006 30969 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:05.318511 30749 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:05.296797 30864 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:05.308110 31026 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:05.391906 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:05.397324 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:05.425539 25314 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:05.329071 31025 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.305895 30786 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.089s
I20260709 13:10:05.315806 30965 consensus_queue.cc:1059] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.080s
I20260709 13:10:05.391705 31000 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 15
I20260709 13:10:05.329622 30916 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.329811 30760 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.394802 30811 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:05.399262 30843 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:05.347958 30477 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:05.436439 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:05.394021 30645 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.463029 25319 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 14: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:05.311010 30726 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.467105 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:05.468355 30726 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:05.385305 30927 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.059s
I20260709 13:10:05.397507 30305 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:05.394914 30659 consensus_queue.cc:1059] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.030s
I20260709 13:10:05.397365 30988 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:05.478554 30652 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:05.479034 30645 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:05.399313 30785 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.465544 25908 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:05.481252 31015 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:05.482332 25908 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:05.483103 31015 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:05.468286 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:05.484465 25319 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
W20260709 13:10:05.484906 31038 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:10:05.485677 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:05.487358 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:05.488324 25319 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 16: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:05.490504 30916 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:05.491117 30916 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:05.491773 30477 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:05.493319 30785 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:05.492980 30483 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.495891 30483 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:05.395701 30516 consensus_queue.cc:1048] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.055s
I20260709 13:10:05.463382 30811 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:05.473513 30305 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:05.497840 30811 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:05.498396 31025 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:05.423986 31008 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.498868 30856 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:05.500726 31008 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:05.483174 30873 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:10:05.483896 31015 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:05.475282 30659 consensus_queue.cc:1243] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 12.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:10:05.397969 30905 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:05.514118 25314 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:05.498407 30760 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:05.522545 30884 raft_consensus.cc:3055] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:05.523103 30884 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:05.524086 30884 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:05.512578 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:05.464507 30998 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:05.505368 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:05.538039 25319 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 17: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:10:05.505312 30991 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.215s	user 0.023s	sys 0.008s
I20260709 13:10:05.539994 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:05.535362 25908 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:05.536662 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:05.546940 25319 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:05.547978 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:05.546021 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 159108us
I20260709 13:10:05.548094 30884 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:05.513624 30622 consensus_queue.cc:1059] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 13:10:05.545360 30998 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.552879 30998 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:05.554137 31026 raft_consensus.cc:3055] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:05.554607 31026 raft_consensus.cc:740] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:05.555668 30305 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.555740 31026 consensus_queue.cc:260] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:05.557816 30969 raft_consensus.cc:3055] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:05.558250 30969 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:05.558478 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:05.559152 30969 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:05.560155 30969 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:05.560168 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:05.561108 25389 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 17: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:05.561335 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:05.562052 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:05.560395 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:05.563598 25392 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 15: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:05.563817 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:05.564404 25390 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:10:05.549860 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:05.566605 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:05.567157 25390 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:05.554018 30998 consensus_queue.cc:1059] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:10:05.570869 25908 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:05.573861 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 208639us
W20260709 13:10:05.545017 30991 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.295s	user 0.024s	sys 0.008s
I20260709 13:10:05.579196 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:05.583441 25241 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:05.567449 31026 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:05.579887 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:05.584417 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:05.584677 25238 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:05.580245 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:05.585799 25243 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:05.585875 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:05.580451 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:05.580636 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:05.578086 25284 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:05.574824 25358 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:05.585958 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:05.587157 25244 raft_consensus.cc:2368] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 13. Current term is 14.
I20260709 13:10:05.575970 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:05.587870 25392 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:10:05.588430 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:05.588709 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:05.589013 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:05.589239 25941 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 16: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:05.592707 25940 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:05.593037 25284 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:05.593796 25937 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:10:05.594086 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:05.594686 25284 leader_election.cc:400] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:10:05.594801 25387 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:05.594964 30998 consensus_queue.cc:1059] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:10:05.596416 30864 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:05.598318 30864 raft_consensus.cc:697] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:05.599401 30864 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.595127 25284 leader_election.cc:403] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 13:10:05.572868 30898 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:05.606312 30387 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:10:05.608443 25942 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:10:05.598313 30998 raft_consensus.cc:3055] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:05.609640 30898 raft_consensus.cc:697] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:05.612033 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:05.609695 25240 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 1005) took 1526 ms (client timeout 1825 ms). Trace:
I20260709 13:10:05.609902 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2574) took 1840 ms. Trace:
I20260709 13:10:05.611510 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:05.615252 25391 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
W20260709 13:10:05.614916 25240 rpcz_store.cc:269] 0709 13:10:04.083303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:04.083424 (+   121us) service_pool.cc:224] Handling call
0709 13:10:05.609500 (+1526076us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x7b54002d0fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:05.609637 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1649504128}
I20260709 13:10:05.615025 25315 rpcz_store.cc:276] 0709 13:10:03.769711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:04.534644 (+764933us) service_pool.cc:224] Handling call
0709 13:10:04.536086 (+  1442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:05.275595 (+739509us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:05.609688 (+334093us) spinlock_profiling.cc:243] Waited 738 ms on lock 0x7b54000f8a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:05.609846 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1698520320}
I20260709 13:10:05.612725 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:05.616364 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:05.617082 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:05.614334 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:05.618462 25315 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:10:05.620079 25387 pending_rounds.cc:85] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 10
I20260709 13:10:05.620568 25387 pending_rounds.cc:107] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 15.11
I20260709 13:10:05.620919 25387 raft_consensus.cc:2889] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:05.621739 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:05.637899 25388 raft_consensus.cc:2410] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 13:10:05.621917 25390 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:05.639719 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:05.576220 30978 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:05.622707 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:05.642483 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:05.642731 30884 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:05.643119 25391 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 10.
I20260709 13:10:05.623195 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:05.623793 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:05.623950 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:05.625541 30516 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:05.625633 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:05.626024 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2571) took 1924 ms. Trace:
I20260709 13:10:05.626812 30811 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.626628 30898 consensus_queue.cc:237] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.634757 25315 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 13:10:05.646636 25316 rpcz_store.cc:276] 0709 13:10:03.701733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:04.358553 (+656820us) service_pool.cc:224] Handling call
0709 13:10:04.358935 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:04.461760 (+102825us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:10:04.462066 (+   306us) log.cc:844] Serialized 14 byte log entry
0709 13:10:04.891691 (+429625us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:04.891743 (+    52us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:10:04.891791 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:05.625747 (+733956us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b54000f9954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:05.625798 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:05.625964 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6862,"prepare.queue_time_us":165,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":89529,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":93,"spinlock_wait_cycles":235869056,"thread_start_us":422266,"threads_started":2,"wal-append.queue_time_us":225799}
I20260709 13:10:05.658915 25392 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 13:10:05.649394 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:05.657253 25238 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:05.609783 30998 raft_consensus.cc:740] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:05.686863 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:05.688149 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:05.688301 25316 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:05.688688 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:05.690801 25392 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 15: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:05.691795 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:05.647619 30978 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:05.696327 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:05.696910 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:05.697713 25937 raft_consensus.cc:2368] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15.
I20260709 13:10:05.707077 25316 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 11. (term mismatch)
I20260709 13:10:05.708925 25392 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:05.710575 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:05.714954 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:05.715906 25390 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 9.
I20260709 13:10:05.716454 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:05.719337 25938 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:10:05.707808 25316 pending_rounds.cc:85] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 10
I20260709 13:10:05.720527 25316 pending_rounds.cc:107] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 15.11
I20260709 13:10:05.720769 25942 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 18: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:05.720949 25316 raft_consensus.cc:2889] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:05.724053 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:05.724231 25392 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
W20260709 13:10:05.728672 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:05.693917 30978 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:05.749185 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:05.751029 25390 raft_consensus.cc:2368] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15.
I20260709 13:10:05.653390 30760 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.074s
I20260709 13:10:05.781513 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:05.758842 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:05.783843 25316 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:10:05.783643 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:05.781632 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:05.784822 25388 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:05.761013 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:05.786000 25390 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:05.698828 30646 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.057s
W20260709 13:10:05.811208 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:05.813613 25944 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 1143) took 2549 ms (client timeout 1669 ms). Trace:
W20260709 13:10:05.814029 25944 rpcz_store.cc:269] 0709 13:10:03.264216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:03.448231 (+184015us) service_pool.cc:224] Handling call
0709 13:10:05.813366 (+2365135us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b5400675554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:05.813518 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4110680960}
I20260709 13:10:05.815192 25386 raft_consensus.cc:2410] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 13:10:05.817690 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:05.791664 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:05.820138 25937 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:05.822237 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:05.766572 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:05.776329 25212 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:05.824328 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 48050.7us
I20260709 13:10:05.783691 25392 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:05.778403 30305 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:05.802737 30512 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:05.805819 30811 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:05.808486 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 951) took 2587 ms. Trace:
I20260709 13:10:05.850862 25392 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:10:05.849269 25312 rpcz_store.cc:276] 0709 13:10:03.221050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:04.069786 (+848736us) service_pool.cc:224] Handling call
0709 13:10:04.143439 (+ 73653us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:04.299497 (+156058us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:10:04.313223 (+ 13726us) log.cc:844] Serialized 14 byte log entry
0709 13:10:04.817838 (+504615us) spinlock_profiling.cc:243] Waited 1.42 ms on lock 0x7b18002dc100. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f297685223a 00007f2976cbca2a
0709 13:10:05.197483 (+379645us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:05.197536 (+    53us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:05.197579 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:05.807777 (+610198us) spinlock_profiling.cc:243] Waited 6.29 ms on lock 0x7b18002dc100. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b932a6 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419
0709 13:10:05.808248 (+   471us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b5400274a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:05.808291 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:05.808425 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5857,"prepare.queue_time_us":14348,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":3036,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":35685,"spinlock_wait_cycles":375946240,"thread_start_us":376334,"threads_started":1,"wal-append.queue_time_us":1081274}
I20260709 13:10:05.809386 30998 consensus_queue.cc:260] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.793705 31021 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:05.762476 30681 raft_consensus.cc:3055] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:05.772326 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:05.827466 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 96876.1us
I20260709 13:10:05.871696 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:05.872782 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:05.873699 25938 raft_consensus.cc:2376] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15. Re-sending same reply.
I20260709 13:10:05.828464 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:05.831368 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:05.833766 25386 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:10:05.876820 25237 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:05.878067 25937 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:05.878135 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:05.832011 25212 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:05.837384 25390 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:05.763118 31048 raft_consensus.cc:3055] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:10:05.828038 25209 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:05.761646 31047 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 0.286s	user 0.000s	sys 0.003s
I20260709 13:10:05.911396 25243 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:10:05.912689 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:05.913774 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 128214us
I20260709 13:10:05.914575 25936 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 13:10:05.915963 25243 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:10:05.883800 25320 raft_consensus.cc:1275] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:10:05.916653 25209 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:05.917253 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:05.914677 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:05.913866 25319 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 18
I20260709 13:10:05.919003 25209 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:05.920971 25386 raft_consensus.cc:2410] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 13:10:05.921319 25246 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:10:05.914459 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:05.921778 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:05.918139 25316 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:05.923997 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:05.927000 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:05.927461 25944 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 17: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:05.927147 25386 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:10:05.927769 25390 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:10:05.882892 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 50997.1us
I20260709 13:10:05.929574 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:05.847869 30512 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:05.885100 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:05.848192 30778 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:05.936308 25244 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 16: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:05.932219 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:05.935452 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:05.937387 25940 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:10:05.764416 30884 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:05.917466 31048 raft_consensus.cc:740] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:05.925616 31012 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:05.941975 25911 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:05.942975 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:05.943498 25242 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 1004) took 1860 ms (client timeout 1643 ms). Trace:
I20260709 13:10:05.950263 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:05.950328 25390 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:05.954668 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:05.951085 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 163119us
I20260709 13:10:05.936576 30512 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:05.883908 31049 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:05.960343 25212 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:05.913844 31023 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:10:05.849092 30559 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:10:05.965513 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:05.968140 25287 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:05.920478 30989 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:05.926008 31025 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:05.930634 30998 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:05.928472 30681 raft_consensus.cc:740] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:05.975102 25909 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:06.002034 25392 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:10:06.006150 25287 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:05.944772 30778 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:10:06.035900 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 60886.8us
W20260709 13:10:05.963452 31000 consensus_meta.cc:287] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.510s	user 0.008s	sys 0.002s
I20260709 13:10:06.036779 25909 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:05.973768 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:05.914247 30811 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.056s
I20260709 13:10:05.965163 30884 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:05.964569 30477 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:06.039981 25386 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 13:10:06.055783 31000 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15.
I20260709 13:10:06.013841 31025 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:06.081568 30681 consensus_queue.cc:260] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:06.022955 31023 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:06.010987 30866 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:10:06.050930 30856 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:06.051939 30896 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:06.096091 25319 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:06.102478 25389 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
W20260709 13:10:05.970018 25242 rpcz_store.cc:269] 0709 13:10:04.082946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:04.083441 (+   495us) service_pool.cc:224] Handling call
0709 13:10:05.943322 (+1859881us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b5400422554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:05.943430 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":542553984}
I20260709 13:10:06.098137 25386 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:10:06.055258 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:06.105398 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:05.974396 25393 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:06.058300 30965 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:06.117847 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 157568us
W20260709 13:10:06.094687 31015 consensus_meta.cc:287] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.564s	user 0.012s	sys 0.000s
I20260709 13:10:06.094511 31025 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:06.099689 31023 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:10:05.971418 31012 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:06.139425 31042 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.481s	user 0.000s	sys 0.002s
W20260709 13:10:06.242581 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 390327us
I20260709 13:10:06.106413 30896 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:06.225884 30856 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:06.116169 31049 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:06.232005 31012 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:06.285540 25361 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.013s	sys 0.019s
I20260709 13:10:06.124204 30559 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:10:06.270764 30896 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:06.144074 30681 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:06.241341 30988 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:06.219110 31060 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:06.139832 31015 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:06.266417 30864 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.656s	user 0.007s	sys 0.023s
W20260709 13:10:06.311067 25361 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.013s	sys 0.019s
I20260709 13:10:06.106088 31048 consensus_queue.cc:260] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:06.326305 31056 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
I20260709 13:10:06.326704 30988 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:06.330888 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:06.331920 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:06.332208 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:06.332810 25313 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:06.140627 30965 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:06.334666 25392 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:06.336922 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2469) took 1049 ms. Trace:
I20260709 13:10:06.338644 25239 rpcz_store.cc:276] 0709 13:10:05.287858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:05.288400 (+   542us) service_pool.cc:224] Handling call
0709 13:10:05.288819 (+   419us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:05.289068 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:05.289098 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:05.289640 (+   542us) log.cc:844] Serialized 26 byte log entry
0709 13:10:05.539422 (+249782us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:05.539531 (+   109us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:05.539631 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:06.336680 (+797049us) raft_consensus.cc:1685] finished
0709 13:10:06.336700 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:06.336844 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":249313,"threads_started":1,"wal-append.queue_time_us":301337}
I20260709 13:10:06.288328 30856 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:06.333470 30864 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.008s	sys 0.023s
I20260709 13:10:06.331238 31060 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:06.287976 30516 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260709 13:10:06.354189 25313 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:06.356611 25209 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:10:06.356825 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 339307us
I20260709 13:10:06.348647 30856 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:06.365420 25911 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:06.355090 31060 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:06.528663 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:06.545059 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 453548us
W20260709 13:10:06.382652 30969 consensus_meta.cc:287] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.811s	user 0.009s	sys 0.004s
I20260709 13:10:06.358819 30559 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election won for term 18
W20260709 13:10:06.519636 30978 consensus_meta.cc:287] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.670s	user 0.005s	sys 0.005s
I20260709 13:10:06.548485 25285 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:10:06.557787 25909 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.028s	sys 0.020s
W20260709 13:10:06.559743 25909 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.028s	sys 0.020s
I20260709 13:10:06.448397 31021 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:06.560317 25387 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.912s	user 0.026s	sys 0.022s
W20260709 13:10:06.553354 25287 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.035s	sys 0.086s
W20260709 13:10:06.566447 25287 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.036s	sys 0.086s
W20260709 13:10:06.565840 25387 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.917s	user 0.026s	sys 0.022s
W20260709 13:10:06.561455 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 524729us
I20260709 13:10:06.568982 25909 leader_election.cc:400] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 13. Current term is 14.
W20260709 13:10:06.628041 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 70340.7us
I20260709 13:10:06.634804 25241 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:06.636376 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:06.640345 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:06.640645 30919 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:10:06.641278 25312 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:10:06.572257 30807 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:06.570465 30965 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:06.550379 30652 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:06.642278 30919 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:10:06.642922 30652 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:06.643739 30919 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:06.646481 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:10:06.569701 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 563657us
I20260709 13:10:06.610448 30988 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:06.637202 25909 leader_election.cc:403] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
W20260709 13:10:06.597179 30998 consensus_meta.cc:287] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.561s	user 0.007s	sys 0.001s
W20260709 13:10:06.649911 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 81136.9us
I20260709 13:10:06.563980 30864 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:06.654218 25909 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:06.647491 31026 consensus_meta.cc:287] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.053s	user 0.013s	sys 0.000s
I20260709 13:10:06.597975 30477 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:06.647167 30965 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:06.651474 30807 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 13. Current term is 14.
I20260709 13:10:06.562196 30969 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:06.687551 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 420244us
I20260709 13:10:06.553822 30811 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.095s
I20260709 13:10:06.656152 25909 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:06.648473 25287 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:06.692767 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 44436.5us
I20260709 13:10:06.594063 31021 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:06.662837 31015 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:06.684880 25359 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.030s	sys 0.044s
W20260709 13:10:06.699090 25359 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.030s	sys 0.045s
I20260709 13:10:06.654677 30864 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:06.693640 25287 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:06.692405 31057 log.cc:927] Time spent T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.458s	user 0.001s	sys 0.000s
W20260709 13:10:06.685105 31066 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.065s	user 0.000s	sys 0.004s
I20260709 13:10:06.692426 25909 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:06.685313 31041 log.cc:927] Time spent T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 1.163s	user 0.005s	sys 0.000s
I20260709 13:10:06.695083 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:06.684880 31045 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 1.043s	user 0.003s	sys 0.015s
W20260709 13:10:06.685631 31058 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.378s	user 0.003s	sys 0.000s
I20260709 13:10:06.564406 30978 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:06.692327 31067 log.cc:927] Time spent T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.066s	user 0.001s	sys 0.003s
I20260709 13:10:06.712723 30864 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:06.657198 25390 raft_consensus.cc:2410] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
W20260709 13:10:06.699925 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 575016us
W20260709 13:10:06.692500 31063 log.cc:927] Time spent T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.309s	user 0.002s	sys 0.000s
W20260709 13:10:06.692160 31061 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260709 13:10:06.690050 30965 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:06.705174 31077 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:10:06.705695 30512 consensus_meta.cc:287] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.706s	user 0.012s	sys 0.000s
I20260709 13:10:06.694520 30652 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:06.684705 30477 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:06.769464 31077 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:06.770108 31071 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:06.767942 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:06.770675 31071 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:06.768843 30965 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:06.771718 31071 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:06.769203 25314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2679) took 1399 ms. Trace:
W20260709 13:10:06.715510 25284 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.68:39211 timed out after 1.641s (SENT) [suppressed 21 similar messages]
W20260709 13:10:06.703027 30778 consensus_meta.cc:287] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.646s	user 0.011s	sys 0.000s
W20260709 13:10:06.739308 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 44309.3us
I20260709 13:10:06.688252 30856 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:06.777427 25358 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:06.715296 25287 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:06.740778 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:06.703801 30807 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:06.695600 31072 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:06.785959 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:06.786689 25937 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:10:06.742588 31075 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:06.772599 25314 rpcz_store.cc:276] 0709 13:10:05.369711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:05.645610 (+275899us) service_pool.cc:224] Handling call
0709 13:10:05.646209 (+   599us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:05.646448 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:05.646488 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:05.658651 (+ 12163us) log.cc:844] Serialized 26 byte log entry
0709 13:10:06.101705 (+443054us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:10:06.103740 (+  2035us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:06.104852 (+  1112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:06.768930 (+664078us) raft_consensus.cc:1685] finished
0709 13:10:06.768959 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:06.769102 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":348046,"threads_started":1,"wal-append.queue_time_us":648890}
I20260709 13:10:06.694039 31073 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:06.795852 31073 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:06.795943 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:10:06.714278 30884 consensus_meta.cc:287] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.627s	user 0.012s	sys 0.000s
I20260709 13:10:06.764436 30927 raft_consensus.cc:3055] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:10:06.798934 30884 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:06.798964 30927 raft_consensus.cc:740] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:06.692183 31023 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:06.801929 31064 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.254s	user 0.000s	sys 0.001s
I20260709 13:10:06.803089 31023 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:06.802965 30927 consensus_queue.cc:260] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:06.804184 30927 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:10:06.805522 25244 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
I20260709 13:10:06.809700 31021 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:06.788023 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:06.789947 25941 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:06.772101 30477 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:06.788782 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:06.752779 31012 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:06.813562 25944 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:10:06.815016 30884 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:06.815477 31012 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:06.816174 31078 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:10:06.816181 30884 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:06.696166 30969 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:06.796412 31073 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:06.820652 30749 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:06.824733 30749 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:06.708972 31076 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.035s
I20260709 13:10:06.820648 30969 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:06.825714 30749 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:06.826879 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2775) took 1211 ms. Trace:
W20260709 13:10:06.827934 31058 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 13:10:06.769623 30512 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:06.795014 25358 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:06.830457 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:06.830878 25238 raft_consensus.cc:2393] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 15: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:06.821779 31079 raft_consensus.cc:2764] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 13:10:06.830309 30305 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:06.832568 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 2394) took 1020 ms. Trace:
I20260709 13:10:06.830739 30884 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:06.817788 31060 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:06.817150 31078 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:06.824182 30856 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:06.822430 30978 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:06.832979 25238 rpcz_store.cc:276] 0709 13:10:05.812321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:05.817421 (+  5100us) service_pool.cc:224] Handling call
0709 13:10:06.832390 (+1014969us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400364a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:06.832511 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2328224000}
I20260709 13:10:06.784555 30778 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:06.845570 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 1154) took 1221 ms. Trace:
I20260709 13:10:06.770124 31077 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:10:06.929908 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:06.827257 25391 rpcz_store.cc:276] 0709 13:10:05.615389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:05.643894 (+ 28505us) service_pool.cc:224] Handling call
0709 13:10:05.649040 (+  5146us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:05.650417 (+  1377us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:05.650456 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:05.651041 (+   585us) log.cc:844] Serialized 26 byte log entry
0709 13:10:05.840729 (+189688us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:10:06.826037 (+985308us) spinlock_profiling.cc:243] Waited 31.8 ms on lock 0x7b1800474280. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f297685a4bc 00007f2973ba848b 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc2455 00007f297a6aa3c7 00007f2976cbc9a7 00007f2976ccc0bf
0709 13:10:06.826211 (+   174us) log.cc:844] Serialized 14 byte log entry
0709 13:10:06.826462 (+   251us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:06.826547 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:06.826586 (+    39us) raft_consensus.cc:1685] finished
0709 13:10:06.826671 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:06.826807 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":284182,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":71423,"raft.queue_time_us":200,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"spinlock_wait_cycles":73242880,"thread_start_us":589190,"threads_started":2,"wal-append.queue_time_us":407789}
I20260709 13:10:06.858933 31075 raft_consensus.cc:697] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:06.939159 30483 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:10:06.939086 31075 consensus_queue.cc:237] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:06.940191 31071 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:06.921516 31079 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:06.941787 31079 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:06.944443 31079 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:06.946593 25212 leader_election.cc:400] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15.
I20260709 13:10:06.946969 25212 leader_election.cc:403] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
W20260709 13:10:06.837908 31025 consensus_meta.cc:287] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.619s	user 0.011s	sys 0.000s
I20260709 13:10:06.830389 30512 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:06.961211 31025 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:06.854853 30646 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:06.879444 25911 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.008s	sys 0.016s
I20260709 13:10:06.835812 31048 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:06.950837 30483 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15.
I20260709 13:10:06.956691 25212 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:06.946492 30681 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:07.007308 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 50683.4us
W20260709 13:10:06.958135 30896 consensus_meta.cc:287] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.624s	user 0.007s	sys 0.004s
I20260709 13:10:06.917238 25358 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:06.911341 25941 rpcz_store.cc:276] 0709 13:10:05.624207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:05.624320 (+   113us) service_pool.cc:224] Handling call
0709 13:10:05.819068 (+194748us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:06.811775 (+992707us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:06.812450 (+   675us) spinlock_profiling.cc:243] Waited 971 ms on lock 0x7b5400675554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:06.845360 (+ 32910us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b5400675550. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:06.845504 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2629453824}
I20260709 13:10:06.845453 31072 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:06.910295 30807 raft_consensus.cc:697] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:06.832885 31068 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:06.949230 30864 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:06.990463 25911 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.008s	sys 0.022s
W20260709 13:10:07.022815 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 657459us
I20260709 13:10:07.008782 25212 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:07.019780 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 102696us
I20260709 13:10:07.027130 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:06.946079 30905 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.064s
I20260709 13:10:06.840276 31073 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.039168 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:07.039505 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:07.046038 25242 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 13:10:07.046774 30749 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.047722 30305 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:07.048744 25239 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:07.026022 31072 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.006263 30778 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:06.952253 31078 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.053341 30749 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:07.055341 31078 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:07.056170 31055 log.cc:927] Time spent T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 1.117s	user 0.002s	sys 0.000s
I20260709 13:10:07.056551 31025 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:07.057928 25244 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:07.058420 31060 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:07.026441 31068 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:07.059123 31068 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:07.010267 30646 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.060338 30884 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:07.032938 30477 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:07.022228 30681 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.061646 30477 raft_consensus.cc:697] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:07.062563 31073 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:07.063185 30681 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:06.853595 31080 consensus_queue.cc:1059] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.178s
I20260709 13:10:07.063787 25395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2779) took 1447 ms. Trace:
I20260709 13:10:07.064198 25395 rpcz_store.cc:276] 0709 13:10:05.616444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:05.658102 (+ 41658us) service_pool.cc:224] Handling call
0709 13:10:05.704238 (+ 46136us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:06.139281 (+435043us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:06.139326 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:06.139857 (+   531us) log.cc:844] Serialized 26 byte log entry
0709 13:10:06.698721 (+558864us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:10:06.745667 (+ 46946us) log.cc:844] Serialized 14 byte log entry
0709 13:10:06.745857 (+   190us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:06.745930 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:07.063560 (+317630us) raft_consensus.cc:1685] finished
0709 13:10:07.063582 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:07.063721 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":92370,"prepare.queue_time_us":50482,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":146196,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":30542,"thread_start_us":812901,"threads_started":2,"wal-append.queue_time_us":804191}
I20260709 13:10:07.066334 25241 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:07.066737 30622 consensus_queue.cc:1059] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:07.061090 30646 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:07.067951 30622 raft_consensus.cc:3055] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:07.068327 30622 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:07.068775 30305 raft_consensus.cc:697] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:07.018476 30512 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:07.042186 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:07.072523 25386 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:10:07.032284 30760 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.026s
I20260709 13:10:07.062668 30477 consensus_queue.cc:237] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.026633 31079 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:10:07.083827 25392 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:07.085656 25241 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:07.092114 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:07.098260 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 59149us
I20260709 13:10:07.099104 25359 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:07.069506 30978 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:07.105862 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:07.022127 30896 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:07.081853 30512 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:07.109454 25391 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 11. (term mismatch)
I20260709 13:10:07.082578 30898 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:07.109426 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 462739us
I20260709 13:10:07.115304 25391 pending_rounds.cc:85] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 10
I20260709 13:10:07.118803 25391 pending_rounds.cc:107] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 11.11
I20260709 13:10:07.053011 31072 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:07.112696 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:07.119282 25391 raft_consensus.cc:2889] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:07.110759 31025 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:07.097707 30811 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:07.125329 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:07.107753 30659 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:10:07.082832 30856 consensus_queue.cc:1059] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.032s
I20260709 13:10:07.135578 25391 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:07.108078 31073 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:07.144191 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:07.145205 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:07.134434 31000 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.843s	user 0.017s	sys 0.006s
I20260709 13:10:07.152508 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:07.135481 25285 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.015s	sys 0.016s
I20260709 13:10:07.153223 25939 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
W20260709 13:10:07.154063 25285 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.015s	sys 0.016s
W20260709 13:10:07.154565 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 522272us
I20260709 13:10:07.140945 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:07.155467 25944 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:10:07.118733 30512 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:07.137595 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:07.110975 30969 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:07.156179 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 440945us
I20260709 13:10:07.139653 31062 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:07.087340 30622 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.114236 30905 consensus_queue.cc:1059] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.032s
I20260709 13:10:07.160844 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2786) took 1471 ms. Trace:
I20260709 13:10:07.166419 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:07.138092 30659 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:07.115442 30978 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.181465 25287 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:07.169931 25212 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:07.173431 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:07.176226 25945 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 1068) took 1304 ms. Trace:
W20260709 13:10:07.177266 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:07.150506 30559 raft_consensus.cc:697] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:10:07.148562 31000 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.870s	user 0.017s	sys 0.006s
I20260709 13:10:07.209473 31068 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.179720 31048 consensus_queue.cc:1059] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.036s
I20260709 13:10:07.175894 25388 rpcz_store.cc:276] 0709 13:10:05.689700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:05.806244 (+116544us) service_pool.cc:224] Handling call
0709 13:10:05.806548 (+   304us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:05.853687 (+ 47139us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:05.853719 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:05.856009 (+  2290us) log.cc:844] Serialized 26 byte log entry
0709 13:10:06.366103 (+510094us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:06.549942 (+183839us) log.cc:844] Serialized 14 byte log entry
0709 13:10:06.812017 (+262075us) spinlock_profiling.cc:243] Waited 9.69 ms on lock 0x7b180044b480. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc2455 00007f2976caa060 00007f2976cbc716 00007f2976ccd287 00007f2976ccd208
0709 13:10:07.146954 (+334937us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:07.147268 (+   314us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:07.147369 (+   101us) raft_consensus.cc:1685] finished
0709 13:10:07.160573 (+ 13204us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:07.160772 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46766,"prepare.queue_time_us":183537,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":61,"raft.queue_time_us":269872,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":30055,"spinlock_wait_cycles":22296832,"thread_start_us":740293,"threads_started":2,"wal-append.queue_time_us":778669}
I20260709 13:10:07.212503 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:07.152158 31080 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:07.213706 30559 consensus_queue.cc:237] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:07.214310 31080 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:07.165438 30989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.109s	user 0.013s	sys 0.018s
I20260709 13:10:07.182935 25287 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:07.143832 30991 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.034s
W20260709 13:10:07.216233 30989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.162s	user 0.013s	sys 0.019s
I20260709 13:10:07.216593 30989 raft_consensus.cc:697] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:07.129168 30898 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.217500 30989 consensus_queue.cc:237] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:07.218968 30898 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:07.219640 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2774) took 1604 ms. Trace:
I20260709 13:10:07.219846 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:07.220016 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:07.220538 25287 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:07.145325 30305 consensus_queue.cc:237] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.221405 31068 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:07.221635 25394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2772) took 1607 ms. Trace:
I20260709 13:10:07.222122 31093 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:10:07.222754 31093 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:07.222878 31076 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:07.223161 31093 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:10:07.117748 30807 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.172330 30969 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:07.221091 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:07.220028 25387 rpcz_store.cc:276] 0709 13:10:05.615074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:05.618157 (+  3083us) service_pool.cc:224] Handling call
0709 13:10:05.619781 (+  1624us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:06.569885 (+950104us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:06.569923 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:06.570580 (+   657us) log.cc:844] Serialized 52 byte log entry
0709 13:10:06.929350 (+358770us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:10:06.955632 (+ 26282us) log.cc:844] Serialized 14 byte log entry
0709 13:10:06.961534 (+  5902us) log.cc:844] Serialized 14 byte log entry
0709 13:10:06.962205 (+   671us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:06.962275 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:07.219458 (+257183us) raft_consensus.cc:1685] finished
0709 13:10:07.219475 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:07.219582 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5409,"prepare.queue_time_us":82876,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":95,"raft.queue_time_us":73011,"raft.run_cpu_time_us":332,"raft.run_wall_time_us":14660,"thread_start_us":1276943,"threads_started":2,"wal-append.queue_time_us":558472}
I20260709 13:10:07.229499 30969 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:07.183688 31087 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:07.189875 30978 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:07.181072 30659 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:10:07.230120 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:07.230232 31087 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.232558 31087 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:07.137868 31025 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.149402 31073 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:07.192544 31008 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:07.165899 31086 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:07.198204 31071 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:07.197985 30512 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:07.200156 30896 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:07.182612 25945 rpcz_store.cc:276] 0709 13:10:05.871848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:05.873342 (+  1494us) service_pool.cc:224] Handling call
0709 13:10:05.927851 (+ 54509us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:05.940912 (+ 13061us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:05.940951 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:05.941496 (+   545us) log.cc:844] Serialized 26 byte log entry
0709 13:10:06.307182 (+365686us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:10:06.307223 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:06.307323 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:06.801209 (+493886us) raft_consensus.cc:1685] finished
0709 13:10:06.801232 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:07.166528 (+365296us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b54005d0050. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:07.176093 (+  9565us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20637,"raft.queue_time_us":101,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":6483,"spinlock_wait_cycles":4448640,"thread_start_us":365227,"threads_started":1,"wal-append.queue_time_us":622578}
I20260709 13:10:07.201241 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:07.241515 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 58079.1us
I20260709 13:10:07.177779 31062 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:07.201680 25271 heartbeater.cc:646] Failed to heartbeat to 127.23.140.126:33305 (1 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.23.140.126:33305 timed out after 15.000s (SENT)
I20260709 13:10:07.205922 31099 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:07.206331 30864 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:07.207079 30843 raft_consensus.cc:3055] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:07.244450 25209 leader_election.cc:400] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15.
I20260709 13:10:07.222029 25394 rpcz_store.cc:276] 0709 13:10:05.614206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:05.614323 (+   117us) service_pool.cc:224] Handling call
0709 13:10:05.621801 (+  7478us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:05.627661 (+  5860us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:05.627708 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:05.628232 (+   524us) log.cc:844] Serialized 26 byte log entry
0709 13:10:07.055481 (+1427249us) spinlock_profiling.cc:243] Waited 4.71 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:07.186432 (+130951us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b1800171c40. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f297685a4bc 00007f2973ba848b 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f29768520d1
0709 13:10:07.186604 (+   172us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:07.186938 (+   334us) log.cc:844] Serialized 14 byte log entry
0709 13:10:07.221319 (+ 34381us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:07.221404 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:07.221443 (+    39us) raft_consensus.cc:1685] finished
0709 13:10:07.221460 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:07.221585 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36243,"prepare.queue_time_us":186,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":104966,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":23502,"spinlock_wait_cycles":49021440,"thread_start_us":81863,"threads_started":1,"wal-append.queue_time_us":226780}
I20260709 13:10:07.225208 25287 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:07.221812 31085 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:07.140756 30811 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.230808 25387 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:07.264252 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:07.265429 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:07.230193 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:07.216892 31080 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:07.238943 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:07.208542 30749 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:10:07.249868 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:07.266920 25911 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:07.246237 25209 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:07.244755 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 529254us
I20260709 13:10:07.278699 25392 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:07.280211 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:07.280792 25911 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:07.283314 25358 leader_election.cc:336] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.520s (SENT)
I20260709 13:10:07.284108 25358 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:07.286742 25387 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260709 13:10:07.294860 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 48709.3us
I20260709 13:10:07.295270 25388 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:07.240657 31094 consensus_queue.cc:1059] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260709 13:10:07.297686 25209 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:07.241024 31008 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:07.244004 31099 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:07.300074 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:07.309098 31090 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:10:07.301569 25936 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:07.310765 25239 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:07.246526 31087 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:07.311362 31099 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:07.312309 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:07.312757 25238 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:07.318054 25239 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 18: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:07.318498 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:07.313565 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:07.319227 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:07.313939 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:07.314241 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:07.321208 25243 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:07.314795 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 123309us
I20260709 13:10:07.320101 25239 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:07.327770 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:07.329491 25243 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:07.331827 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:07.332073 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:07.332403 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:07.332680 25937 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:07.286617 30864 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:07.294802 31060 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:07.296041 30978 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:07.295069 30866 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:07.246246 30843 raft_consensus.cc:740] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:07.340538 25319 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:10:07.341483 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:07.343194 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.344077 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:07.360721 25314 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:07.345341 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.347373 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:07.346529 25239 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:07.349726 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:07.286337 30681 raft_consensus.cc:2764] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election decision vote started in defunct term 14: won
I20260709 13:10:07.299544 30991 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:10:07.351130 30749 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.003s
I20260709 13:10:07.356355 31008 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:07.357386 30978 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:07.365047 25318 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:07.297819 30760 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 13:10:07.392248 30516 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:10:07.369166 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 85888.2us
I20260709 13:10:07.397467 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.360061 25320 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:07.401278 25319 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:07.295094 31077 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.403010 30978 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:07.404364 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.426133 25312 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
W20260709 13:10:07.426473 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 188411us
I20260709 13:10:07.427085 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:07.300856 30483 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.092s
I20260709 13:10:07.404632 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.406258 30811 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:07.407125 25284 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:07.406942 31071 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:10:07.408393 30512 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:07.409775 31025 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:07.410465 25318 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:07.397868 31000 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:07.413486 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:07.435768 25284 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:07.436170 25317 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:07.414057 30991 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:07.392722 30866 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.347955 31088 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:07.309871 31090 consensus_queue.cc:1059] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.083s
I20260709 13:10:07.427745 25321 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:07.428934 25315 raft_consensus.cc:2393] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 15: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:07.333966 30856 consensus_queue.cc:1059] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.068s
I20260709 13:10:07.358212 31095 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:07.456702 31095 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:07.368597 31080 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:07.459754 31080 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:07.432832 25314 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:07.392398 30864 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:07.472694 30864 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:07.475826 30864 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:07.476330 31089 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.268s	user 0.001s	sys 0.000s
I20260709 13:10:07.476598 30864 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:07.395704 31060 raft_consensus.cc:697] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:10:07.439816 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 144341us
I20260709 13:10:07.243069 31086 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.477955 31060 consensus_queue.cc:237] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.478770 30622 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:07.431375 31096 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:07.480525 30864 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:07.480410 31096 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.481715 30864 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:07.485996 25316 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:10:07.494925 25241 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:07.432358 30516 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:07.500231 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 81069.3us
I20260709 13:10:07.503859 25317 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:07.507509 25321 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:10:07.431207 30843 consensus_queue.cc:260] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:07.514087 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 55154us
I20260709 13:10:07.487874 30807 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:07.511300 31091 log.cc:927] Time spent T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.305s	user 0.004s	sys 0.000s
I20260709 13:10:07.535657 25284 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:07.537537 25284 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:07.513612 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:07.487147 31088 raft_consensus.cc:697] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:07.452251 30991 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:07.516870 30969 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:07.507598 31076 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:07.552009 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:07.547266 25908 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:07.571637 25389 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2769) took 1959 ms (client timeout 2092 ms). Trace:
W20260709 13:10:07.558342 31053 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 1.502s	user 0.009s	sys 0.000s
W20260709 13:10:07.573503 25390 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41830 (request call id 1179) took 1633 ms (client timeout 2140 ms). Trace:
I20260709 13:10:07.549947 31090 consensus_queue.cc:1243] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer f646dfd532974dceadfe00005d2bbf59 log is divergent from this leader: its last log entry 14.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
W20260709 13:10:07.572237 25389 rpcz_store.cc:269] 0709 13:10:05.612305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:05.612454 (+   149us) service_pool.cc:224] Handling call
0709 13:10:07.571461 (+1959007us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b54003228d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:07.571577 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1117033216}
W20260709 13:10:07.575018 31103 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 13:10:07.560710 30873 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:07.550858 31094 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.032s
I20260709 13:10:07.579543 31077 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:07.573256 30864 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.008s
I20260709 13:10:07.559135 30969 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:07.544595 30807 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.587060 30866 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:07.512616 30659 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.613848 31090 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:07.614435 31090 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.615347 30969 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:07.615607 30807 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:07.615953 31122 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:07.617581 31080 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:07.618338 31080 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:07.618978 31080 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:07.621742 31109 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
W20260709 13:10:07.625895 31107 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260709 13:10:07.627913 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.628767 25321 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:10:07.592859 30927 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:07.630828 30927 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.595451 30305 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:07.636001 30305 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:07.591638 25390 rpcz_store.cc:269] 0709 13:10:05.940149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:06.029345 (+ 89196us) service_pool.cc:224] Handling call
0709 13:10:07.573308 (+1543963us) spinlock_profiling.cc:243] Waited 552 ms on lock 0x7b5400147154. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:07.573448 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1269247744}
I20260709 13:10:07.555789 31073 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:07.560232 31076 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.644356 31076 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:07.543864 31115 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 13:10:07.651583 31112 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:07.653108 31112 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:07.654498 25318 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 13:10:07.655269 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.655555 25312 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: [14.8-16.9]
I20260709 13:10:07.656359 25315 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:07.657998 25313 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 13:10:07.624576 31088 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.659610 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.660473 25319 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:07.547484 31071 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 13:10:07.611068 30873 raft_consensus.cc:697] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:07.636488 30305 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:07.662334 31071 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:07.662081 30873 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.662894 31071 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.609488 30512 consensus_queue.cc:1059] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:10:07.603899 25209 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:07.598421 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2472) took 2040 ms. Trace:
I20260709 13:10:07.665096 31071 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:07.684597 31090 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:07.685732 31080 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:07.685631 31099 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:07.618309 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.665856 30512 raft_consensus.cc:3055] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Stepping down as leader of term 13
W20260709 13:10:07.665840 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 62017.7us
I20260709 13:10:07.668322 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.670508 25245 rpcz_store.cc:276] 0709 13:10:05.558075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:05.559467 (+  1392us) service_pool.cc:224] Handling call
0709 13:10:05.559755 (+   288us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:05.584753 (+ 24998us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:05.584785 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:05.585387 (+   602us) log.cc:844] Serialized 52 byte log entry
0709 13:10:05.761485 (+176098us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:10:05.793023 (+ 31538us) log.cc:844] Serialized 14 byte log entry
0709 13:10:05.793328 (+   305us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:06.844265 (+1050937us) spinlock_profiling.cc:243] Waited 24.6 ms on lock 0x7b5400456d54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:06.844313 (+    48us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:07.535560 (+691247us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x7b2400120758. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973a4b9c4 00007f2973a49acb 00007f2973c04dc9 00007f2973c04a55 00007f2973c0409d 00007f297683610e 00007f297684d88c 00007f297684cbb6 00007f297684c7d8 00007f2976856221
0709 13:10:07.598099 (+ 62539us) raft_consensus.cc:1685] finished
0709 13:10:07.598124 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:07.598336 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5011,"prepare.queue_time_us":32752,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":22813,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":185,"spinlock_wait_cycles":102098048,"thread_start_us":154768,"threads_started":1,"wal-append.queue_time_us":337627}
I20260709 13:10:07.673156 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.708102 25319 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:07.709582 25319 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 10. Current term is 11. Ops: []
I20260709 13:10:07.676133 25318 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:07.710215 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:07.677999 25321 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:07.678742 25315 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 13:10:07.679984 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.666163 31121 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:10:07.683724 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:07.693737 30927 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:07.694909 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:07.717459 25313 raft_consensus.cc:2393] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:07.717775 25314 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:10:07.719079 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:07.695590 30811 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:10:07.720793 25387 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:10:07.722048 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:07.728221 25393 raft_consensus.cc:2410] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
W20260709 13:10:07.729202 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:07.729300 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.725183 25319 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:07.654160 31112 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:07.731884 25393 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:07.732802 31080 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:07.733393 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:07.733376 31080 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:07.731873 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:07.735273 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.737200 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:07.737504 25395 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:07.737047 25319 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:07.706307 30512 raft_consensus.cc:740] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:07.742552 25319 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:07.744549 25911 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:07.745101 31080 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:07.742522 25209 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:07.748729 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.749876 25313 raft_consensus.cc:2410] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 15 because replica has last-logged OpId of term: 13 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 12.
I20260709 13:10:07.750468 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:07.751122 25386 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:07.747511 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.755980 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:07.756903 25318 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 15: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:07.757892 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:07.739037 30927 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:07.762010 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:07.767700 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:07.771052 25209 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:07.766297 25942 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:07.777331 25318 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:10:07.784169 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.786233 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:07.787818 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.788569 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:07.790385 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.790848 25209 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:07.792833 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:07.764439 31093 consensus_meta.cc:287] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.537s	user 0.011s	sys 0.001s
I20260709 13:10:07.751485 30905 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:10:07.745105 31112 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:07.801901 25209 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:07.801182 25358 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:07.808523 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:07.808750 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:07.761412 31075 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:07.746215 31087 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:07.821336 25944 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:07.769701 30559 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:07.807900 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.800468 30483 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:07.757403 30916 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 13:10:07.755599 31000 consensus_queue.cc:1048] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.054s
I20260709 13:10:07.819566 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:10:07.828088 31061 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 13:10:07.801597 31073 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:07.824971 31087 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:07.835063 30856 consensus_queue.cc:1059] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.008s
I20260709 13:10:07.809558 30989 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:07.765169 30898 consensus_queue.cc:1059] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.063s
I20260709 13:10:07.788048 30969 consensus_queue.cc:1059] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.045s
I20260709 13:10:07.846848 25209 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:07.828274 31121 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:07.827508 31126 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:07.789511 30927 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:07.831146 30559 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:07.824635 31093 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.827106 31076 consensus_queue.cc:1059] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 13:10:07.862118 31087 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:07.856737 31124 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:10:07.844189 30811 consensus_queue.cc:1059] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 13:10:07.865831 31125 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:07.854671 30905 consensus_queue.cc:1059] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 13:10:07.884196 31126 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:07.920233 25312 raft_consensus.cc:2368] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11.
I20260709 13:10:07.899901 30559 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:07.862561 30978 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:07.928407 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 64655.7us
I20260709 13:10:07.912374 31125 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:07.935685 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.922294 31126 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:07.952235 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:07.926079 31099 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:07.958004 25313 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:10:07.958624 30927 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:07.958555 30305 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:07.966410 31131 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
W20260709 13:10:07.968554 31128 log.cc:927] Time spent T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 13:10:07.970312 31008 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:07.971026 30978 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:07.970893 31008 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.941534 31125 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:07.974423 31062 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:07.951697 30749 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:07.976297 30749 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:07.981462 25245 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 17 index: 8. (index mismatch)
I20260709 13:10:07.982450 25237 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:10:07.983428 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:07.984314 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:07.985002 31099 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:07.981626 25241 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 13:10:07.984375 25239 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:10:07.985600 31099 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.986572 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:07.985513 25386 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:07.988014 31099 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:07.986572 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:07.990888 30516 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:07.993005 31134 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:10:07.996826 30873 raft_consensus.cc:3055] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:10:07.997357 30873 raft_consensus.cc:740] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:07.997462 30305 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:07.998449 30873 consensus_queue.cc:260] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:07.999917 30873 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:08.000075 25240 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: [14.10-14.10]
W20260709 13:10:08.003752 25392 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.006s	sys 0.013s
W20260709 13:10:08.004129 25392 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.006s	sys 0.013s
I20260709 13:10:08.009781 31087 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:08.010768 30988 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:08.013875 31096 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:08.021963 25239 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: [14.12-16.13]
I20260709 13:10:08.014930 31085 raft_consensus.cc:3055] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:10:08.022507 31085 raft_consensus.cc:740] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:08.023464 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:08.011417 31098 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.798s	user 0.000s	sys 0.004s
I20260709 13:10:08.019785 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:08.020498 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:08.024361 25239 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:08.025935 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:08.036517 25314 raft_consensus.cc:1275] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
W20260709 13:10:08.036530 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 277061us
I20260709 13:10:08.041442 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:08.042248 25390 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:08.054284 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:08.055727 25317 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:10:08.057081 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:08.057050 25390 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:08.126696 25911 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:08.128509 25243 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 15: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:08.039551 30989 raft_consensus.cc:697] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 17 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:08.136373 31080 raft_consensus.cc:3055] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:08.136802 30483 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:08.137619 30483 consensus_queue.cc:1059] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:10:08.138779 31113 raft_consensus.cc:3055] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:08.139235 31113 raft_consensus.cc:740] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:08.140230 31113 consensus_queue.cc:260] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:08.143916 30807 raft_consensus.cc:3055] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:10:08.144714 31079 raft_consensus.cc:3055] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:08.081564 30998 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:08.038411 31099 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:08.134182 30898 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 13:10:08.166611 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 345814us
I20260709 13:10:08.038854 30988 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:08.170244 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 390203us
I20260709 13:10:08.023533 31085 consensus_queue.cc:260] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:08.173285 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:08.174063 25317 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:08.175197 25313 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:10:08.178777 25314 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:08.136831 31080 raft_consensus.cc:740] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:08.150976 31079 raft_consensus.cc:740] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:08.150682 30807 raft_consensus.cc:740] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:08.183312 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:08.184072 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 57953.3us
I20260709 13:10:08.192287 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:08.164428 31113 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:08.157272 31121 raft_consensus.cc:697] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 17 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:08.165546 31073 raft_consensus.cc:697] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:08.194583 30969 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.010s
I20260709 13:10:08.191102 30898 raft_consensus.cc:697] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:08.161983 31099 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.204921 31145 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:08.204896 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:08.210199 31121 consensus_queue.cc:237] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.213337 31111 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.021s
I20260709 13:10:08.214423 31087 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:08.238145 25390 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:08.181362 31124 raft_consensus.cc:697] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:08.217298 31080 consensus_queue.cc:260] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.059450 31126 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:08.235224 31099 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:08.216311 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:08.246559 25315 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:08.161798 30998 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.240198 25358 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:08.251108 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:08.241566 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:08.220621 31145 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.222905 30516 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:08.222236 31085 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:10:08.248085 25242 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:08.183218 30559 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.256003 25358 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:08.257293 25209 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:08.197001 30305 consensus_queue.cc:1059] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260709 13:10:08.243916 30843 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:08.270109 31100 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.902s	user 0.007s	sys 0.002s
I20260709 13:10:08.248924 31096 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:08.273030 31151 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:08.248781 30898 consensus_queue.cc:237] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:08.279692 31049 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:08.283617 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:08.285921 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:08.287822 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:08.289937 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:08.291023 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:08.291734 25318 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:08.293737 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:08.295240 25317 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:08.296833 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:08.276562 30645 raft_consensus.cc:3055] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:08.260599 30516 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:08.301685 25212 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:08.302042 25317 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: [14.8-16.9]
I20260709 13:10:08.263331 30622 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:10:08.266146 31080 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:08.308009 25243 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:08.298761 30645 raft_consensus.cc:740] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:10:08.308140 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:08.308457 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:08.309062 25244 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:10:08.308642 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:08.308701 25209 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:08.307552 25317 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:10:08.318150 25241 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 10. Current term is 11. Ops: []
I20260709 13:10:08.318861 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:08.319773 31126 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:08.280308 31151 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.277489 30998 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:08.278802 30988 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:08.277652 30927 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:08.243430 30989 consensus_queue.cc:237] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.334326 31142 raft_consensus.cc:3055] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:08.277665 31105 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:08.348381 25237 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 15: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:08.348091 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:08.375123 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 66530.6us
I20260709 13:10:08.375383 25390 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:08.299484 30760 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:08.314477 30516 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:08.363806 30559 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:08.338183 31156 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:08.292865 31146 consensus_queue.cc:1059] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.116s
I20260709 13:10:08.365852 31112 consensus_queue.cc:1059] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:10:08.366740 30778 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:10:08.368320 30477 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:08.335559 31124 consensus_queue.cc:237] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:08.373997 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:08.413659 25393 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
W20260709 13:10:08.361104 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 59536.3us
I20260709 13:10:08.377391 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:08.414722 25390 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:08.391264 25394 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:08.415216 25358 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:08.395233 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:08.416069 25395 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:08.338311 30864 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.049s
W20260709 13:10:08.416816 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 46142.5us
I20260709 13:10:08.343694 30622 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:08.369367 31142 raft_consensus.cc:740] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:08.412391 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:08.415295 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:08.343906 31048 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:08.355476 25315 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:08.387034 31087 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:08.449070 25389 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:08.450520 25387 raft_consensus.cc:3055] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:08.453058 25387 raft_consensus.cc:740] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:08.394788 31024 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.058s
I20260709 13:10:08.384161 30919 consensus_queue.cc:1059] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 13:10:08.459818 25908 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:08.464869 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:08.349401 31073 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:08.471390 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:08.420578 31105 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:08.473605 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:08.475932 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 49237.9us
I20260709 13:10:08.445364 31145 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:08.388537 30760 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:08.418675 30884 consensus_queue.cc:1059] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 13:10:08.467487 25387 consensus_queue.cc:260] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.492637 25387 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:08.484325 25315 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:10:08.457567 31087 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:08.489671 25395 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:08.474270 25388 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:08.466476 30477 raft_consensus.cc:697] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:08.456238 31090 consensus_queue.cc:1059] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 13:10:08.479406 31060 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:08.480556 31105 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:08.488930 31071 raft_consensus.cc:3055] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:08.522047 31163 consensus_queue.cc:1059] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:10:08.522504 30919 raft_consensus.cc:3055] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:08.585292 30919 raft_consensus.cc:740] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:08.547485 31071 raft_consensus.cc:740] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:08.585598 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:08.586700 31071 consensus_queue.cc:260] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.587303 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:08.588079 31071 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:08.525527 25394 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 17 index: 8. (index mismatch)
I20260709 13:10:08.523535 31168 raft_consensus.cc:3055] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:08.525426 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2703) took 1180 ms. Trace:
I20260709 13:10:08.526127 31153 consensus_queue.cc:1059] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.167s
I20260709 13:10:08.589401 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:08.589866 31153 raft_consensus.cc:3055] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:08.590312 31153 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:08.524300 30659 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:08.539778 31093 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:08.541188 31008 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:08.591116 31153 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:08.466734 30873 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260709 13:10:08.501909 31087 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:08.547184 31105 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:08.559051 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:08.589077 25316 rpcz_store.cc:276] 0709 13:10:07.344887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:07.345047 (+   160us) service_pool.cc:224] Handling call
0709 13:10:08.525244 (+1180197us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b5400274f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:08.525368 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":285611392}
I20260709 13:10:08.596725 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:08.597841 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:08.560974 31163 raft_consensus.cc:3055] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:08.555145 25358 leader_election.cc:400] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11.
I20260709 13:10:08.546267 31142 consensus_queue.cc:260] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.556010 31146 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:08.561039 30760 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:10:08.588977 31168 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:10:08.613333 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 148538us
W20260709 13:10:08.570021 31125 consensus_meta.cc:287] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.596s	user 0.011s	sys 0.000s
I20260709 13:10:08.574142 31079 consensus_queue.cc:260] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.575917 30749 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:08.578964 30512 consensus_queue.cc:260] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:08.525377 31152 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
I20260709 13:10:08.582571 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:10:08.584465 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:08.586652 30919 consensus_queue.cc:260] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.588135 25389 raft_consensus.cc:2393] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:08.589977 25394 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:08.461655 31048 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.465950 30622 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.469041 31137 consensus_queue.cc:1059] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
I20260709 13:10:08.598661 25358 leader_election.cc:403] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
W20260709 13:10:08.635490 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 81968us
I20260709 13:10:08.636749 25287 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:08.555807 31159 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
W20260709 13:10:08.603209 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 293932us
I20260709 13:10:08.604147 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:08.644101 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:08.645372 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:08.647255 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:08.648948 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:08.650704 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:08.656533 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:08.668831 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:08.608363 25387 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:10:08.568203 30477 consensus_queue.cc:237] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:08.615933 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:08.616775 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:08.677840 25316 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:08.618455 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:08.595746 31153 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:08.603032 31163 raft_consensus.cc:740] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:08.607653 31146 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:08.626148 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:08.632220 25391 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:10:08.631472 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:08.669091 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:08.691819 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 60474.6us
I20260709 13:10:08.692759 25209 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:08.669557 25394 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:08.669677 25386 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:10:08.669848 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:08.603708 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 185550us
I20260709 13:10:08.694038 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:08.694357 25246 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:08.694552 25285 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:08.674242 25387 raft_consensus.cc:1275] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 12. (term mismatch)
I20260709 13:10:08.696223 25387 pending_rounds.cc:85] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 11
I20260709 13:10:08.627120 31168 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:08.696491 25387 pending_rounds.cc:107] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 17.12
I20260709 13:10:08.696379 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:08.638867 31079 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:08.674882 30512 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:10:08.686699 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 197688us
I20260709 13:10:08.673552 30873 consensus_queue.cc:1059] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:08.695670 25287 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:08.690426 25318 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:10:08.713923 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:08.698276 25392 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: [13.9-14.10]
I20260709 13:10:08.646992 31008 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 13:10:08.699461 25209 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:08.678303 31177 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:08.677155 30919 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:08.700606 25285 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:08.681440 31000 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 18 current term: 18 is higher.
I20260709 13:10:08.656859 30516 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:08.684464 25389 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:08.626001 31125 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.715544 25212 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:08.715868 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:08.696919 25387 raft_consensus.cc:2889] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:08.717063 25394 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: [14.10-16.11]
I20260709 13:10:08.657200 30927 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 13:10:08.733206 25209 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:08.685719 31137 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:10:08.647490 31166 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.074s
I20260709 13:10:08.697271 31146 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:08.752182 25239 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:10:08.729650 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:08.701524 30559 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:08.756716 25212 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:08.724465 31168 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:08.745813 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:08.723448 31093 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:08.706561 30864 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:08.740228 30884 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:08.707046 30988 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:08.778128 30988 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:08.778595 30988 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:08.753049 31085 consensus_queue.cc:1048] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.031s
W20260709 13:10:08.757827 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 98778.6us
I20260709 13:10:08.784499 31095 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:08.785652 31169 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
I20260709 13:10:08.793171 31180 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:10:08.740633 25246 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: [14.12-16.13]
I20260709 13:10:08.724866 31099 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:08.804344 30516 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:08.804863 31099 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:08.805397 31099 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:08.737717 31171 consensus_queue.cc:1059] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.082s
I20260709 13:10:08.762995 25391 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:10:08.732177 31156 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:08.765290 25237 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14. Ops: []
I20260709 13:10:08.765419 25394 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:10:08.748034 25389 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 13:10:08.767860 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:08.769109 25212 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:08.766343 30387 consensus_queue.cc:1059] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.005s
I20260709 13:10:08.766477 31111 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:08.769497 31025 consensus_queue.cc:1059] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:10:08.770933 31090 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:08.772413 31154 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:08.773288 30559 raft_consensus.cc:697] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:08.845386 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:08.746975 30778 consensus_queue.cc:1059] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.027s
I20260709 13:10:08.773931 25318 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 13:10:08.774290 31170 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:08.775905 31125 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:08.777539 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:08.798557 25315 raft_consensus.cc:1217] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Deduplicated request from leader. Original: 15.10->[17.11-17.11]   Dedup: 17.11->[]
I20260709 13:10:08.855938 25315 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
W20260709 13:10:08.802526 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 146053us
I20260709 13:10:08.808799 25239 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:08.821655 31171 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:08.818454 25313 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:08.819984 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:08.792943 31165 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.262s	user 0.000s	sys 0.001s
W20260709 13:10:08.822258 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 53230.8us
I20260709 13:10:08.824102 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:08.825171 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:10:08.830403 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:08.768991 31093 raft_consensus.cc:697] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:08.843045 25241 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 17 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:08.844936 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:08.851125 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:08.867663 25936 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:08.815055 31156 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:08.818783 31068 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 10. Current term is 11.
I20260709 13:10:08.743717 31177 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:08.821455 31111 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:08.856900 25909 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:08.815837 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:08.880692 25246 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:08.882028 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:08.857174 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:08.859356 31181 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:08.866688 25942 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:08.780416 31105 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:08.892390 25909 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:08.842315 31090 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:08.877048 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:08.888403 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 1022) took 1575 ms. Trace:
I20260709 13:10:08.855404 31095 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:08.894550 25239 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 17 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 13:10:08.872020 31156 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:08.863852 31048 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:08.900470 25242 rpcz_store.cc:276] 0709 13:10:07.312778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:07.312915 (+   137us) service_pool.cc:224] Handling call
0709 13:10:07.313194 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:08.260413 (+947219us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:08.888024 (+627611us) spinlock_profiling.cc:243] Waited 935 ms on lock 0x7b5400456854. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:08.888303 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2149618944}
I20260709 13:10:08.902272 25320 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:10:08.865710 31175 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:10:08.851889 31154 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.870807 31171 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:08.927215 31171 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:08.902906 31113 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:08.907835 30559 consensus_queue.cc:237] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:08.914948 31137 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:08.946513 25242 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14. Ops: []
I20260709 13:10:08.917656 31105 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:08.921589 25942 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:08.922772 30988 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:08.890080 31111 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:08.901724 31090 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:08.949422 25393 raft_consensus.cc:2368] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17.
W20260709 13:10:08.950191 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 70054.2us
I20260709 13:10:08.949422 25359 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:08.896333 30785 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:08.875626 30873 consensus_queue.cc:1059] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.065s
W20260709 13:10:08.947782 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 124421us
I20260709 13:10:08.960920 25212 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:08.963795 25359 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:08.973472 31105 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:08.974681 31093 consensus_queue.cc:237] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.976267 30988 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:08.961609 31137 raft_consensus.cc:697] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:08.979339 25244 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:08.980335 25941 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:08.981885 25359 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:08.977694 31137 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:08.986786 25359 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:08.992034 25359 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
W20260709 13:10:08.994225 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 101923us
I20260709 13:10:08.940711 31095 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:08.998624 25359 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:09.000309 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:09.003844 31188 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 13:10:09.008567 25936 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:09.010636 25359 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:09.012599 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 145556us
I20260709 13:10:09.014174 25909 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:09.012630 30927 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:09.015830 30927 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:09.016343 30927 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.017176 31086 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:08.993234 31105 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:09.020845 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:09.000144 31048 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:08.995500 30516 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:09.029202 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:09.010125 31062 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:08.904270 31181 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:09.034715 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 73879.8us
I20260709 13:10:09.047801 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:09.013302 31075 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:09.026063 31025 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 13:10:09.026697 31059 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:09.038012 30927 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:09.033284 30991 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:09.033927 31048 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:09.050253 31086 consensus_queue.cc:1059] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.015s
I20260709 13:10:09.019132 30785 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.044s
I20260709 13:10:09.046788 31062 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:09.070397 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:09.044986 31144 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:09.042483 30305 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:09.010986 30760 consensus_queue.cc:1059] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 13:10:09.076022 25936 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 10.
I20260709 13:10:09.012445 31156 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:09.080070 25392 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15. Ops: []
W20260709 13:10:09.093113 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:09.038173 30873 consensus_queue.cc:1059] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 13:10:09.085346 25395 raft_consensus.cc:1275] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:10:09.085938 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:09.088408 25943 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:10:09.088581 25940 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
W20260709 13:10:09.099912 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:09.100070 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:09.101903 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:09.104012 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:09.083981 31048 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:09.085021 31062 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:09.111110 25212 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:09.112543 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:09.119261 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:09.101361 30927 raft_consensus.cc:3055] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:10:09.095712 31144 raft_consensus.cc:697] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 16 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:09.084399 31087 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.126348 25245 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:09.120622 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:09.084985 31172 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.041s
I20260709 13:10:09.134932 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:09.136178 25936 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:10:09.138813 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:09.140733 31184 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 13:10:09.123311 30873 consensus_queue.cc:1243] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 15.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:10:09.120260 31177 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:09.142189 31146 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:09.148779 31191 log.cc:927] Time spent T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 13:10:09.150401 25938 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:09.124749 30927 raft_consensus.cc:740] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:09.152012 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:09.158634 25387 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:10:09.153668 31146 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:09.141034 30559 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:10:09.151726 31175 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:09.144639 31157 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260709 13:10:09.187143 31156 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:09.188550 25387 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
W20260709 13:10:09.184743 25284 leader_election.cc:336] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Timed out: RequestConsensusVote RPC to 127.23.140.68:39211 timed out after 1.641s (SENT)
I20260709 13:10:09.191833 25284 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:09.193377 31192 log.cc:927] Time spent T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:10:09.209935 25936 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:10:09.210551 25392 pending_rounds.cc:85] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 11
I20260709 13:10:09.210916 25392 pending_rounds.cc:107] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260709 13:10:09.211301 25392 raft_consensus.cc:2889] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:09.175040 31181 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:09.214425 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:09.215703 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:09.217249 25387 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:09.217962 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:09.219044 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:09.219551 25936 raft_consensus.cc:2376] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13. Re-sending same reply.
W20260709 13:10:09.211778 31079 consensus_meta.cc:287] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.504s	user 0.012s	sys 0.000s
I20260709 13:10:09.188282 31025 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.030s
I20260709 13:10:09.181646 31175 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.225153 25395 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 9. (term mismatch)
I20260709 13:10:09.229377 25395 pending_rounds.cc:85] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 8
I20260709 13:10:09.229646 25395 pending_rounds.cc:107] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 16.9
I20260709 13:10:09.229759 25387 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:09.229910 25395 raft_consensus.cc:2889] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:09.188215 31144 consensus_queue.cc:237] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:09.238735 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:09.240245 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:09.242741 31079 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:09.243325 31079 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.247277 31079 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:09.268152 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:09.258816 31111 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:09.274005 30778 consensus_queue.cc:1243] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer a2a77ecf2c0941618b5bc79d4c04db61 log is divergent from this leader: its last log entry 16.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:10:09.274765 30778 raft_consensus.cc:3055] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:09.275290 30778 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:09.272814 31111 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:09.288486 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 103791us
W20260709 13:10:09.289883 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:09.294263 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:09.296373 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:09.297605 25284 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:09.276341 30778 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.306432 30778 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:09.308216 30622 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:09.309840 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:09.309942 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:09.310631 25243 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:09.313436 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:09.314459 25238 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 17
I20260709 13:10:09.313651 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:09.316358 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:09.316979 25246 raft_consensus.cc:2393] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:09.317413 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:09.317536 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:09.318239 25243 raft_consensus.cc:2393] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:09.320992 25287 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:09.286101 31087 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:09.323036 25388 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: [14.10-16.11]
I20260709 13:10:09.252837 30807 consensus_queue.cc:260] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.325171 25284 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:09.326409 25238 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:09.293929 30919 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:09.330242 25359 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:09.332980 25313 raft_consensus.cc:1813] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Handling vote request from an unknown peer a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:09.333698 25313 raft_consensus.cc:2410] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 16 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 13:10:09.329579 30387 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:09.338366 30864 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:10:09.338912 30864 raft_consensus.cc:697] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:09.339805 30864 consensus_queue.cc:237] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.333771 30622 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:09.345928 30622 raft_consensus.cc:697] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:09.346832 30622 consensus_queue.cc:237] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.346997 31062 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:09.348827 25945 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 1070) took 2021 ms (client timeout 1924 ms). Trace:
W20260709 13:10:09.349320 25945 rpcz_store.cc:269] 0709 13:10:07.327238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:07.327469 (+   231us) service_pool.cc:224] Handling call
0709 13:10:09.348612 (+2021143us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400168854. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:09.348754 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2586150784}
I20260709 13:10:09.350646 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:09.321660 30856 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:09.347035 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:09.328399 31078 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:09.368897 30749 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:10:09.370105 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:09.358492 30856 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.374892 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:09.379844 31163 consensus_queue.cc:260] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.300307 31099 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:09.357061 30919 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.004s
I20260709 13:10:09.360929 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:09.345151 31059 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:10:09.385797 31204 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 13:10:09.386209 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:09.388667 25940 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:09.388964 25238 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14. Ops: []
I20260709 13:10:09.387878 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:09.390282 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:09.390832 25319 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:10:09.368692 31078 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:09.335168 31153 consensus_meta.cc:287] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.642s	user 0.013s	sys 0.000s
I20260709 13:10:09.359033 25284 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:09.404726 25240 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 17. Current term is 18. Ops: [15.7-17.8]
W20260709 13:10:09.404951 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 46005.5us
W20260709 13:10:09.382972 31200 log.cc:927] Time spent T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260709 13:10:09.383203 31149 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:09.391037 25246 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:09.322901 31175 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:09.392686 30905 consensus_queue.cc:1059] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 13:10:09.393501 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:09.419548 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:09.419970 25358 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:09.393767 31062 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:09.395639 30919 consensus_queue.cc:1059] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:10:09.363325 30927 consensus_queue.cc:260] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:09.401254 25394 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 13:10:09.407301 25284 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:09.409462 31090 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:09.421257 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:09.429988 31068 raft_consensus.cc:3055] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:10:09.431205 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:09.431799 30905 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:09.434473 25246 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:09.445254 25284 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:09.453438 25359 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:09.456234 25284 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:09.457721 31079 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:10:09.458341 31079 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:09.458849 31079 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:10:09.447882 31068 raft_consensus.cc:740] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:09.453447 30749 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:09.462378 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:09.467571 25945 raft_consensus.cc:2393] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 16: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:09.443063 31105 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:10:09.474398 30305 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:09.473464 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:09.471201 31024 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:10:09.470557 30927 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:09.451567 31177 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260709 13:10:09.502811 31078 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:09.499387 30305 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:09.506649 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:09.507710 25246 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:09.509449 25359 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:09.532366 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:10:09.533828 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:09.536072 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:09.536741 25246 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:09.442791 31149 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.540277 31149 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:09.541324 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:09.460968 31068 consensus_queue.cc:260] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.504611 31060 raft_consensus.cc:3055] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:10:09.545454 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:09.549957 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:09.521543 30905 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:09.551424 25391 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:10:09.534626 31024 raft_consensus.cc:697] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 13 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:09.542791 31172 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:09.573836 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:09.547065 31099 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:09.551085 31060 raft_consensus.cc:740] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:09.551337 30988 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:09.565477 30905 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:09.551321 30749 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.036s
I20260709 13:10:09.586599 30778 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:09.577698 31012 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:09.585420 31168 consensus_meta.cc:287] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.820s	user 0.009s	sys 0.005s
I20260709 13:10:09.596180 30905 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:09.581835 31172 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.650398 31024 consensus_queue.cc:237] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.656095 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:09.660220 31079 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:09.631520 31071 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:09.598590 30988 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.592939 30305 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:09.607167 31012 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.650908 31090 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:09.703404 31168 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:09.704543 31185 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
W20260709 13:10:09.763195 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 218726us
I20260709 13:10:09.817072 25285 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:09.647460 31060 consensus_queue.cc:260] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.819734 31060 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:10:09.736550 31088 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.875s	user 0.012s	sys 0.024s
I20260709 13:10:09.702935 30645 consensus_queue.cc:260] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:09.830159 31216 log.cc:927] Time spent T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260709 13:10:09.830597 30645 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:09.823302 31088 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.968s	user 0.012s	sys 0.024s
W20260709 13:10:09.837492 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 268738us
W20260709 13:10:09.839303 31206 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.533s	user 0.001s	sys 0.000s
I20260709 13:10:09.840014 30681 consensus_queue.cc:1059] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20260709 13:10:09.840746 31210 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
I20260709 13:10:09.845481 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:10:09.846681 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:09.852991 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:09.816973 31168 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.846715 31105 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:09.854993 31168 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:09.849140 31180 raft_consensus.cc:3055] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:09.854732 31105 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:09.849517 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:09.859398 25394 raft_consensus.cc:2393] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:09.841099 30681 raft_consensus.cc:3055] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 13:10:09.894290 31211 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20260709 13:10:09.895887 31212 log.cc:927] Time spent T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:10:09.843104 30988 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:09.842293 31151 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:09.855967 31167 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:09.855751 31180 raft_consensus.cc:740] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:10:09.909372 31221 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 13:10:09.886479 30681 raft_consensus.cc:740] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:10:10.093662 30927 consensus_meta.cc:287] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.594s	user 0.013s	sys 0.004s
I20260709 13:10:09.849269 31062 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:09.897604 31170 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.017s	sys 0.025s
I20260709 13:10:10.096675 31167 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:10.103614 31170 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.030s	sys 0.051s
I20260709 13:10:10.114365 31167 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:10.120050 31180 consensus_queue.cc:260] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.116271 31172 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.137642 31167 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.140563 31167 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.180989 25314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 1001) took 1358 ms. Trace:
W20260709 13:10:10.181619 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:10.183050 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:10.184180 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2742) took 1113 ms. Trace:
W20260709 13:10:10.178628 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:10.184760 25317 rpcz_store.cc:276] 0709 13:10:09.070500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:09.281640 (+211140us) service_pool.cc:224] Handling call
0709 13:10:09.282045 (+   405us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:09.311911 (+ 29866us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:09.311947 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:09.312539 (+   592us) log.cc:844] Serialized 52 byte log entry
0709 13:10:09.595961 (+283422us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:10:09.596010 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:09.596085 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:10.183504 (+587419us) raft_consensus.cc:1685] finished
0709 13:10:10.183550 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:10.183937 (+   387us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17434,"raft.queue_time_us":22863,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":134,"thread_start_us":283004,"threads_started":1,"wal-append.queue_time_us":628976}
W20260709 13:10:10.197101 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:10.182054 25314 rpcz_store.cc:276] 0709 13:10:08.822740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:08.861895 (+ 39155us) service_pool.cc:224] Handling call
0709 13:10:08.862657 (+   762us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:09.277566 (+414909us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:09.277594 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:09.277963 (+   369us) log.cc:844] Serialized 26 byte log entry
0709 13:10:09.580642 (+302679us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:09.580686 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:09.580756 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:10.180046 (+599290us) raft_consensus.cc:1685] finished
0709 13:10:10.180126 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:10.180654 (+   528us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":436206,"raft.queue_time_us":417136,"raft.run_cpu_time_us":68,"raft.run_wall_time_us":2385,"thread_start_us":302373,"threads_started":1,"wal-append.queue_time_us":427605}
W20260709 13:10:10.201419 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:10.201421 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 486121us
I20260709 13:10:09.867184 30387 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:10.219782 30387 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.222913 25237 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:10:10.224191 30387 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.225965 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:10.227416 25237 raft_consensus.cc:2393] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:10.228960 30387 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:10.233660 30622 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:10.234386 25937 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:10.236611 30681 consensus_queue.cc:260] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.245983 30387 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.246731 30622 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.253293 30864 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:10.254467 30864 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:10.281162 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 52147.4us
I20260709 13:10:10.264766 30387 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:10.114952 31079 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:10.293757 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:10.293954 31024 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:10.306288 31024 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.296777 30864 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:10.294852 30305 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:10.309551 30864 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:10.310840 30864 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:10.312381 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:10.106217 31062 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.315462 31000 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:10.316375 31062 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.317750 31167 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:10.318761 31167 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:05.768175 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:10:10.321408 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 457040us
I20260709 13:10:10.322388 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2494) took 1567 ms. Trace:
I20260709 13:10:10.322917 25244 rpcz_store.cc:276] 0709 13:10:08.754723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:08.754977 (+   254us) service_pool.cc:224] Handling call
0709 13:10:08.755349 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:08.979994 (+224645us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:10.321745 (+1341751us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b5400363154. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:10.322123 (+   378us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":514636672}
W20260709 13:10:10.327204 25245 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 2419) took 2309 ms (client timeout 1809 ms). Trace:
W20260709 13:10:10.327818 25245 rpcz_store.cc:269] 0709 13:10:08.017357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:08.025653 (+  8296us) service_pool.cc:224] Handling call
0709 13:10:10.326683 (+2301030us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400192b54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:10.326945 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2392687360}
I20260709 13:10:10.328320 31170 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:10.295269 31105 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.295887 30988 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:10.301299 31093 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:10.333562 31232 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:10.338573 31232 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.339068 31093 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.339795 25389 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:10:10.329605 31170 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.351791 31232 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:10.357424 25388 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:10:10.354570 30988 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.360009 25388 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: [14.10-16.11]
I20260709 13:10:10.361718 25389 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:10:10.316521 31000 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.360996 31170 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:10.383882 31060 consensus_meta.cc:287] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.564s	user 0.023s	sys 0.000s
I20260709 13:10:10.384068 31000 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.397723 30919 raft_consensus.cc:3055] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:10.398394 30919 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:10.403888 31012 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.409950 31075 raft_consensus.cc:3055] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:10:10.410610 31075 raft_consensus.cc:740] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:10.412156 31075 consensus_queue.cc:260] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.417732 31096 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:10.418462 31096 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:10.422710 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:10.423817 25192 placement_policy_util.cc:248] tablet eda233ace2a24b379fef320a5f610c7a: not considering replicas for movement since the tablet's status is 'UNAVAILABLE'
I20260709 13:10:10.424818 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:10.425746 25317 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:10:10.426232 31113 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:10.426802 31113 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:10:10.373836 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 484937us
I20260709 13:10:10.374238 31168 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.055s
I20260709 13:10:10.383072 25940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 1049) took 1336 ms. Trace:
I20260709 13:10:10.374650 30622 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:10.372118 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:10.433107 25943 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 12.
I20260709 13:10:10.433679 25943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 1173) took 1191 ms. Trace:
I20260709 13:10:10.433822 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:10.434026 25943 rpcz_store.cc:276] 0709 13:10:09.242340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:10.371505 (+1129165us) service_pool.cc:224] Handling call
0709 13:10:10.433590 (+ 62085us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:10.435458 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:10.435608 31142 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:10.436450 25943 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 16: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:10.399920 30919 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.438108 31145 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:10.439199 31145 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.439750 31068 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:10.431265 25940 rpcz_store.cc:276] 0709 13:10:09.046887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:09.151784 (+104897us) service_pool.cc:224] Handling call
0709 13:10:10.382657 (+1230873us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b54004ad8d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:10.382863 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":533446016}
I20260709 13:10:10.416710 31095 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:10.370302 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:10.442514 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 1060) took 1345 ms. Trace:
I20260709 13:10:10.364171 30988 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.444824 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:10.458261 31180 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:10.374305 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:10.363988 31167 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.374681 30864 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:10.461254 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2492) took 1737 ms. Trace:
I20260709 13:10:10.423808 25314 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:10:10.464804 25241 rpcz_store.cc:276] 0709 13:10:08.723270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:08.723812 (+   542us) service_pool.cc:224] Handling call
0709 13:10:08.724122 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:08.867180 (+143058us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:10.460636 (+1593456us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b54003ec454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:10.461026 (+   390us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":272871168}
I20260709 13:10:10.375202 31093 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:10.468689 31145 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:10.379787 31234 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:10.375532 25941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 1078) took 2609 ms (client timeout 1933 ms). Trace:
W20260709 13:10:10.474567 25941 rpcz_store.cc:269] 0709 13:10:07.766242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:07.766440 (+   198us) service_pool.cc:224] Handling call
0709 13:10:10.374995 (+2608555us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54006a31d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:10.375242 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2738836736}
I20260709 13:10:10.364528 31024 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:10.473806 31234 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.476019 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:10.481601 31075 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:10.483307 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:10.484463 25316 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 11: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:10.480166 31234 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.480895 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 1170) took 1252 ms. Trace:
W20260709 13:10:10.473701 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 45525.8us
I20260709 13:10:10.438359 30919 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:10.362639 31048 consensus_meta.cc:287] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.243s	user 0.008s	sys 0.020s
I20260709 13:10:10.485086 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:10.518936 30884 raft_consensus.cc:3055] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:10.519564 30884 raft_consensus.cc:740] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:10.488327 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:10.520828 25314 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:10.488703 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:10.522593 30622 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:10.523074 30622 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:10.523439 30622 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:10.528872 30884 consensus_queue.cc:260] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.530491 30884 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:10.531771 31154 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:10.489260 31095 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:10.533669 31095 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.534736 30622 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.535989 31095 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:10.537793 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 52858us
W20260709 13:10:10.538930 31218 log.cc:927] Time spent T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.719s	user 0.001s	sys 0.000s
I20260709 13:10:10.539121 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:10.442929 25942 rpcz_store.cc:276] 0709 13:10:09.096937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:10.369614 (+1272677us) service_pool.cc:224] Handling call
0709 13:10:10.442425 (+ 72811us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:10:10.542239 31227 log.cc:927] Time spent T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.419s	user 0.001s	sys 0.000s
I20260709 13:10:10.543654 30516 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:10.544564 30516 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.493036 25243 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:10:10.546006 31090 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:10.546790 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2438) took 1007 ms. Trace:
I20260709 13:10:10.546603 31090 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:10.494746 25386 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 2113) took 1964 ms (client timeout 2155 ms). Trace:
I20260709 13:10:10.547252 25243 rpcz_store.cc:276] 0709 13:10:09.539429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:09.542007 (+  2578us) service_pool.cc:224] Handling call
0709 13:10:09.542304 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:10.545675 (+1003371us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:10.546413 (+   738us) spinlock_profiling.cc:243] Waited 950 ms on lock 0x7b54003eb554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:10.546699 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2184743808}
W20260709 13:10:10.548151 25386 rpcz_store.cc:269] 0709 13:10:08.529939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:08.648597 (+118658us) service_pool.cc:224] Handling call
0709 13:10:10.494262 (+1845665us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x7b5400234954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:10.494492 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":45155584}
I20260709 13:10:10.507961 25395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2833) took 1427 ms. Trace:
W20260709 13:10:10.506814 25393 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 2821) took 2093 ms (client timeout 1746 ms). Trace:
I20260709 13:10:10.549160 25395 rpcz_store.cc:276] 0709 13:10:09.080101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:09.096608 (+ 16507us) service_pool.cc:224] Handling call
0709 13:10:09.097004 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:09.230441 (+133437us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:10.507390 (+1276949us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b54003d4ad4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:10.507742 (+   352us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":293491968}
W20260709 13:10:10.549955 25393 rpcz_store.cc:269] 0709 13:10:08.414923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:08.415056 (+   133us) service_pool.cc:224] Handling call
0709 13:10:10.506293 (+2091237us) spinlock_profiling.cc:243] Waited 488 ms on lock 0x7b54001211d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:10.506582 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1123180416}
I20260709 13:10:10.516197 30856 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:10.487725 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:10.537119 31214 log.cc:927] Time spent T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.408s	user 0.001s	sys 0.002s
I20260709 13:10:10.494627 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2617) took 1198 ms. Trace:
I20260709 13:10:10.552480 25316 raft_consensus.cc:2410] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 17 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 13:10:10.552657 25313 rpcz_store.cc:276] 0709 13:10:09.295671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:09.296004 (+   333us) service_pool.cc:224] Handling call
0709 13:10:10.493072 (+1197068us) spinlock_profiling.cc:243] Waited 35.7 ms on lock 0x7b5400043dd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:10.494388 (+  1316us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":82169088}
I20260709 13:10:10.542109 31086 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:10.553982 31154 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:10.554124 31086 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:10.554579 31086 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:10.554546 31154 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.556005 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:10.557271 30516 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:10.550096 31090 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:10.556999 31154 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:10.558856 25944 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 13:10:10.559621 25944 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1053) took 1511 ms. Trace:
I20260709 13:10:10.560099 25944 rpcz_store.cc:276] 0709 13:10:09.048055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:10.558056 (+1510001us) service_pool.cc:224] Handling call
0709 13:10:10.558509 (+   453us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:10.559315 (+   806us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:10.559535 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:10.505913 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:10.497575 25285 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:10.563040 25943 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
W20260709 13:10:10.563524 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 66439.7us
I20260709 13:10:10.565073 25943 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:10.565291 31024 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:10.565898 25943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1057) took 1510 ms. Trace:
I20260709 13:10:10.565995 31024 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.566310 25943 rpcz_store.cc:276] 0709 13:10:09.055865 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:10.564198 (+1508333us) service_pool.cc:224] Handling call
0709 13:10:10.564698 (+   500us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:10.565634 (+   936us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:10.565836 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:10.566921 31167 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:10:10.568070 25943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1059) took 1504 ms. Trace:
I20260709 13:10:10.567967 31167 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:10:10.569829 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:10.568424 31024 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:10.492272 25941 rpcz_store.cc:276] 0709 13:10:09.229020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:10.475566 (+1246546us) service_pool.cc:224] Handling call
0709 13:10:10.480803 (+  5237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:10.571264 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:10.508276 25319 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:10:10.537029 30622 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:10.578162 31059 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:10.578670 25358 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:10.578933 31059 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.579962 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:10.582744 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:10.583436 30622 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:10.583984 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:10.583436 25316 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 14
I20260709 13:10:10.568576 25943 rpcz_store.cc:276] 0709 13:10:09.064028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:10.567091 (+1503063us) service_pool.cc:224] Handling call
0709 13:10:10.567477 (+   386us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:10.567718 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:10.567741 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:10.567783 (+    42us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:10:10.567841 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:10.567897 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:10.568001 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:10:10.582978 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:10.600272 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:10.603331 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:10.604346 30516 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:10.604701 25395 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 13:10:10.605144 30516 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.607012 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:10.609781 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:10.577764 31154 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:10.614578 31154 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:10.614907 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:10.615159 31154 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:10:10.616535 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 262180us
I20260709 13:10:10.607774 30516 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:10.615762 31059 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:10.618981 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:10.628504 25314 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:10.619297 25312 raft_consensus.cc:2410] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 13:10:10.622051 25940 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 13:10:10.623131 25943 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:10.623525 25942 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:10.623941 31095 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:10:10.614624 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:10.625900 25937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 1086) took 1802 ms (client timeout 1600 ms). Trace:
I20260709 13:10:10.630517 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:10.631699 25937 rpcz_store.cc:269] 0709 13:10:08.823766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:08.823873 (+   107us) service_pool.cc:224] Handling call
0709 13:10:10.625665 (+1801792us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54006905d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:10.625822 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3144767872}
I20260709 13:10:10.635783 25940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1052) took 1583 ms. Trace:
I20260709 13:10:10.636448 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:10.636359 25940 rpcz_store.cc:276] 0709 13:10:09.047800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:10.557077 (+1509277us) service_pool.cc:224] Handling call
0709 13:10:10.616135 (+ 59058us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:10.629444 (+ 13309us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:10.631674 (+  2230us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:10.637532 25319 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:10.639416 25943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 1172) took 1397 ms. Trace:
I20260709 13:10:10.639856 25943 rpcz_store.cc:276] 0709 13:10:09.236552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:10.588452 (+1351900us) service_pool.cc:224] Handling call
0709 13:10:10.622797 (+ 34345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:10.633651 (+ 10854us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:10.633873 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:10.509776 31048 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.642107 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:10.634665 31095 raft_consensus.cc:697] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:10.651823 25394 raft_consensus.cc:2368] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:10:10.654419 25312 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 11. (term mismatch)
I20260709 13:10:10.652825 25358 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:10.659989 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:10.660166 31095 consensus_queue.cc:237] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.664669 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2431) took 1928 ms. Trace:
I20260709 13:10:10.654847 25312 pending_rounds.cc:85] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 10
I20260709 13:10:10.665568 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:10.665867 25312 pending_rounds.cc:107] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 16.11
I20260709 13:10:10.666267 25312 raft_consensus.cc:2889] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:10.666358 25244 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:10.667456 25240 rpcz_store.cc:276] 0709 13:10:08.736590 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:08.737205 (+   615us) service_pool.cc:224] Handling call
0709 13:10:08.752200 (+ 14995us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:09.405310 (+653110us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:10.664034 (+1258724us) spinlock_profiling.cc:243] Waited 652 ms on lock 0x7b540028afd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:10.664473 (+   439us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1499873536}
I20260709 13:10:10.669317 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:10.669569 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:10.670629 25321 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 19: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:10.670895 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:10.675531 25314 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:10.681792 25942 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: [15.11-15.11]
W20260709 13:10:10.691809 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:10.698828 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 1175) took 1443 ms. Trace:
I20260709 13:10:10.703377 25942 rpcz_store.cc:276] 0709 13:10:09.255697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:10.639223 (+1383526us) service_pool.cc:224] Handling call
0709 13:10:10.681437 (+ 42214us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:10.698453 (+ 17016us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:10.698743 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:10.682341 30516 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:10.715020 25318 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:10.714824 31090 consensus_queue.cc:1059] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:10:10.715407 31093 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:10.716413 31000 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:10.716172 31093 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.717238 31000 raft_consensus.cc:697] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:10.699128 31167 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:10.709283 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:10.715142 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:10.722600 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 93140.9us
I20260709 13:10:10.718073 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:10.723290 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:10.722724 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:10.723541 25240 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:10.728451 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:10.732913 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:10.736400 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:10.737475 25390 raft_consensus.cc:2410] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
W20260709 13:10:10.737815 30905 consensus_meta.cc:287] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.091s	user 0.009s	sys 0.017s
I20260709 13:10:10.739727 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:10.744577 31096 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.743039 25940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 1178) took 1454 ms. Trace:
I20260709 13:10:10.738862 25395 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:10.751324 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:10.754927 25244 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
W20260709 13:10:10.735265 25284 leader_election.cc:336] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Timed out: RequestConsensusVote RPC to 127.23.140.68:39211 timed out after 1.626s (SENT)
I20260709 13:10:10.748896 25940 rpcz_store.cc:276] 0709 13:10:09.288357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:10.637701 (+1349344us) service_pool.cc:224] Handling call
0709 13:10:10.678426 (+ 40725us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:10.735729 (+ 57303us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:10.735778 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:10.742804 (+  7026us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:10.742844 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:10.742909 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:10.742996 (+    87us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2967,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":5611,"spinlock_wait_cycles":19328}
I20260709 13:10:10.766462 25284 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:10.765309 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:10.772671 25388 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:10.732295 30516 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.778729 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:10.780396 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:10.780732 25318 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 13. (term mismatch)
I20260709 13:10:10.781174 25318 pending_rounds.cc:85] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 12
I20260709 13:10:10.781452 25318 pending_rounds.cc:107] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 16.13
I20260709 13:10:10.781714 25318 raft_consensus.cc:2889] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:10.782439 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:10.785328 31241 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:10.786558 31241 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.793395 31222 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:10.794131 31222 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.796406 31222 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.797932 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:10.802202 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:10.745965 31113 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:10.807407 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:10.808183 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:10.810915 25316 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:10.815102 25393 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:10:10.754616 30905 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.816172 25386 raft_consensus.cc:2410] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 13:10:10.814350 25314 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:10.821626 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:10.771221 31167 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.809938 31062 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:10:10.831310 25245 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:10.832983 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:10.833086 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:10.836640 25244 raft_consensus.cc:2393] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 14: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:10.833717 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:10.838742 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:10.819119 31093 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.836444 31229 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:10.816470 31095 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:10.840838 25394 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 16: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:10:10.840893 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 118103us
I20260709 13:10:10.844875 25358 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:10.818733 31223 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:10.848186 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:10.850452 25392 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16. Ops: []
W20260709 13:10:10.850522 25209 connection.cc:256] RPC call timeout handler was delayed by -1.0808951649999017s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16610000000000003s for a response
I20260709 13:10:10.851763 31113 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:10.851828 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:10.850553 31090 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:10.826349 31163 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:10.856076 25390 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:10.846302 31168 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:10.838462 31234 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:10.860283 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:10.861359 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:10.868254 30516 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.868502 31048 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.868853 30305 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:10.869356 30305 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:10.869766 30305 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:10.873624 30681 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:10.874133 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:10.874845 25391 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:10.873693 30896 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:10.876190 25939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 1047) took 1845 ms (client timeout 1549 ms). Trace:
I20260709 13:10:10.875847 30896 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:10.876657 25939 rpcz_store.cc:269] 0709 13:10:09.030410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:10.875775 (+1845365us) service_pool.cc:211] Skipping call since client already timed out
0709 13:10:10.875972 (+   197us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:10:10.841874 31059 consensus_queue.cc:1059] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.015s
I20260709 13:10:10.878580 31059 raft_consensus.cc:3055] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:10.878762 30896 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.879027 31059 raft_consensus.cc:740] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:10.879426 30896 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:10.879789 25317 raft_consensus.cc:2410] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 13:10:10.880722 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 1171) took 1646 ms. Trace:
I20260709 13:10:10.880960 31095 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.881191 25941 rpcz_store.cc:276] 0709 13:10:09.233937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:10.597761 (+1363824us) service_pool.cc:224] Handling call
0709 13:10:10.619228 (+ 21467us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:10.620194 (+   966us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:10.620240 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:10.621206 (+   966us) log.cc:844] Serialized 52 byte log entry
0709 13:10:10.767212 (+146006us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:10.767248 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:10.767309 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:10.880447 (+113138us) raft_consensus.cc:1685] finished
0709 13:10:10.880471 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:10.880641 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5557,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":106,"thread_start_us":145609,"threads_started":1,"wal-append.queue_time_us":205220}
I20260709 13:10:10.882884 31167 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.883452 31095 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:10.883617 31147 raft_consensus.cc:3055] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:10.884147 31147 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:10.887456 25241 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 13:10:10.888528 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:10.868220 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2842) took 1613 ms. Trace:
I20260709 13:10:10.880071 30896 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.889276 25387 rpcz_store.cc:276] 0709 13:10:09.255040 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:09.255205 (+   165us) service_pool.cc:224] Handling call
0709 13:10:09.280792 (+ 25587us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:09.281106 (+   314us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:09.281181 (+    75us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:09.281537 (+   356us) log.cc:844] Serialized 26 byte log entry
0709 13:10:10.365395 (+1083858us) spinlock_profiling.cc:243] Waited 1.8 ms on lock 0x7b180034aa40. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f29768520d1 00007f2976c6da0f
0709 13:10:10.755123 (+389728us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:10.761239 (+  6116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:10.771067 (+  9828us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:10.867275 (+ 96208us) raft_consensus.cc:1685] finished
0709 13:10:10.867294 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:10.868046 (+   752us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":101,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":4474,"spinlock_wait_cycles":4142336,"thread_start_us":352349,"threads_started":1,"wal-append.queue_time_us":1530652}
I20260709 13:10:10.890926 30896 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.891731 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:10.892513 25241 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:10.889148 25237 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:10.893796 31234 raft_consensus.cc:697] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 16 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:10:10.894659 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:10.894963 31234 consensus_queue.cc:237] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.897444 25391 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:10.831677 31000 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.901278 31093 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:10.901999 31093 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.902590 30807 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:10.904383 31093 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:10.905548 30978 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 13:10:10.910506 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 50288.6us
I20260709 13:10:10.913923 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:10.917497 31096 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.918829 31096 consensus_queue.cc:1059] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:10:10.920189 31096 raft_consensus.cc:3055] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:10:10.920634 31096 raft_consensus.cc:740] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:10:10.921870 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 59979.5us
W20260709 13:10:10.922926 31202 log.cc:927] Time spent T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 1.642s	user 0.003s	sys 0.000s
I20260709 13:10:10.924654 31241 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:10.868150 31236 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:10.925668 31236 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.925806 31090 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.928296 31236 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:10.880105 31059 consensus_queue.cc:260] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.930429 25386 raft_consensus.cc:2393] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 16: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:10.930703 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 1082) took 2110 ms. Trace:
I20260709 13:10:10.931041 25938 rpcz_store.cc:276] 0709 13:10:08.820245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:08.820389 (+   144us) service_pool.cc:224] Handling call
0709 13:10:08.820736 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:09.150856 (+330120us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:10.930258 (+1779402us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b5400703854. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:10.930523 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":757656960}
I20260709 13:10:10.932068 31177 consensus_queue.cc:1059] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:10:10.933035 31177 raft_consensus.cc:3055] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:10.933401 31177 raft_consensus.cc:740] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:10.933437 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:10.934219 25394 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:10.934695 31177 consensus_queue.cc:260] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.8, Last appended by leader: 8, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.885003 31147 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.935906 31147 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:10.937752 31123 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:10.938400 31123 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.939989 30622 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:10.940670 31123 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:10.932071 25391 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:10:10.942503 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:10.943154 25318 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:10.949709 25240 raft_consensus.cc:2410] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 13:10:10.951779 25285 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:10:10.952432 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:10.928297 31090 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:10.954624 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:10.955549 25317 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:10.955271 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:10.956352 31245 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260709 13:10:10.957525 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:10.959905 31096 consensus_queue.cc:260] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:10.924878 25212 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:10.859207 31229 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:10.963217 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 50706.2us
I20260709 13:10:10.964679 31232 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:10:10.964836 31168 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:10.965968 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:10.966308 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:10.967020 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:10.967401 31168 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:10.967830 25242 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 14: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:10.957532 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:10.969812 25313 raft_consensus.cc:2410] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 18 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
W20260709 13:10:10.965160 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:10.952694 25284 leader_election.cc:336] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Timed out: RequestConsensusVote RPC to 127.23.140.68:39211 timed out after 2.350s (SENT)
W20260709 13:10:10.977218 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 52496.4us
I20260709 13:10:10.977394 25241 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:10.948570 31095 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:10.983924 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:10.941084 30905 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:11.012692 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:10.980072 31168 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:10.998183 25285 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:10.943596 31086 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:11.023283 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:10.980165 31232 consensus_queue.cc:1059] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.036s
W20260709 13:10:11.026943 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:11.027683 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:11.028141 31113 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:11.028578 31113 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:11.029003 31113 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:11.031235 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 1223) took 1187 ms. Trace:
I20260709 13:10:11.031638 25392 rpcz_store.cc:276] 0709 13:10:09.843997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:09.850402 (+  6405us) service_pool.cc:224] Handling call
0709 13:10:09.850668 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:10.866614 (+1015946us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:11.030934 (+164320us) spinlock_profiling.cc:243] Waited 989 ms on lock 0x7b54001216d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:11.031169 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2274045824}
I20260709 13:10:11.033721 31060 consensus_queue.cc:1059] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:10:11.036224 31142 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:10:11.036274 31090 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:10:11.039215 31096 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:11.043545 31086 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:10.992478 30978 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:11.047475 30978 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.048730 31059 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:11.049557 31235 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:11.049917 30978 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:11.050391 31235 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:11.051550 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:11.052224 25245 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:11.054133 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:11.054872 25241 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:11.015239 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:11.056001 25242 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:11.056870 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:11.057451 25944 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1056) took 2001 ms. Trace:
I20260709 13:10:11.057894 25944 rpcz_store.cc:276] 0709 13:10:09.055640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:10.561401 (+1505761us) service_pool.cc:224] Handling call
0709 13:10:10.561823 (+   422us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:10.587407 (+ 25584us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:10.587493 (+    86us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:10.616638 (+ 29145us) log.cc:844] Serialized 26 byte log entry
0709 13:10:10.780015 (+163377us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:10.780054 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:10.780109 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:11.057199 (+277090us) raft_consensus.cc:1685] finished
0709 13:10:11.057220 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:11.057373 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9512,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":30824,"thread_start_us":128633,"threads_started":1,"wal-append.queue_time_us":342016}
I20260709 13:10:11.059012 30905 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:11.059590 25388 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:11.060362 31181 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:11.061043 31181 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.063310 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:11.064651 31154 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.065209 25238 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
W20260709 13:10:11.067308 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 48930.9us
I20260709 13:10:11.028908 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:11.071343 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 51463.7us
I20260709 13:10:11.072794 25318 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:11.082072 25359 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:11.083740 25359 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:11.087610 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:11.005812 31095 raft_consensus.cc:697] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:11.089893 25287 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:11.067107 31154 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:11.092116 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 111102us
W20260709 13:10:11.092070 25143 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53264 (request call id 208) took 19696 ms (client timeout 15000 ms). Trace:
I20260709 13:10:11.094151 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:11.097556 31236 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:11.101224 30807 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:11.102036 30807 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.102378 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:11.103096 31142 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:11.024401 31168 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:10:11.110395 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 94476.7us
I20260709 13:10:11.111413 25908 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:11.095505 25143 rpcz_store.cc:269] 0709 13:09:51.365958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:51.366258 (+   300us) service_pool.cc:224] Handling call
0709 13:10:11.062504 (+19696246us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:10:02.027660 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:10:02.441491 (+413831us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:10:02.461163 (+ 19672us) write_op.cc:435] Acquired schema lock
0709 13:10:02.463461 (+  2298us) tablet.cc:662] Decoding operations
0709 13:10:02.516325 (+ 52864us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:10:02.516424 (+    99us) write_op.cc:641] Partition lock acquired for write op
0709 13:10:02.516450 (+    26us) tablet.cc:685] Acquiring locks for 15 operations
0709 13:10:02.517631 (+  1181us) tablet.cc:701] Row locks acquired
0709 13:10:02.517687 (+    56us) write_op.cc:260] PREPARE: finished
0709 13:10:02.517871 (+   184us) write_op.cc:270] Start()
0709 13:10:02.518035 (+   164us) write_op.cc:276] Timestamp: P: 1783602602517805 usec, L: 0
0709 13:10:02.518060 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 13:10:02.518801 (+   741us) log.cc:844] Serialized 5861 byte log entry
0709 13:10:04.279321 (+1760520us) op_driver.cc:464] REPLICATION: finished
0709 13:10:04.655011 (+375690us) write_op.cc:301] APPLY: starting
0709 13:10:04.750077 (+ 95066us) tablet.cc:1367] starting BulkCheckPresence
0709 13:10:05.006675 (+256598us) tablet.cc:1370] finished BulkCheckPresence
0709 13:10:05.012480 (+  5805us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:10:05.316855 (+304375us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:10:05.354159 (+ 37304us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=30,key_file_lookups=30,delta_file_lookups=30,mrs_lookups=0
0709 13:10:05.354245 (+    86us) write_op.cc:312] APPLY: finished
0709 13:10:05.354839 (+   594us) log.cc:844] Serialized 139 byte log entry
0709 13:10:05.444112 (+ 89273us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:10:05.462075 (+ 17963us) write_op.cc:454] Released schema lock
0709 13:10:05.470576 (+  8501us) write_op.cc:341] FINISH: Updating metrics
0709 13:10:05.545988 (+ 75412us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:10:05.547752 (+  1764us) write_op.cc:454] Released schema lock
0709 13:10:05.760894 (+213142us) spinlock_profiling.cc:243] Waited 26.3 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
Metrics: {"mutex_wait_us":464672,"reported_tablets":35,"child_traces":[["op",{"apply.queue_time_us":317641,"apply.run_cpu_time_us":40040,"apply.run_wall_time_us":855865,"cfile_cache_hit":62,"cfile_cache_hit_bytes":66767,"num_ops":15,"prepare.queue_time_us":245402,"prepare.run_cpu_time_us":23618,"prepare.run_wall_time_us":737700,"raft.queue_time_us":841074,"raft.run_cpu_time_us":48189,"raft.run_wall_time_us":260962,"replication_time_us":1736565,"spinlock_wait_cycles":60425088,"thread_start_us":1036349,"threads_started":5,"wal-append.queue_time_us":393746,"wal-append.run_cpu_time_us":40607,"wal-append.run_wall_time_us":3572219}]]}
I20260709 13:10:11.112320 30978 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:11.113502 31093 consensus_queue.cc:1059] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:10:11.114816 30864 raft_consensus.cc:3055] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:11.115217 30864 raft_consensus.cc:740] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:10.997555 31123 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.092s
I20260709 13:10:11.107071 25241 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:11.068545 31181 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:11.036113 31147 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.052s
I20260709 13:10:11.119042 25359 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:11.129349 25359 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:11.116130 30864 consensus_queue.cc:260] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:11.104461 31223 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:11.091866 30622 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:11.059629 30905 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:11.089354 31235 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:11.117545 31177 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:11.138422 31166 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:10:11.131153 30884 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:11.131142 30807 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:11.140738 30905 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:11.140831 31181 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:11.141497 30864 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:10:11.141896 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 195107us
I20260709 13:10:11.142792 25938 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:10:11.145452 31167 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:10:11.134516 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 160949us
I20260709 13:10:11.146585 25209 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:11.148638 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:11.149466 31235 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 13:10:11.149792 31250 log.cc:927] Time spent T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 13:10:11.118420 30305 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:11.144402 31095 consensus_queue.cc:237] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:11.127378 25287 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:11.135206 25285 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:11.135914 31236 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:11.130692 31154 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:11.137670 31239 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:10:11.117341 31168 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:10:11.153179 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:11.162678 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:11.137760 25908 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:11.158352 30305 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:11.158358 31123 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:11.154145 25359 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:11.159929 30884 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:11.160115 31223 raft_consensus.cc:697] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:10:11.164507 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:11.177604 25287 leader_election.cc:400] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:10:11.178431 25287 leader_election.cc:403] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:10:11.173064 25285 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:11.174124 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:11.174661 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:11.181054 25245 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:11.175798 25936 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:11.183357 25359 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:11.172384 30884 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:11.185906 25945 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:10:11.186192 30884 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:11.187148 31180 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:11.188239 25392 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 18: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:11.187880 31180 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.188514 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:11.171936 30807 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:11.189160 25394 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
W20260709 13:10:11.189608 25936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 1066) took 1981 ms (client timeout 1626 ms). Trace:
I20260709 13:10:11.190133 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:11.190045 25936 rpcz_store.cc:269] 0709 13:10:09.208119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:09.224021 (+ 15902us) service_pool.cc:224] Handling call
0709 13:10:11.189425 (+1965404us) spinlock_profiling.cc:243] Waited 1.88 s on lock 0x7b5400690354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:11.189546 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4314270976}
W20260709 13:10:11.187223 25945 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 1168) took 1967 ms (client timeout 2022 ms). Trace:
I20260709 13:10:11.190845 25386 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
W20260709 13:10:11.191089 25945 rpcz_store.cc:269] 0709 13:10:09.219894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:09.531885 (+311991us) service_pool.cc:224] Handling call
0709 13:10:11.187030 (+1655145us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b5400460fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:11.187154 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3497160576}
I20260709 13:10:11.187228 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:11.194095 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:11.189834 31223 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.194691 25945 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:11.195724 25287 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:11.199132 31093 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:11.200340 25287 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:11.201112 31093 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:11.201622 31093 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:11.197494 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.202054 31093 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:11.197613 30884 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.207516 30919 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:10:11.208318 30785 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:11.208830 30785 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.209393 30884 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:11.213296 30807 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:11.213168 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:11.214123 25245 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:11.213967 30807 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.216293 30807 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:11.207151 25287 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:11.221736 31024 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:11.222222 31024 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:11.177574 31167 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:11.222530 31024 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:11.223340 25945 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:11.224632 31167 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:11.225425 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:11.167832 31235 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:11.178155 31181 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.052s
I20260709 13:10:11.230353 25287 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:11.235411 25317 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
W20260709 13:10:11.241442 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 58199.4us
W20260709 13:10:11.242081 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:11.172003 30622 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:10:11.251839 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 103322us
I20260709 13:10:11.192476 25141 catalog_manager.cc:5697] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 7 to 8, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 8 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:11.172508 31222 consensus_queue.cc:1059] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.074s
I20260709 13:10:11.259310 25284 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:11.182690 31229 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:11.261961 25394 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:10:11.261888 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.261888 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:11.263415 30305 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:11.266278 25940 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:10:11.266736 30305 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.266993 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.213037 30785 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:11.265753 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.245491 31123 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:11.269178 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:11.245769 31235 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:11.250885 31236 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:11.273237 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:11.275663 25941 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:11.275923 30305 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:11.274077 25386 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:11.277549 25209 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:11.255527 31167 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:11.278827 25245 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:11.275086 30919 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:11.280696 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:11.281026 30919 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.272879 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.281802 25141 catalog_manager.cc:5697] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 6 to 9. New cstate: current_term: 9 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:11.283468 30919 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:11.283722 25141 catalog_manager.cc:5697] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 6 to 10, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 10 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:10:11.272814 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:11.286350 25393 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:11.286409 30896 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:10:11.273169 31123 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:11.287349 30896 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:11.287576 31123 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:11.288050 30896 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.277030 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:11.290582 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.291894 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.292623 25942 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:10:11.293936 25943 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:10:11.294847 31180 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:11.295176 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.281479 25241 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:11.295648 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.296262 25937 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:11.297266 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.298635 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:10:11.300774 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:11.302037 25945 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:10:11.241398 30884 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20260709 13:10:11.307247 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.310191 25359 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:11.311618 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.265502 31239 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:11.311852 25141 catalog_manager.cc:5697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 8 to 9, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 9 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:11.312352 25939 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:11.314459 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.303098 25941 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:11.305243 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.320034 25936 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:10:11.275462 31235 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:11.321803 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.324098 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:11.333057 25938 raft_consensus.cc:2393] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 15: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:11.334008 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.299484 31167 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:11.334905 25943 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:11.372725 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:11.375464 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 65373.7us
I20260709 13:10:11.376533 25359 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:11.321352 30622 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:11.388108 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:11.389029 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.387964 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:11.392655 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 134606us
I20260709 13:10:11.388885 25937 raft_consensus.cc:2376] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18. Re-sending same reply.
I20260709 13:10:11.394822 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.397836 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.374452 31147 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:10:11.348471 31236 raft_consensus.cc:697] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:11.395870 25942 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:11.375437 31167 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:11.315142 31113 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.315105 31093 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:11.389288 31068 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:11.327448 31168 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.372362 31239 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:11.438534 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:11.440667 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.442121 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.445490 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.427628 25141 catalog_manager.cc:5697] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 5 to 10. New cstate: current_term: 10 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:11.446727 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.445369 25395 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:11.448360 25937 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:10:11.446151 25938 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:10:11.419320 30622 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:11.449260 31068 raft_consensus.cc:697] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:11.470773 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:10:11.480387 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:11.482590 31242 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:11.483174 31242 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:11.483546 31242 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:11.500244 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:11.520547 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:11.533262 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 497366us
W20260709 13:10:11.537045 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 445435us
I20260709 13:10:11.510344 30896 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:11.538671 25212 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:11.541312 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:11.513541 31168 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:11.525704 31234 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.010s	sys 0.017s
I20260709 13:10:11.556668 25212 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:11.566883 25243 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.018s	sys 0.021s
W20260709 13:10:11.567998 25243 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.018s	sys 0.021s
W20260709 13:10:11.569355 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 396293us
W20260709 13:10:11.570750 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 311530us
W20260709 13:10:11.571926 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:11.572178 25386 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
W20260709 13:10:11.563201 31234 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.010s	sys 0.018s
I20260709 13:10:11.581218 25285 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:11.594288 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:11.594540 25358 leader_election.cc:336] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Timed out: RequestConsensusVote RPC to 127.23.140.68:39211 timed out after 1.924s (SENT)
I20260709 13:10:11.574944 25141 catalog_manager.cc:5697] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 9 to 10, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 10 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:11.595304 25394 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:10:11.583680 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:11.596997 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.597896 25943 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:11.588804 25287 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:11.604809 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:11.606266 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:11.606767 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:11.607398 25316 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:11.609443 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:11.610224 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:11.610459 25321 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 9. (term mismatch)
I20260709 13:10:11.536968 31024 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.615794 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:11.616494 25312 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:10:11.617641 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:11.610879 25320 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:11.618670 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:11.618880 25246 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.026s	sys 0.031s
W20260709 13:10:11.619299 25246 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.026s	sys 0.031s
I20260709 13:10:11.610786 25321 pending_rounds.cc:85] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 8
I20260709 13:10:11.621368 25321 pending_rounds.cc:107] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 16.9
I20260709 13:10:11.621627 25321 raft_consensus.cc:2889] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:10:11.623224 31254 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.158s	user 0.001s	sys 0.000s
W20260709 13:10:11.625272 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:11.626163 31000 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:11.626741 31000 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:11.627441 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:11.630147 31151 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:11.630671 31151 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:11.631021 31151 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:11.633242 30864 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:11.593592 31068 consensus_queue.cc:237] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:11.582392 31239 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:11.634186 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:11.634727 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 54132us
I20260709 13:10:11.634886 25313 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:11.635044 30622 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:11.635524 25284 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:11.635830 31093 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:11.602210 31229 raft_consensus.cc:3055] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:11.636873 31229 raft_consensus.cc:740] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:11.637054 31113 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:11.637758 25911 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:11.620867 25141 catalog_manager.cc:5697] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a reported cstate change: config changed from index 10 to 4, term changed from 15 to 12, leader changed from <none> to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68), VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) added, now has a pending config: VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65), VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66), VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 12 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } attrs { replace: true } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } } pending_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:11.638825 31223 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:11.611222 30896 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:11.639286 31180 consensus_queue.cc:1059] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:10:11.616289 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:11.618593 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:11.636554 31093 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:11.642022 25319 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:11.637861 31229 consensus_queue.cc:260] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:11.647225 31166 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:11.652117 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:11.652987 25245 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:11.650944 31229 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:11.639863 30896 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:10:11.661692 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 84673.3us
I20260709 13:10:11.662874 25393 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:11.629119 31000 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:11.645915 31093 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:11.645464 31147 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:11.657874 31166 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:11.674278 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:11.683207 25238 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:11.676520 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:11.566934 31236 consensus_queue.cc:237] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.671690 30896 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:11.686779 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:11.679183 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:11.639405 31223 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:11.688957 25284 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:11.694837 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 96075.5us
I20260709 13:10:11.708518 25319 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:11.694273 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:11.679083 31093 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:11.686554 31166 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:11.733426 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:11.767225 25285 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:11.690233 25316 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:11.723096 25358 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:11.726826 31096 consensus_meta.cc:287] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.687s	user 0.010s	sys 0.002s
I20260709 13:10:11.727960 30905 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:11.694468 30622 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:10:11.733888 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 129165us
W20260709 13:10:11.769915 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 87223.6us
I20260709 13:10:11.787603 25245 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
W20260709 13:10:11.788311 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 54945.2us
I20260709 13:10:11.789409 25209 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:11.694096 31000 consensus_queue.cc:1059] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.074s
I20260709 13:10:11.774612 31222 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:10:11.805534 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 82535.8us
I20260709 13:10:11.797897 25287 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:11.792178 31123 consensus_meta.cc:287] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.504s	user 0.012s	sys 0.000s
I20260709 13:10:11.773017 30884 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:11.812374 25287 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:11.814577 25240 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:11.778365 30864 raft_consensus.cc:2764] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 13:10:11.821643 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2439) took 1255 ms. Trace:
I20260709 13:10:11.824028 31123 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.822830 31177 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:11.829552 31177 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:11.826145 25243 rpcz_store.cc:276] 0709 13:10:10.566500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:10.569266 (+  2766us) service_pool.cc:224] Handling call
0709 13:10:10.569679 (+   413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:10.570006 (+   327us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:10:10.810023 (+240017us) log.cc:844] Serialized 14 byte log entry
0709 13:10:11.818682 (+1008659us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:11.818772 (+    90us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:11.818855 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:11.820970 (+  2115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:11.821549 (+   579us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":276481,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":71238,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":13923,"thread_start_us":1191071,"threads_started":3,"wal-append.queue_time_us":961593}
I20260709 13:10:11.830075 31177 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:11.802474 31242 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.834707 31024 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:11.812325 31272 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.028s
I20260709 13:10:11.835091 31137 consensus_queue.cc:1059] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 15.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.123s
I20260709 13:10:11.836205 31242 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:11.836414 31137 raft_consensus.cc:3055] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:11.836781 31137 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:11.809842 31222 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:11.837584 31222 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:11.793668 31096 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:11.839308 31096 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:11.841995 31096 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:11.803930 31167 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:11.794546 30622 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:11.847137 30622 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:11.849697 25317 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14. Ops: []
I20260709 13:10:11.803093 31075 raft_consensus.cc:3055] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:11.857348 25316 raft_consensus.cc:1240] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:10:11.853871 25312 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:11.801671 31113 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:11.847000 31167 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:11.857723 31075 raft_consensus.cc:740] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:11.864007 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:11.836886 31242 raft_consensus.cc:3055] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:11.856622 25394 raft_consensus.cc:2368] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15.
W20260709 13:10:11.854189 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 43689.7us
I20260709 13:10:11.822423 31234 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:11.867323 25393 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:11.868278 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:11.847344 31000 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:11.851456 30919 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:11.847020 31090 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:11.836795 31273 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:11.872371 25312 raft_consensus.cc:2410] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 18 because replica has last-logged OpId of term: 17 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 6.
I20260709 13:10:11.807793 30905 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:11.872931 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:11.882769 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:11.863852 31113 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:11.868829 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 300752us
I20260709 13:10:11.886989 25321 raft_consensus.cc:2393] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 16: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:10:11.887425 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 387386us
I20260709 13:10:11.896214 25316 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:11.870868 31242 raft_consensus.cc:740] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:11.877701 30919 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:11.871361 31000 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:11.855508 30884 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:11.904477 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:11.905040 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:11.865866 31229 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
W20260709 13:10:11.905694 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:11.905906 25312 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 17: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:11.881582 31273 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:11.885808 31123 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:11.902865 30919 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:11.911222 31075 consensus_queue.cc:260] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:11.888010 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 293456us
I20260709 13:10:11.892207 31113 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:11.900121 31095 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:11.904963 31000 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:11.912452 31075 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:11.918879 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:11.918690 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 129413us
I20260709 13:10:11.938374 25312 raft_consensus.cc:2393] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 15: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:10:11.939005 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 126752us
I20260709 13:10:11.947177 25316 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:11.947341 25318 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 13:10:11.909845 31273 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:11.952400 25318 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:11.953227 25395 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16. Ops: [15.13-15.13]
I20260709 13:10:11.947546 25944 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:45428
I20260709 13:10:11.955063 25944 raft_consensus.cc:606] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:11.957783 30884 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:11.958678 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:11.959331 25391 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 10.
I20260709 13:10:11.959509 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:11.960147 25392 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:11.960285 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:11.914793 31137 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:11.961647 25321 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:10:11.962132 31137 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:11.965507 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:11.966184 25387 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:11.975296 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:11.977090 25316 raft_consensus.cc:2410] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 13:10:11.979082 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:11.979763 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 111920us
I20260709 13:10:11.981426 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:11.982591 25314 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 16: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:11.984572 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:11.985951 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:11.986120 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:11.986563 25939 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:11.986913 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:11.987534 25312 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:11.987412 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:11.988564 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:11.988139 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:11.994827 25245 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:10:11.994793 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:11.998328 25314 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:11.999635 25284 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:11.954308 31145 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:12.001641 25943 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 16: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:12.000805 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.004685 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.005419 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:12.005964 25243 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:12.006281 25318 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 13:10:12.006247 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:12.006970 25945 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:10:12.008764 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:12.009368 25242 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:12.009433 25245 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16. Ops: []
W20260709 13:10:12.012036 31276 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
I20260709 13:10:12.001415 31068 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.014s
W20260709 13:10:12.001152 31275 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
W20260709 13:10:12.016609 31235 consensus_meta.cc:287] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.640s	user 0.010s	sys 0.003s
I20260709 13:10:12.017097 31235 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.017982 30905 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:12.001574 31242 consensus_queue.cc:260] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.020233 31242 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:12.020939 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.021944 25246 raft_consensus.cc:2393] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 18: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:12.026314 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:12.028466 31060 raft_consensus.cc:3055] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Stepping down as leader of term 16
W20260709 13:10:12.031567 31261 log.cc:927] Time spent T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.235s	user 0.001s	sys 0.001s
W20260709 13:10:12.034036 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:12.034133 25284 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:12.031481 31151 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:12.039245 30864 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:12.028928 31060 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:12.038094 31290 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:12.041849 31290 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.043773 31166 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.039179 25942 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:45428
I20260709 13:10:12.046378 25358 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:12.045533 25942 raft_consensus.cc:606] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:12.047771 25942 raft_consensus.cc:614] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:10:12.050542 25242 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:12.023241 31093 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.056391 25392 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:12.056695 25387 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:12.057000 25284 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:12.037055 30884 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:12.040520 31060 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.058475 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.045221 31290 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:12.067348 25321 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:12.070582 25941 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 13:10:12.062261 30884 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:12.064610 31024 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:10:12.075073 31024 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:12.075711 31024 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:10:12.075995 31147 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:12.076475 31147 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:12.074589 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:12.067780 31154 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:12.077396 25938 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:12.079028 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:12.079488 25318 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:12.079738 25314 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:12.081058 25287 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:12.087260 25939 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:12.081319 31290 consensus_queue.cc:1059] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:12.090969 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.091454 25316 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 13:10:12.074512 31235 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:12.087469 25391 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:12.044876 31177 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:12.092216 25242 raft_consensus.cc:2410] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 13:10:12.090461 31168 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:12.035303 31145 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:12.056027 30864 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.036s
I20260709 13:10:12.089443 30884 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:12.096663 31168 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:12.097263 31168 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:12.099269 31145 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:12.100632 25237 raft_consensus.cc:1275] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:10:12.094497 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:12.101595 25318 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:10:12.095237 25284 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:12.089779 31060 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:12.106353 25318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:10:12.104290 25284 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:12.109373 25318 raft_consensus.cc:606] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:12.109963 25318 raft_consensus.cc:614] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:10:12.111284 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:12.113509 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 109601us
I20260709 13:10:12.114542 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:12.072582 31222 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:12.129793 31274 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.258s	user 0.000s	sys 0.004s
I20260709 13:10:12.131526 31222 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:12.132319 25315 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:12.136402 31093 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:12.137527 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:12.138150 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:12.138763 25938 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:12.139374 25395 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:12.142195 31146 raft_consensus.cc:993] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: : Instructing follower a2a77ecf2c0941618b5bc79d4c04db61 to start an election
I20260709 13:10:12.142706 30807 raft_consensus.cc:1081] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Signalling peer a2a77ecf2c0941618b5bc79d4c04db61 to start an election
I20260709 13:10:12.143064 25378 tablet_service.cc:1558] Processing DeleteTablet for tablet eda233ace2a24b379fef320a5f610c7a with delete_type TABLET_DATA_TOMBSTONED (TS a2a77ecf2c0941618b5bc79d4c04db61 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44978
I20260709 13:10:12.146180 30856 raft_consensus.cc:3055] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 13:10:12.146461 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:12.146668 30856 raft_consensus.cc:740] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:12.112177 31242 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.035s
I20260709 13:10:12.122182 25908 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:12.123814 30898 consensus_queue.cc:1059] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.006s
I20260709 13:10:12.120533 31290 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:12.125768 31177 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:12.148743 30898 raft_consensus.cc:3055] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Stepping down as leader of term 16
W20260709 13:10:12.126256 31285 log.cc:927] Time spent T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 13:10:12.148765 31290 raft_consensus.cc:697] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:12.110611 30864 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:12.150204 30864 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:12.150596 30864 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:12.150888 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.150264 31290 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.126194 31180 consensus_queue.cc:1059] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:10:12.127034 31280 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:10:12.152660 25287 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:12.149080 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:12.152770 31280 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:12.153380 31075 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:12.162271 25287 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:12.164387 25240 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:12.153702 25239 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:12.128352 30884 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.166265 31272 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:12.166849 31272 raft_consensus.cc:697] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:12.167033 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:12.167838 31272 consensus_queue.cc:237] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:12.169028 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:12.169085 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:12.169617 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:12.167917 30884 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:12.169991 25238 raft_consensus.cc:2368] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15.
I20260709 13:10:12.170275 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.170743 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:12.155565 25908 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:12.118886 31259 consensus_queue.cc:1059] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 13:10:12.176133 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:12.176266 30807 consensus_queue.cc:1059] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:10:12.176561 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:12.177047 25388 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 18: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:12.149207 30898 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:12.174013 31075 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:12.178187 31075 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:12.178145 30898 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.179275 30898 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:12.172915 31180 consensus_queue.cc:1059] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.008s
I20260709 13:10:12.169157 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:12.181315 31180 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:12.181792 31180 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:12.182132 31180 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:12.107982 30919 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:12.186199 25285 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:12.186547 25313 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:12.167840 25241 raft_consensus.cc:2393] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 14: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:12.188650 30919 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:12.189397 25395 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:41830
I20260709 13:10:12.190032 25395 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:10:12.190382 25395 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:12.190490 25285 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:12.191085 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.191664 25285 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:12.192523 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:12.192572 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.205022 25287 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:12.205600 25387 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:12.199090 25392 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:12.192971 25943 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:12.208395 25284 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:12.228032 25387 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:12.193789 25241 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 12. Current term is 14. Ops: []
I20260709 13:10:12.219208 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:12.240335 25239 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:12.241873 25938 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:12.243264 25240 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:12.247563 31299 raft_consensus.cc:3055] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:12.248204 31299 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:12.249372 31299 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.265820 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:12.267942 25391 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:10:12.268605 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 113006us
W20260709 13:10:12.273973 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 54898.5us
I20260709 13:10:12.275015 25287 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:12.273005 31024 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.172035 31280 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:10:12.279794 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 157100us
I20260709 13:10:12.192143 31093 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:12.193244 31177 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:12.241299 30884 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:12.208865 31151 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:12.293377 31167 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:12.245000 30919 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:12.255192 31270 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:12.313329 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:10:12.303336 30896 consensus_meta.cc:287] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.574s	user 0.009s	sys 0.001s
I20260709 13:10:12.319729 25395 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:12.315325 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 87823.9us
I20260709 13:10:12.281638 31259 consensus_queue.cc:1243] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer 2329e428ebbf484fa3ea9452133df567 log is divergent from this leader: its last log entry 19.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:10:12.328205 25391 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:12.331260 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.336196 25239 raft_consensus.cc:2410] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 13:10:12.315047 31166 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:12.321390 30919 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:12.315171 31177 raft_consensus.cc:697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:10:12.334980 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 60103.4us
W20260709 13:10:12.336474 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 53166.5us
I20260709 13:10:12.342689 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:12.322232 31299 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:12.338112 25395 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:12.341516 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 133218us
I20260709 13:10:12.343254 25287 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:12.346349 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.320663 31168 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:12.345005 30919 raft_consensus.cc:697] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:12.349462 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:10:12.349944 30919 consensus_queue.cc:237] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.350910 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
W20260709 13:10:12.352421 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:12.357259 25239 raft_consensus.cc:2368] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15.
W20260709 13:10:12.357451 25192 auto_rebalancer.cc:629] Failed to schedule move for tablet 1b8652aecc6d4869bb4cad9e0749f3ba: Illegal state: Leader has not yet committed an operation in its own term
I20260709 13:10:12.358258 31284 raft_consensus.cc:2764] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election decision vote started in defunct term 18: won
I20260709 13:10:12.329802 30896 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:12.343396 30884 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:12.373265 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:12.375290 25358 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:12.333117 31273 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.380084 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:12.340291 31075 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.382174 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:12.382989 30305 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:12.383442 30305 raft_consensus.cc:697] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:12.383369 25358 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:12.386901 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:12.393747 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 238227us
I20260709 13:10:12.399334 25358 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:12.396948 30305 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.379108 31284 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:12.403045 25320 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:12.404428 25358 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:12.394138 31093 raft_consensus.cc:697] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:12.385488 31242 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:12.385629 31236 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:12.369796 31180 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.369778 31113 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.412309 25358 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:12.398530 31024 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:12.405754 30905 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:12.394320 31167 raft_consensus.cc:697] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:12.423269 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.407833 31284 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:12.412472 31095 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:12.429539 25245 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:12.412771 31296 tablet_replica.cc:333] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:10:12.409451 31068 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:12.428578 25246 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:12.406234 31147 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:12.427745 31293 log.cc:927] Time spent T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.118s	user 0.001s	sys 0.001s
I20260709 13:10:12.433722 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.433966 31095 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:12.429935 31154 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:12.430984 31284 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:12.432467 31024 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:12.430080 31242 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:12.412328 30622 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 13:10:12.425899 31290 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:12.415045 31177 consensus_queue.cc:237] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:12.432029 30905 raft_consensus.cc:697] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:12.413159 31308 consensus_queue.cc:1059] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.054s
W20260709 13:10:12.475119 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 131983us
I20260709 13:10:12.466461 25284 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:12.465466 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 134464us
I20260709 13:10:12.467746 31296 raft_consensus.cc:2243] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Raft consensus shutting down.
I20260709 13:10:12.476419 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.477169 31242 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:12.479108 31093 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.461710 31180 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:12.461377 31095 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:12.454200 31312 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:12.475060 31024 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:12.481573 31147 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:12.461540 31154 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:12.476646 31000 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.542557 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:12.484475 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:12.485379 31308 raft_consensus.cc:3055] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:10:12.571760 25319 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:10:12.490272 31292 log.cc:927] Time spent T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.154s	user 0.000s	sys 0.002s
I20260709 13:10:12.490414 31113 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:12.493602 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.496833 31168 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:12.573609 25237 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:12.497804 30896 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:12.498237 25245 raft_consensus.cc:1275] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:10:12.498744 31137 raft_consensus.cc:3055] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:12.501459 31290 raft_consensus.cc:697] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:12.502588 31229 raft_consensus.cc:3055] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:10:12.504068 31075 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:12.505208 31273 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:12.509608 25911 leader_election.cc:400] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15.
I20260709 13:10:12.597983 25911 leader_election.cc:403] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:10:12.504961 30856 consensus_queue.cc:260] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.512410 25246 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:10:12.514717 25285 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:12.515372 25239 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:10:12.515956 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 49608.6us
I20260709 13:10:12.483044 25359 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:12.525278 25313 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:12.529565 30864 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.530269 31024 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:12.530963 31312 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:12.531893 31259 consensus_queue.cc:1059] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:10:12.524569 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:12.523406 31321 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:12.537143 31242 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:12.537640 31280 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:12.572966 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:12.497614 31223 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:12.579078 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:12.631287 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:12.631500 25317 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:12.511888 30905 consensus_queue.cc:237] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.542757 31154 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:12.588842 30807 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:10:12.603143 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 120151us
I20260709 13:10:12.603674 30305 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:10:12.606104 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 96644.2us
I20260709 13:10:12.572010 31308 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:10:12.609478 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 197250us
I20260709 13:10:12.723935 25317 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:12.583599 31229 raft_consensus.cc:740] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:12.581473 31137 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:10:12.666101 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:12.650100 31290 consensus_queue.cc:237] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:12.542430 31167 consensus_queue.cc:237] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:12.638545 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 96462.4us
W20260709 13:10:12.632929 25909 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 1.808s (SENT) [suppressed 23 similar messages]
I20260709 13:10:12.642129 31284 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.615705 30622 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:12.639804 31296 raft_consensus.cc:2272] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Raft consensus is shut down!
I20260709 13:10:12.718197 25908 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:12.636935 31000 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:12.605733 30856 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:12.740774 25359 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:12.742015 30864 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:12.741931 31308 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.617887 31075 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:12.722576 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:12.748009 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:12.723116 30305 raft_consensus.cc:697] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:12.724476 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:12.613895 25319 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:12.713912 31270 raft_consensus.cc:697] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:12.714123 30884 raft_consensus.cc:697] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:10:12.745424 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 230777us
I20260709 13:10:12.724756 30807 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:12.796321 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:12.740883 25908 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:12.804361 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 63552us
I20260709 13:10:12.745822 31308 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:12.743171 31259 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:10:12.745633 31123 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:12.809010 25285 leader_election.cc:400] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15.
I20260709 13:10:12.730088 31068 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
W20260709 13:10:12.803062 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 78678.8us
I20260709 13:10:12.823781 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:12.811192 30305 consensus_queue.cc:237] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.812846 31300 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:12.815117 31242 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:12.804883 31166 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:12.815171 30884 consensus_queue.cc:237] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:12.815984 31180 raft_consensus.cc:3055] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:12.808233 31309 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:12.749168 31321 raft_consensus.cc:697] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:12.799638 31236 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:12.827787 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:12.828104 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:12.831035 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:12.829008 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:12.816535 31323 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260709 13:10:12.842886 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.843808 25246 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:12.842949 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.844314 25238 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:10:12.844635 25242 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 11.
I20260709 13:10:12.810237 31145 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:12.845947 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:12.846378 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:12.846776 25319 raft_consensus.cc:2393] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 19: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:12.846961 25244 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:12.849251 31309 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.850004 31095 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:12.852077 31095 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:12.852432 31309 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:12.852708 31095 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:12.839133 31180 raft_consensus.cc:740] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:12.853536 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.854175 25240 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:12.854096 31180 consensus_queue.cc:260] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.843132 31236 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:12.855576 31236 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:12.839414 31223 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:12.856666 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:12.856899 31223 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.859735 31223 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:12.836692 31300 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.843102 31166 raft_consensus.cc:697] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:12.863215 31300 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:12.863729 31296 ts_tablet_manager.cc:1916] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:10:12.863461 31166 consensus_queue.cc:237] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.848878 31321 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.868808 25319 raft_consensus.cc:1275] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:10:12.865628 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:12.813894 31312 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:12.805945 31000 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
W20260709 13:10:12.883388 31316 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 0.159s	user 0.000s	sys 0.000s
I20260709 13:10:12.809084 31239 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 13:10:12.885403 31151 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:12.885627 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:12.886894 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:12.887312 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:12.887485 25320 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:12.887876 25318 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
W20260709 13:10:12.889876 31322 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.190s	user 0.001s	sys 0.000s
I20260709 13:10:12.890339 31318 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:12.890906 31318 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:12.894373 30807 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:12.895200 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:12.895692 31284 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:12.895862 25317 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:12.876187 31145 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:12.897975 31315 log.cc:927] Time spent T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 13:10:12.898250 31180 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:12.898738 31145 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:12.899257 31145 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:12.899601 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:12.899725 31145 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.900502 25395 raft_consensus.cc:2410] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 16 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
W20260709 13:10:12.901305 31330 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 13:10:12.902642 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:12.903261 25390 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:10:12.822507 31280 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:12.905503 25237 raft_consensus.cc:2410] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 13:10:12.905385 25911 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:12.899266 25386 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:12.747275 30622 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.881160 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:12.905791 25244 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:12.909575 25245 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:12.909802 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:12.876410 31095 raft_consensus.cc:697] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:12.909369 31312 consensus_queue.cc:1059] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.004s
I20260709 13:10:12.909478 30898 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:12.913326 30898 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:12.913520 31312 raft_consensus.cc:3055] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:12.913734 30898 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:12.913970 31312 raft_consensus.cc:740] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:10:12.914206 31327 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 13:10:12.915836 31273 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:10:12.885027 31154 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:12.917444 31328 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.176s	user 0.003s	sys 0.000s
I20260709 13:10:12.918198 31154 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:12.904400 31270 consensus_queue.cc:237] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.920153 31229 consensus_queue.cc:260] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.820402 31137 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:12.922614 31299 consensus_meta.cc:287] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.574s	user 0.012s	sys 0.000s
I20260709 13:10:12.923592 31229 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:12.884996 31236 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:12.913555 31095 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.926249 31236 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:12.930495 30856 consensus_queue.cc:1059] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:12.931203 31137 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:12.931559 30856 raft_consensus.cc:3055] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:12.932343 30856 raft_consensus.cc:740] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:12.933405 30856 consensus_queue.cc:260] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:12.935907 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:12.938575 25313 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:12.937896 25318 raft_consensus.cc:1275] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:10:12.914980 31312 consensus_queue.cc:260] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:12.942776 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 193176us
I20260709 13:10:12.942777 25390 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:12.945303 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:12.946049 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 313140us
I20260709 13:10:12.950744 25319 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 13:10:12.974510 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:12.990065 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:12.991079 25245 raft_consensus.cc:2368] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14.
I20260709 13:10:12.993610 25285 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:12.994503 25285 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:12.995246 25285 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:12.996556 31313 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:12.997722 31273 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:12.998344 31317 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:10:13.003949 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:13.004227 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:13.004539 25395 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:13.004949 25240 raft_consensus.cc:2393] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 19: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:12.993242 25394 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:10:12.999975 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:13.005779 25321 raft_consensus.cc:2368] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12.
I20260709 13:10:13.006284 25237 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:10:13.007303 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:13.008103 25246 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:13.011497 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:13.013341 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:13.014074 25390 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:10:13.015045 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:13.015869 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:13.017823 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:13.019114 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:13.021478 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:13.022159 25239 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:12.938349 31075 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15.
I20260709 13:10:13.023610 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:12.941661 31280 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:13.028082 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:12.997042 31313 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:13.030711 25317 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:13.035441 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:13.035832 25395 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 19.
I20260709 13:10:12.950268 31236 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:13.036561 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:13.037665 25242 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:10:13.038326 25943 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:10:13.038924 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:13.040064 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:12.992590 30896 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:10:13.054721 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:13.058915 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:13.043377 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:13.059625 25387 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:13.058624 25945 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 13:10:13.003820 31317 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:13.062556 25941 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:10:13.063223 25319 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:13.069455 25387 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14. Ops: []
I20260709 13:10:13.005115 31273 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:13.029891 31299 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:13.075686 25387 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:13.060026 25394 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:10:13.066537 25391 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:13.063195 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:13.057442 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:13.076843 25939 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:10:13.077687 25394 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:10:13.023130 31000 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:13.085986 25317 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:12.998370 31326 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:13.091974 25389 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:13.076826 25938 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:13.064702 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:13.091712 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:13.098310 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:13.098819 25321 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
W20260709 13:10:13.097512 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 277855us
I20260709 13:10:13.102649 25209 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:13.089775 31242 raft_consensus.cc:697] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:13.075984 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:13.094664 25285 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:13.087610 25313 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:13.041531 30856 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:13.103489 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: true dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:13.104882 25395 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: [13.9-15.10]
I20260709 13:10:13.105865 25938 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:13.110643 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:13.112040 25395 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 15: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:13.075834 31312 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:13.075629 31273 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:13.072683 31180 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:13.114012 25387 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:10:13.073342 31317 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:10:13.119764 25391 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:13.120550 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:13.132215 25395 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 15: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:13.121719 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:13.107062 31310 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:13.121438 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:13.134192 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:13.129179 25387 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 13:10:13.121416 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:13.061729 30898 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:13.133288 25945 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:13.138324 25940 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:10:13.068856 31236 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:13.068873 31280 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:13.080724 30896 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:13.115038 31167 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:13.114382 31326 raft_consensus.cc:697] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:13.121429 31321 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:13.092283 31299 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:13.170467 31145 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:13.170809 31299 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:13.171408 25359 leader_election.cc:400] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14.
I20260709 13:10:13.171931 25359 leader_election.cc:403] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 13:10:13.148972 31167 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:13.172889 31167 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:13.173796 30305 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:13.176385 31321 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:13.100267 31000 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:13.180703 25392 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:13.174614 31068 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:10:13.179534 25389 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:10:13.166803 30622 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:13.191404 25941 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:13.143126 31223 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:13.163502 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 60978.6us
I20260709 13:10:13.214900 25940 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 13. (term mismatch)
I20260709 13:10:13.215337 25940 pending_rounds.cc:85] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 12
I20260709 13:10:13.215548 30898 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:13.217283 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:13.218633 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:13.215549 25940 pending_rounds.cc:107] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 15.13
I20260709 13:10:13.220695 25940 raft_consensus.cc:2889] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:13.221256 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:13.222118 25391 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:10:13.223431 25390 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:10:13.224217 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:13.224682 25390 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:13.126968 31329 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.035s
I20260709 13:10:13.234238 25908 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:13.228399 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:13.134672 31180 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:13.247447 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:13.241726 25212 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:13.252839 25388 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:13.253576 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:13.254992 25391 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:13.141158 31310 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:13.260360 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:13.263020 30305 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14.
I20260709 13:10:13.266634 25212 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:13.270093 25937 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:10:13.271209 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:13.272071 25945 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:10:13.273155 25209 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:13.274367 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:13.274618 25939 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:10:13.277247 25392 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:13.285392 25940 raft_consensus.cc:2393] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:13.249645 31299 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:13.293939 25242 raft_consensus.cc:3055] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:13.295819 25242 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:13.300549 25390 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:13.303022 25938 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:10:13.317261 25942 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:13.233939 31223 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:13.262472 31145 consensus_queue.cc:1059] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.043s
I20260709 13:10:13.296701 31335 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:13.304914 31151 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:13.301939 31334 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:10:13.306173 31299 raft_consensus.cc:697] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:13.373255 25387 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:13.304603 31113 consensus_queue.cc:1059] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.053s
I20260709 13:10:13.286345 31242 consensus_queue.cc:237] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:13.386723 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:13.387169 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:13.388715 25395 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 13:10:13.391290 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:13.392071 25239 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:13.394764 30905 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:13.395321 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:13.395382 30905 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:13.397213 31000 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:13.397969 30905 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:13.398237 31113 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:10:13.398561 31310 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:13.398998 31113 consensus_queue.cc:1059] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:10:13.405416 31319 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:10:13.406271 25358 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:13.407688 25358 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:13.415076 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:13.418465 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:13.426421 25386 raft_consensus.cc:2410] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 17 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
W20260709 13:10:13.409873 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 136826us
I20260709 13:10:13.427470 25209 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:13.430351 25358 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:13.433434 25358 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:13.434535 25284 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:13.435695 25284 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:13.433701 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:13.437395 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:13.437822 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:13.400004 31335 consensus_queue.cc:1059] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:10:13.441390 25358 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:13.448405 31239 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:13.448716 31236 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:13.445989 31229 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:10:13.472427 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:13.472750 31229 raft_consensus.cc:697] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:13.473294 31340 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:10:13.474045 31229 consensus_queue.cc:237] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:13.476076 31353 raft_consensus.cc:3055] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:13.476588 31353 raft_consensus.cc:740] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:10:13.454280 31347 log.cc:927] Time spent T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
W20260709 13:10:13.455109 31345 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.148s	user 0.000s	sys 0.000s
W20260709 13:10:13.456178 31348 log.cc:927] Time spent T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:10:13.471485 31239 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:13.315191 31326 consensus_queue.cc:237] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:13.448803 31068 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:13.481154 31319 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:13.481592 31068 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:13.482038 31068 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:13.481751 31319 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:13.483924 31319 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:13.457286 31137 consensus_meta.cc:287] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.526s	user 0.012s	sys 0.000s
I20260709 13:10:13.485183 25944 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 13. (term mismatch)
I20260709 13:10:13.485564 25944 pending_rounds.cc:85] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 12
I20260709 13:10:13.485787 25944 pending_rounds.cc:107] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 18.13
I20260709 13:10:13.486054 25944 raft_consensus.cc:2889] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:13.487290 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:13.487847 31361 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:13.487942 25239 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:10:13.460152 31167 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:13.488579 31361 raft_consensus.cc:697] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:13.489984 31338 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:13.490535 31338 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:13.492427 31321 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:13.459091 30896 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:13.461030 31272 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:13.493794 31272 raft_consensus.cc:697] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:13.494328 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:13.494756 31272 consensus_queue.cc:237] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:13.384086 31313 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:13.463563 31308 consensus_meta.cc:287] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.627s	user 0.006s	sys 0.002s
I20260709 13:10:13.464834 25942 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 13. (term mismatch)
I20260709 13:10:13.496762 25942 pending_rounds.cc:85] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 12
I20260709 13:10:13.497042 25942 pending_rounds.cc:107] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 17.13
I20260709 13:10:13.497303 25942 raft_consensus.cc:2889] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:13.445979 30905 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:13.440231 31000 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:13.460593 31180 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:13.470216 30305 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:13.504709 30305 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:13.505239 30305 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:10:13.471261 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 43931.9us
I20260709 13:10:13.477514 31353 consensus_queue.cc:260] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:13.466934 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:13.502414 31000 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:10:13.426296 31318 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:13.480995 31239 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:13.492892 31338 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:13.512722 25358 leader_election.cc:400] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12.
I20260709 13:10:13.532965 25358 leader_election.cc:403] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 13:10:13.472502 31280 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:13.527293 31344 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.219s	user 0.000s	sys 0.002s
I20260709 13:10:13.551550 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:13.528168 25942 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:10:13.537387 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:13.538774 25941 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:10:13.540400 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:13.490548 31167 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:13.557243 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:13.559893 25940 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 16: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:13.568539 25941 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:10:13.570338 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:13.571171 25945 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:13.521456 31151 raft_consensus.cc:697] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 19 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:13.531127 30896 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.036s
I20260709 13:10:13.584215 31145 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:10:13.517545 30905 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 13:10:13.567608 31236 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12.
I20260709 13:10:13.576287 31353 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:13.596043 30305 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:13.605597 25358 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:13.528200 31123 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.056s
I20260709 13:10:13.621372 25908 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:13.619678 31151 consensus_queue.cc:237] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:13.622958 31340 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:13.623397 31340 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:13.623761 31340 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:13.624280 31362 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:10:13.626163 31147 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:13.626607 31334 raft_consensus.cc:3055] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:13.627015 31334 raft_consensus.cc:740] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:13.627924 31334 consensus_queue.cc:260] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:13.629364 31317 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:13.629696 31137 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:13.630244 31137 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:13.630626 31137 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:13.631613 31317 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:13.642874 31296 ts_tablet_manager.cc:1929] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 12.9
I20260709 13:10:13.643517 31296 log.cc:1199] T eda233ace2a24b379fef320a5f610c7a P a2a77ecf2c0941618b5bc79d4c04db61: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-2-root/wals/eda233ace2a24b379fef320a5f610c7a
I20260709 13:10:13.643993 25245 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:10:13.644582 31075 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:13.645213 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:13.645388 31296 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44978 (request call id 0) took 1505 ms. Trace:
I20260709 13:10:13.645341 31075 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:13.647415 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:13.646119 31296 rpcz_store.cc:276] 0709 13:10:12.139833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:12.142746 (+  2913us) service_pool.cc:224] Handling call
0709 13:10:12.378388 (+235642us) ts_tablet_manager.cc:1151] Deleting tablet eda233ace2a24b379fef320a5f610c7a
0709 13:10:12.399663 (+ 21275us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 13:10:13.633529 (+1233866us) tablet_metadata.cc:665] Metadata flushed
0709 13:10:13.642734 (+  9205us) tablet_metadata.cc:665] Metadata flushed
0709 13:10:13.645214 (+  2480us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13275,"tablet-delete.queue_time_us":218865,"thread_start_us":184839,"threads_started":1}
I20260709 13:10:13.642944 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:13.649660 25131 catalog_manager.cc:5028] TS a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): tablet eda233ace2a24b379fef320a5f610c7a (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:10:13.650807 31361 consensus_queue.cc:237] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:13.626629 31147 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:13.646838 25240 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:10:13.656213 31147 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:13.656720 31336 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.200s	user 0.000s	sys 0.001s
I20260709 13:10:13.656747 31223 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:13.659538 31137 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:13.523717 31299 consensus_queue.cc:237] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:13.661700 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 56209.7us
I20260709 13:10:13.673514 25359 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:13.599752 31338 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:13.686331 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:13.687611 30305 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:13.692894 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:10:13.695184 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:13.696080 25941 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:13.703449 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:13.704298 25319 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:13.708077 25321 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:10:13.711866 25945 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:10:13.711918 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:13.715482 25321 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 15: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:13.715659 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:13.716742 25317 raft_consensus.cc:2410] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 17 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 13:10:13.700371 25242 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:13.700377 31273 consensus_meta.cc:287] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.564s	user 0.009s	sys 0.003s
I20260709 13:10:13.719398 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:13.603552 31167 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.095s
W20260709 13:10:13.721300 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:13.724061 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:13.726105 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:13.841009 31319 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:13.845608 25320 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:13.845971 31319 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:13.846796 31325 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.153s
I20260709 13:10:13.754042 25242 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:13.848291 31319 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:13.849411 31324 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.159s
I20260709 13:10:13.756289 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:13.760730 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:13.763773 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:13.853320 25315 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:10:13.761689 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:13.763968 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:13.765725 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:13.768342 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 417247us
W20260709 13:10:13.771215 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:13.710785 31075 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:13.778867 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:13.664950 31068 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:13.698966 31000 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.064s
I20260709 13:10:13.784277 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:13.785497 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:13.786828 25237 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
W20260709 13:10:13.765166 31328 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 13:10:13.709113 31284 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.062s
I20260709 13:10:13.699970 31338 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:13.799082 25313 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:13.709322 31280 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.069s
I20260709 13:10:13.764676 31145 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:13.843592 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:13.756160 31180 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:13.846387 31339 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:13.709676 31236 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:10:13.856702 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:13.853869 25321 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
W20260709 13:10:13.943288 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:13.862488 25242 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:10:13.863085 25319 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:13.865303 25312 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 13:10:13.756961 31273 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:13.861456 30856 consensus_meta.cc:287] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.746s	user 0.008s	sys 0.004s
I20260709 13:10:13.926651 31113 raft_consensus.cc:3055] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:10:13.873317 25238 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 15: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:13.799302 31363 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:13.925222 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:13.848877 31319 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:13.973066 25943 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:13.927995 25391 raft_consensus.cc:1240] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: []
I20260709 13:10:13.862321 31329 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:13.935258 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:13.936738 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 93266.7us
I20260709 13:10:13.941499 31229 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:13.868016 31310 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:13.873769 25314 raft_consensus.cc:2393] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 18: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:10:13.975160 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 403737us
I20260709 13:10:13.976578 25287 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:13.974752 25246 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:13.919507 31360 consensus_queue.cc:1059] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 13:10:13.946760 31236 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:13.956279 31113 raft_consensus.cc:740] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:13.960695 31339 raft_consensus.cc:697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:14.038246 25313 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:13.982645 31137 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:14.001667 31364 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 13:10:14.004060 31147 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:14.085928 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 164821us
I20260709 13:10:13.980777 31340 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:13.992396 31236 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:13.947813 31145 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:13.949025 31180 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:14.010243 31113 consensus_queue.cc:260] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:14.010854 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:14.011861 31075 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:14.014616 31367 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.250s	user 0.002s	sys 0.000s
W20260709 13:10:14.027770 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:14.144093 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:14.029613 31317 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:14.030983 30884 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:14.032096 31371 consensus_queue.cc:1059] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:10:14.032804 31290 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:14.163563 31290 consensus_queue.cc:1243] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer f646dfd532974dceadfe00005d2bbf59 log is divergent from this leader: its last log entry 14.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:10:14.163625 31371 raft_consensus.cc:3055] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:14.166642 31371 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:14.167752 31371 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:14.035295 25287 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:14.170495 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 135305us
I20260709 13:10:14.171802 31306 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:14.036065 31321 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:10:14.172498 31306 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:14.172971 31306 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:14.174100 31312 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:14.174903 31312 raft_consensus.cc:697] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:14.176044 31312 consensus_queue.cc:237] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:14.036922 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 49693.8us
W20260709 13:10:14.011960 31369 log.cc:927] Time spent T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
I20260709 13:10:13.980243 31319 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:14.113495 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:14.183332 31319 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:14.184142 25317 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 18: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:14.183449 31339 consensus_queue.cc:237] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:14.186031 25314 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:14.186556 25319 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:14.187381 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:14.110898 31154 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:14.188892 31154 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:14.189898 31273 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:14.189515 25287 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:14.128465 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:13.980671 31329 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:14.131280 25243 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:14.131359 31326 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:14.193943 25389 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:10:14.196060 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:14.139463 31180 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:14.141287 31365 raft_consensus.cc:3055] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:14.204344 31365 raft_consensus.cc:740] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:14.205677 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:14.205430 31365 consensus_queue.cc:260] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:14.141039 31376 consensus_queue.cc:1059] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:13.979079 31363 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:14.091120 31308 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.093s
I20260709 13:10:14.144104 31075 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:14.154070 31317 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:14.154918 30884 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:14.155977 25909 leader_election.cc:336] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 2.079s (SENT)
I20260709 13:10:14.188251 31145 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:14.197614 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:14.198920 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:14.202095 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:14.122216 31137 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:13.992499 31310 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:14.214932 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:14.142355 25212 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:14.221841 25359 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.012s	sys 0.027s
I20260709 13:10:14.210379 31154 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:14.209178 31319 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:14.264655 25316 raft_consensus.cc:2393] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 12: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:10:14.236771 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:14.264237 25319 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 15: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:14.265381 25909 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:14.209142 31095 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:14.277393 31371 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:14.216460 31308 consensus_queue.cc:1243] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer a2a77ecf2c0941618b5bc79d4c04db61 log is divergent from this leader: its last log entry 16.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
W20260709 13:10:14.269721 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 127468us
W20260709 13:10:14.270720 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:14.279106 31145 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:10:14.269081 25359 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.013s	sys 0.027s
W20260709 13:10:14.262409 25358 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.052s	sys 0.079s
W20260709 13:10:14.334718 25358 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.052s	sys 0.080s
W20260709 13:10:14.334611 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 626199us
I20260709 13:10:14.265723 30856 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.060s
I20260709 13:10:14.207449 31299 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:14.291906 25285 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.009s	sys 0.033s
I20260709 13:10:14.235375 31365 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:14.279656 25313 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 19.
W20260709 13:10:14.313372 25209 connection.cc:256] RPC call timeout handler was delayed by -2.0867260780000834s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16510000000000002s for a response
I20260709 13:10:14.268000 31068 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:14.332628 31229 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:14.335209 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 639791us
I20260709 13:10:14.225745 25318 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:14.206571 31326 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:14.340358 25312 raft_consensus.cc:2393] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 16: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:10:14.340387 25285 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.010s	sys 0.033s
I20260709 13:10:14.346040 25359 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:14.335440 31338 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:14.269594 31137 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:14.394657 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:14.356002 25319 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:10:14.361698 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:14.369367 25209 connection.cc:256] RPC call timeout handler was delayed by -2.0705260780000572s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16690000000000002s for a response
I20260709 13:10:14.330109 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:14.373744 25212 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:14.383431 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:14.324409 31319 raft_consensus.cc:697] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:10:14.393358 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 47378.9us
I20260709 13:10:14.393448 30905 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:10:14.394652 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 238700us
I20260709 13:10:14.400483 25313 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:14.408033 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:14.400851 25209 connection.cc:256] RPC call timeout handler was delayed by -1.8599260780000577s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19030000000000002s for a response
W20260709 13:10:14.405834 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:14.362200 31145 raft_consensus.cc:697] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:14.376497 31180 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:10:14.412748 25209 connection.cc:256] RPC call timeout handler was delayed by -1.8509260780000432s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19130000000000003s for a response
W20260709 13:10:14.416978 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:14.417101 25209 connection.cc:256] RPC call timeout handler was delayed by -1.5089260780000586s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22930000000000003s for a response
I20260709 13:10:14.335733 31095 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:14.355799 31325 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.017s	sys 0.008s
I20260709 13:10:14.428382 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:14.431032 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:14.432559 25389 raft_consensus.cc:2410] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 20 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:10:14.393623 31075 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:14.410557 31338 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:14.402438 31000 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:14.377071 31299 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:14.414867 31334 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:14.407213 30905 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:14.414541 31376 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:14.444665 25285 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:14.428318 31068 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:14.468820 25209 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:14.413417 31363 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:14.481415 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:14.420962 31389 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:14.418689 31147 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:14.386577 31229 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:14.438421 25313 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:10:14.394371 31390 consensus_queue.cc:1059] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.061s
W20260709 13:10:14.526636 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 82061.2us
W20260709 13:10:14.530726 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 47108.8us
I20260709 13:10:14.546916 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:14.509003 30905 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:14.535831 31147 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.019s
W20260709 13:10:14.570153 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 355322us
I20260709 13:10:14.542088 31376 raft_consensus.cc:697] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:14.485085 31280 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.069s
W20260709 13:10:14.581627 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:14.484813 31272 consensus_queue.cc:1059] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.068s
I20260709 13:10:14.546393 31273 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:14.480139 31338 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:14.486159 31000 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:14.603264 25319 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:10:14.605707 31229 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:14.606719 31376 consensus_queue.cc:237] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:14.609468 31326 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:14.609936 25321 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:10:14.611106 31137 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:14.611698 25943 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260709 13:10:14.613790 31317 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:14.631946 31389 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:14.635820 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:14.590986 31382 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.300s	user 0.000s	sys 0.002s
I20260709 13:10:14.518216 31384 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.109s
I20260709 13:10:14.532562 31068 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:14.668607 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 70347.5us
I20260709 13:10:14.545928 31329 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:14.685086 31389 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:14.594867 31151 raft_consensus.cc:2764] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election decision vote started in defunct term 14: won
W20260709 13:10:14.677249 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 269297us
I20260709 13:10:14.720774 25358 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:14.621795 31273 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:14.677219 31374 log.cc:927] Time spent T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.349s	user 0.000s	sys 0.001s
W20260709 13:10:14.688236 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 274230us
I20260709 13:10:14.698453 25319 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:10:14.578332 31310 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:14.594496 31180 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.091s
I20260709 13:10:14.567638 31319 consensus_queue.cc:237] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:14.701917 31395 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:14.757486 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:14.719605 31380 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.345s	user 0.001s	sys 0.001s
I20260709 13:10:14.699993 31383 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:14.769680 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:14.771632 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:14.770923 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:14.732314 31273 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:14.775192 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 306452us
W20260709 13:10:14.760432 31306 consensus_meta.cc:287] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.587s	user 0.012s	sys 0.002s
I20260709 13:10:14.782900 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:14.790220 30305 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:14.807888 25238 raft_consensus.cc:2410] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
W20260709 13:10:14.776854 31387 log.cc:927] Time spent T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.239s	user 0.000s	sys 0.001s
W20260709 13:10:14.775050 31381 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.370s	user 0.000s	sys 0.001s
I20260709 13:10:14.680956 31145 consensus_queue.cc:237] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:14.810459 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 381405us
W20260709 13:10:14.782611 31394 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 13:10:14.823374 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:14.824186 30305 consensus_queue.cc:1059] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 13:10:14.819365 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 54812.4us
W20260709 13:10:14.803260 31399 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 13:10:14.762060 31385 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.054s
I20260709 13:10:14.831511 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2564) took 1192 ms. Trace:
I20260709 13:10:14.829396 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:14.820188 31095 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:14.777797 31403 raft_consensus.cc:2764] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 13:10:14.782620 31339 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:14.779579 31284 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:14.832237 25239 rpcz_store.cc:276] 0709 13:10:13.638645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:13.640868 (+  2223us) service_pool.cc:224] Handling call
0709 13:10:13.641247 (+   379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:13.641638 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:13.641666 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:13.642167 (+   501us) log.cc:844] Serialized 26 byte log entry
0709 13:10:14.115970 (+473803us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:10:14.116028 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:14.116125 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:14.831200 (+715075us) raft_consensus.cc:1685] finished
0709 13:10:14.831230 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:14.831425 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13884,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":86,"thread_start_us":473362,"threads_started":1,"wal-append.queue_time_us":648411}
I20260709 13:10:14.845835 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:14.850876 25312 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:14.823416 31299 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:14.860411 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:14.890303 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 67016.5us
I20260709 13:10:14.891161 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:14.894022 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:14.894551 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:14.894917 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:14.848416 31391 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.280s	user 0.001s	sys 0.000s
I20260709 13:10:14.774747 31395 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:14.844205 31154 raft_consensus.cc:3055] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:10:14.786531 30905 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:14.910184 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:14.917307 25312 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:14.917212 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:14.918078 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:14.918653 25242 raft_consensus.cc:2410] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 13:10:14.827018 31384 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.063s
I20260709 13:10:14.914551 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:14.920357 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:14.920969 25943 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:14.920650 31390 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:14.924368 31390 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:14.924794 25240 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:10:14.853646 31339 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:14.926610 31390 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:14.927008 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:14.927556 31339 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:14.933537 31151 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:14.937393 31413 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:14.940186 31239 consensus_meta.cc:287] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.375s	user 0.009s	sys 0.004s
W20260709 13:10:14.941789 31396 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
I20260709 13:10:14.943158 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2495) took 1184 ms. Trace:
W20260709 13:10:14.943502 31400 log.cc:927] Time spent T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 13:10:14.943557 25244 rpcz_store.cc:276] 0709 13:10:13.759171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:13.759294 (+   123us) service_pool.cc:224] Handling call
0709 13:10:13.764065 (+  4771us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:13.764472 (+   407us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:13.764506 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:13.796166 (+ 31660us) log.cc:844] Serialized 26 byte log entry
0709 13:10:14.294537 (+498371us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:10:14.294595 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:14.294691 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:14.942775 (+648084us) raft_consensus.cc:1685] finished
0709 13:10:14.942815 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:14.943085 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3744,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":60437,"spinlock_wait_cycles":30976,"thread_start_us":372233,"threads_started":1,"wal-append.queue_time_us":621115}
I20260709 13:10:14.946385 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:14.948051 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:14.949433 31000 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:14.949640 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2674) took 1106 ms. Trace:
I20260709 13:10:14.950016 25320 rpcz_store.cc:276] 0709 13:10:13.843147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:13.846756 (+  3609us) service_pool.cc:224] Handling call
0709 13:10:13.847238 (+   482us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:14.272572 (+425334us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:14.272620 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:14.273239 (+   619us) log.cc:844] Serialized 26 byte log entry
0709 13:10:14.488177 (+214938us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:10:14.488241 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:14.488342 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:14.949419 (+461077us) raft_consensus.cc:1685] finished
0709 13:10:14.949443 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:14.949581 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":709,"raft.queue_time_us":8432,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":7412,"thread_start_us":622409,"threads_started":2,"wal-append.queue_time_us":324539}
I20260709 13:10:14.954689 31395 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:14.920868 31405 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:14.955746 31405 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:14.956295 31371 consensus_meta.cc:287] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.621s	user 0.009s	sys 0.000s
I20260709 13:10:14.957212 25239 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 13. (term mismatch)
I20260709 13:10:14.957720 25239 pending_rounds.cc:85] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 12
I20260709 13:10:14.957921 25239 pending_rounds.cc:107] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 15.13
I20260709 13:10:14.958139 25239 raft_consensus.cc:2889] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:14.958238 31405 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:14.959287 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:14.809847 31306 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:14.921255 31365 consensus_meta.cc:287] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.544s	user 0.007s	sys 0.004s
I20260709 13:10:14.852434 31284 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:14.962795 31338 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:14.965174 30905 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:14.961042 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:14.965466 25316 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 13: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:14.941697 31068 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:14.971823 25240 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
W20260709 13:10:14.968525 25192 auto_rebalancer.cc:629] Failed to schedule move for tablet e3025c38ad5a4bdfa61cfca91ba3c409: Illegal state: Replica 46de6919f85341c6a9b574dff112e35a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 18 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:14.916793 31154 raft_consensus.cc:740] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:14.977316 25939 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 13:10:14.973094 25244 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:10:14.978905 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 198084us
I20260709 13:10:14.988399 25358 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:14.980370 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:15.002022 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:14.989549 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:15.003043 25387 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
W20260709 13:10:14.948081 31353 consensus_meta.cc:287] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.327s	user 0.011s	sys 0.000s
I20260709 13:10:14.940663 31239 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:14.968268 31389 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:14.998335 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:14.998600 25943 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:14.999861 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:15.032666 25242 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:15.001149 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:15.033816 25393 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:15.004012 25389 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:10:15.004128 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:15.005152 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:15.005555 25387 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:10:15.002902 25391 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:15.001905 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:10:14.994940 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:14.920046 31383 raft_consensus.cc:697] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:14.967610 31284 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:15.009428 25358 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:15.011432 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:15.089798 25939 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:15.012044 25392 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:10:14.975183 31306 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:14.912060 31223 consensus_queue.cc:1059] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.082s
W20260709 13:10:15.102092 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 93015.5us
I20260709 13:10:15.094064 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:15.095780 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:14.990641 31365 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:15.062000 25237 raft_consensus.cc:2410] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 17 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 13:10:15.102974 25358 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:15.103173 25395 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:15.096798 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:15.103773 25287 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:15.115621 25390 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:15.117216 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:15.117884 25244 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:10:15.118490 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:15.122716 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:15.128242 25389 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:10:15.128994 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:15.130355 25358 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:15.101904 31145 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:15.142849 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:15.147321 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 43619.9us
I20260709 13:10:15.202095 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:15.064488 31151 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:15.152308 25390 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:15.137190 25393 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:15.061273 30905 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:15.124303 31408 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:15.178241 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.191032 25209 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:15.115664 31284 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:15.008769 31390 consensus_queue.cc:1059] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.124s
W20260709 13:10:15.205221 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 83571.5us
I20260709 13:10:15.226785 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.209048 25939 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:10:15.212792 25392 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 13:10:15.216981 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:15.176673 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:10:15.203469 31236 consensus_meta.cc:287] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.076s	user 0.013s	sys 0.000s
I20260709 13:10:15.231679 25386 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:15.231678 25943 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:15.197263 31384 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:15.234835 25389 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:15.235013 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:15.246682 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:15.238864 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:15.242379 25244 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
W20260709 13:10:15.242740 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 51807.2us
I20260709 13:10:15.196792 31000 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:15.197901 31422 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:15.250723 25386 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:15.215785 31068 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:15.218946 31385 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:15.252200 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:15.253669 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.252700 25209 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:15.125442 31365 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:15.260155 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:15.262126 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.236354 31223 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:15.266199 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:15.268510 25943 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:15.268667 25939 raft_consensus.cc:2368] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17.
I20260709 13:10:15.250308 31384 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:15.270994 25209 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:15.254976 31151 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:10:15.273324 31395 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:15.274914 25246 raft_consensus.cc:2410] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 13:10:15.275882 31425 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:15.276037 31422 raft_consensus.cc:697] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:10:15.276389 31412 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.162s	user 0.000s	sys 0.001s
I20260709 13:10:15.257041 30905 raft_consensus.cc:697] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:15.131083 31154 consensus_queue.cc:260] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:15.266887 31334 consensus_meta.cc:287] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.757s	user 0.012s	sys 0.000s
I20260709 13:10:15.285382 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:15.292173 25389 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:15.274601 31000 raft_consensus.cc:697] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:15.294591 31419 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 13:10:15.295369 31317 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:15.295819 31365 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:15.296455 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:15.389906 25939 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 17: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:15.298297 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.240670 31408 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:15.390798 25242 raft_consensus.cc:2393] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 16: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:15.391944 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:15.216667 31145 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:15.392668 31408 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:15.392726 25242 raft_consensus.cc:2368] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 18. Current term is 19.
I20260709 13:10:15.395447 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.396207 25242 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:15.395972 31306 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:15.389045 31409 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.017s
I20260709 13:10:15.269117 31383 consensus_queue.cc:237] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:15.203992 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:10:15.387295 31154 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:10:15.388170 31242 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:15.372244 31414 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260709 13:10:15.369896 31353 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:15.353475 31425 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:15.374691 31423 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 13:10:15.374809 30905 consensus_queue.cc:237] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:15.377521 31415 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.162s	user 0.002s	sys 0.000s
I20260709 13:10:15.378646 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:15.282877 31384 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:15.297256 31389 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:15.299185 31401 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:15.369403 31430 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 13:10:15.386770 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.383306 31095 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:15.424405 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:15.421990 25245 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
W20260709 13:10:15.393106 31402 consensus_queue.cc:1175] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46de6919f85341c6a9b574dff112e35a" responder_term: 11 status { last_received { term: 11 index: 9 } last_committed_idx: 9 last_received_current_leader { term: 11 index: 9 } }
I20260709 13:10:15.393222 31317 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:15.266054 31385 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:15.431751 25939 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 13:10:15.433503 31317 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:15.433552 31385 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:15.250605 31236 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:15.435913 31236 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:15.394526 31145 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:15.267889 31223 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:15.407506 25937 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:15.437038 31223 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:15.410697 31068 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:15.437784 31068 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:15.440053 31068 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:15.409550 31377 raft_consensus.cc:3055] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:15.442584 31377 raft_consensus.cc:740] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:10:15.443810 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 160922us
I20260709 13:10:15.443539 31377 consensus_queue.cc:260] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:15.402458 31363 raft_consensus.cc:3055] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:15.445297 31363 raft_consensus.cc:740] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:10:15.446161 31393 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.813s	user 0.000s	sys 0.008s
I20260709 13:10:15.446534 31360 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:15.446256 31363 consensus_queue.cc:260] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:15.447574 31363 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:15.447717 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1112) took 1426 ms. Trace:
I20260709 13:10:15.449162 31405 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:10:15.427105 31376 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:15.450495 31376 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:15.427220 31361 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:10:15.453611 25317 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2669) took 1731 ms (client timeout 2248 ms). Trace:
W20260709 13:10:15.454226 25317 rpcz_store.cc:269] 0709 13:10:13.722714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:13.722942 (+   228us) service_pool.cc:224] Handling call
0709 13:10:15.453417 (+1730475us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b54000b22d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:15.453548 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":756371200}
W20260709 13:10:15.410712 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:15.425347 25239 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:15.414330 25944 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
I20260709 13:10:15.397091 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.430303 25945 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1111) took 1409 ms. Trace:
I20260709 13:10:15.457163 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.407960 31242 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:15.431187 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:15.401072 31373 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 1.073s	user 0.002s	sys 0.013s
I20260709 13:10:15.441066 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.412155 31422 consensus_queue.cc:237] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:15.411754 31284 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:15.442004 25394 raft_consensus.cc:2368] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17.
I20260709 13:10:15.447211 31360 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:15.448274 25941 rpcz_store.cc:276] 0709 13:10:14.021493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:14.022756 (+  1263us) service_pool.cc:224] Handling call
0709 13:10:14.023102 (+   346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:14.023551 (+   449us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:14.023591 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:14.024169 (+   578us) log.cc:844] Serialized 26 byte log entry
0709 13:10:14.464187 (+440018us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:10:14.464251 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:14.464356 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:15.445393 (+981037us) spinlock_profiling.cc:243] Waited 3.72 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:15.447422 (+  2029us) raft_consensus.cc:1685] finished
0709 13:10:15.447445 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:15.447650 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1276,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":251,"spinlock_wait_cycles":8565632,"thread_start_us":439536,"threads_started":1,"wal-append.queue_time_us":607928}
I20260709 13:10:15.423231 25359 leader_election.cc:400] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 18. Current term is 19.
I20260709 13:10:15.458541 25391 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
W20260709 13:10:15.475600 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:15.420225 31378 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 1.091s	user 0.005s	sys 0.007s
I20260709 13:10:15.429994 31095 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:15.459842 25242 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:15.398679 25389 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:15.462591 25245 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:15.440912 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 1037) took 1111 ms. Trace:
W20260709 13:10:15.463591 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:15.538524 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:15.411041 31425 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:15.402020 31321 raft_consensus.cc:3055] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:15.529855 25359 leader_election.cc:403] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 13:10:15.555652 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:15.563591 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 140559us
W20260709 13:10:15.477424 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:15.534984 25940 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: [15.11-16.12]
I20260709 13:10:15.400691 31239 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:15.537293 25316 rpcz_store.cc:276] 0709 13:10:14.329701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:14.329826 (+   125us) service_pool.cc:224] Handling call
0709 13:10:15.440715 (+1110889us) spinlock_profiling.cc:243] Waited 565 ms on lock 0x7b5400274f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:15.440852 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1299566464}
I20260709 13:10:15.529412 31362 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:15.457418 25945 rpcz_store.cc:276] 0709 13:10:14.021027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:14.022023 (+   996us) service_pool.cc:224] Handling call
0709 13:10:14.022486 (+   463us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:14.137774 (+115288us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:10:14.137808 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:14.138281 (+   473us) log.cc:844] Serialized 26 byte log entry
0709 13:10:14.439848 (+301567us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:14.439916 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:15.429897 (+989981us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b540062af54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:15.429949 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:15.429992 (+    43us) raft_consensus.cc:1685] finished
0709 13:10:15.430006 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:15.430221 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2517,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"spinlock_wait_cycles":264047360,"thread_start_us":292626,"threads_started":1,"wal-append.queue_time_us":494425}
I20260709 13:10:15.428270 31319 raft_consensus.cc:3055] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:15.528115 31405 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:15.544154 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:15.416523 31389 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:15.571219 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:15.579895 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:15.554334 31321 raft_consensus.cc:740] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:15.530395 31376 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:15.464349 25285 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:15.577071 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:15.416777 31401 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:15.444891 31377 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:15.388878 31000 consensus_queue.cc:237] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:15.585376 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 121065us
I20260709 13:10:15.446295 31223 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:15.578120 31319 raft_consensus.cc:740] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:15.566296 31236 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 18. Current term is 19.
I20260709 13:10:15.595834 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.573169 30884 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:15.409054 31353 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:15.590098 25285 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:15.537787 31284 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:15.603457 25246 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 17: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:15.602020 25389 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:10:15.610954 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.612859 25246 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:15.614142 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.614988 25246 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:15.590340 31425 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:15.618407 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2563) took 2112 ms. Trace:
I20260709 13:10:15.616242 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.619441 25241 rpcz_store.cc:276] 0709 13:10:13.497992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:13.499485 (+  1493us) service_pool.cc:224] Handling call
0709 13:10:13.524701 (+ 25216us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:13.525162 (+   461us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:13.525203 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:13.532045 (+  6842us) log.cc:844] Serialized 26 byte log entry
0709 13:10:13.983814 (+451769us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:13.983853 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:13.983926 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:15.416685 (+1432759us) spinlock_profiling.cc:243] Waited 2.24 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:15.603422 (+186737us) raft_consensus.cc:1685] finished
0709 13:10:15.603455 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:15.610796 (+  7341us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":56447,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":2518,"spinlock_wait_cycles":5150464,"thread_start_us":395636,"threads_started":1,"wal-append.queue_time_us":646280}
I20260709 13:10:15.617882 25245 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
W20260709 13:10:15.610492 31388 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 1.233s	user 0.001s	sys 0.005s
I20260709 13:10:15.582826 31362 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:15.623893 25246 raft_consensus.cc:2368] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15.
I20260709 13:10:15.611318 31317 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:15.593858 31272 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.027s
I20260709 13:10:15.649025 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.587622 31405 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:15.649757 25246 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:15.635852 31317 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:15.623909 25285 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:15.626184 31425 raft_consensus.cc:697] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:15.652747 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.665290 25246 raft_consensus.cc:2410] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 17 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
W20260709 13:10:15.652714 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 420433us
I20260709 13:10:15.667068 25287 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:15.664228 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 463862us
I20260709 13:10:15.653172 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.670620 25241 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:15.672290 25388 raft_consensus.cc:2410] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 13:10:15.676506 25287 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:15.611052 31321 consensus_queue.cc:260] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:15.679793 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 56020.3us
W20260709 13:10:15.666674 31273 consensus_meta.cc:287] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.857s	user 0.009s	sys 0.003s
W20260709 13:10:15.657094 31413 consensus_meta.cc:287] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.719s	user 0.012s	sys 0.000s
I20260709 13:10:15.682581 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.685441 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:15.684306 25287 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:15.653012 31317 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:15.683079 25285 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:15.686072 25391 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:10:15.686141 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.685997 25241 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:15.688978 25246 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:15.654501 31389 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:15.690115 25392 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:15.691015 25911 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:15.631359 31319 consensus_queue.cc:260] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:15.697042 31319 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:15.697348 31425 consensus_queue.cc:237] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.8, Last appended by leader: 8, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:15.699481 31405 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:15.699882 31353 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:15.701002 25321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2770) took 1505 ms (client timeout 1588 ms). Trace:
W20260709 13:10:15.701478 25321 rpcz_store.cc:269] 0709 13:10:14.195607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:14.195727 (+   120us) service_pool.cc:224] Handling call
0709 13:10:15.700801 (+1505074us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b5400115654. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:15.700932 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":506056448}
I20260709 13:10:15.702888 31360 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:15.680280 31438 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:15.713626 30856 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:15.714478 31401 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:15.690174 31413 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:15.715430 25244 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:15.685297 31386 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 13:10:15.680162 31419 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:15.718148 31419 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:15.718518 31419 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:15.691659 31273 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:15.713647 31438 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:15.720988 31317 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:15.721660 31438 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:15.723234 31317 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:15.723481 31273 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:15.680101 31435 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:15.710606 31436 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:15.681797 31151 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:10:15.712704 31384 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:15.714236 30856 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:15.710173 31362 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:15.726684 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:15.693164 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.727296 25244 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:15.727679 25387 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 13:10:15.750881 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:15.728302 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 44096.5us
I20260709 13:10:15.709686 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.751739 25394 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:15.730619 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:15.735391 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:15.736250 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 492669us
W20260709 13:10:15.741835 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:15.691993 31321 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:15.745054 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:15.749181 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1114) took 1722 ms. Trace:
I20260709 13:10:15.749568 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:15.754323 25938 rpcz_store.cc:276] 0709 13:10:14.027143 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:14.030167 (+  3024us) service_pool.cc:224] Handling call
0709 13:10:14.030640 (+   473us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:10:14.272676 (+242036us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:14.272712 (+    36us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:10:14.273423 (+   711us) log.cc:844] Serialized 78 byte log entry
0709 13:10:14.599049 (+325626us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:14.599118 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:15.748662 (+1149544us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b5400690fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:15.748725 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:15.748785 (+    60us) raft_consensus.cc:1685] finished
0709 13:10:15.748802 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:15.749098 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":966,"raft.queue_time_us":59552,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":6140,"spinlock_wait_cycles":543392000,"thread_start_us":254211,"threads_started":1,"wal-append.queue_time_us":454914}
I20260709 13:10:15.755242 25390 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:15.755108 25287 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:15.724570 31075 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:15.757649 25386 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:15.757195 31075 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:15.760035 31075 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:15.740621 31389 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:15.761430 31389 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:15.761809 31389 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:15.765358 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:15.766821 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:15.740324 31436 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:15.733686 31095 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:15.769958 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:15.740486 31409 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:15.792160 30856 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:15.744131 31318 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:15.793444 31318 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:15.794821 31386 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:15.795477 31386 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:15.795683 31318 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:15.796036 31386 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:15.802696 31384 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:15.803244 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:15.803877 25238 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:15.765576 31401 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:15.804600 31401 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:15.806874 31401 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:15.774232 31436 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:15.810863 25192 auto_rebalancer.cc:629] Failed to schedule move for tablet e5cbaabf54304de089abde51df6dc2be: Illegal state: Leader has not yet committed an operation in its own term
I20260709 13:10:15.811115 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.775132 25387 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 11.
I20260709 13:10:15.811125 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.815063 25911 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:15.816082 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:15.816686 25392 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:15.818051 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:15.818305 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:15.819259 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:15.819844 25313 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 13.
W20260709 13:10:15.820124 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 47927.6us
I20260709 13:10:15.776862 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:15.778879 25389 raft_consensus.cc:1275] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:10:15.820660 25238 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:15.779249 25909 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:15.782469 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2676) took 1399 ms. Trace:
I20260709 13:10:15.788131 31284 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:15.823946 25316 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:15.821981 25318 rpcz_store.cc:276] 0709 13:10:14.383282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:14.402656 (+ 19374us) service_pool.cc:224] Handling call
0709 13:10:14.790322 (+387666us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:14.790803 (+   481us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:14.790842 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:14.791623 (+   781us) log.cc:844] Serialized 52 byte log entry
0709 13:10:15.001626 (+210003us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:15.250392 (+248766us) log.cc:844] Serialized 14 byte log entry
0709 13:10:15.618559 (+368167us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:15.618635 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:15.618681 (+    46us) raft_consensus.cc:1685] finished
0709 13:10:15.618697 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:15.782238 (+163541us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b54000f8a50. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:15.782394 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6677,"prepare.queue_time_us":371636,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":135508,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":7892,"spinlock_wait_cycles":890520704,"thread_start_us":713946,"threads_started":3,"wal-append.queue_time_us":452608}
I20260709 13:10:15.824115 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:15.825475 25246 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:15.825850 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 46670.7us
W20260709 13:10:15.827087 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 50316.3us
I20260709 13:10:15.830027 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:15.836561 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 50956.3us
I20260709 13:10:15.837838 25358 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:15.838817 31095 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:10:15.839550 31095 raft_consensus.cc:697] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:15.840205 25313 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 13:10:15.840683 30856 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:10:15.840767 31447 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:15.841171 30856 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:15.841625 30856 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:10:15.841753 31447 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:15.842612 31113 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:15.843187 25909 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:15.844014 31447 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:10:15.845019 31075 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:15.845325 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:15.845476 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:15.846082 31447 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:15.846580 25320 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:15.847918 25315 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:15.847966 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:15.846278 31095 consensus_queue.cc:237] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:15.851136 31447 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:15.854159 31308 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:15.854714 31308 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:15.856722 31308 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:15.857450 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:15.858088 25938 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:15.858450 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:15.859030 25945 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:10:15.859807 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:15.860195 25242 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:10:15.860400 25317 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:15.861186 31319 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:10:15.861567 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:15.862414 25944 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 13: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:15.859844 25941 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:10:15.856745 25936 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:10:15.852797 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:15.875613 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:15.884869 25285 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:15.887818 31448 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:15.869560 25212 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:15.886176 25936 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:15.891353 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:15.875897 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:15.897373 25314 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:15.899586 25908 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:15.901615 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:15.902321 25938 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:15.917945 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:15.919108 25319 raft_consensus.cc:2410] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 13:10:15.940917 31436 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:15.949977 25321 raft_consensus.cc:2368] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17.
I20260709 13:10:15.868412 31363 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:10:15.885221 31437 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
I20260709 13:10:15.970223 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:15.970141 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:15.970920 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 69676.6us
I20260709 13:10:15.971781 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:15.975294 25937 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:10:15.985271 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:15.845489 31075 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:15.885556 31371 consensus_queue.cc:1059] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.118s
I20260709 13:10:15.961109 31401 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:16.009598 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:16.020046 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:16.021471 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 46532.7us
I20260709 13:10:15.966930 31448 raft_consensus.cc:697] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:15.967846 31363 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:16.023103 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:15.858093 31438 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:16.026648 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:16.056900 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:10:16.058388 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:16.058314 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:16.059474 25938 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:16.028342 25312 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:16.030392 25941 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:10:15.997265 31075 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:15.966148 31151 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.055s
I20260709 13:10:16.038486 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:16.081311 25945 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:15.961802 31284 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.049s
I20260709 13:10:16.029812 31095 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.011s
I20260709 13:10:16.041476 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:16.042523 31436 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:15.882407 31000 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.111s
I20260709 13:10:16.055452 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:15.966219 25237 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:10:16.098443 25392 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:10:16.067021 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:16.081091 25284 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:16.084097 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:10:16.095414 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:16.095844 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 196327us
I20260709 13:10:16.101548 25908 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:16.055986 31363 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:16.119124 31290 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:16.113751 25284 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:16.127549 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 43595.3us
I20260709 13:10:16.128752 25209 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:16.129170 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:16.129925 25237 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:16.131946 25908 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:16.086259 31377 raft_consensus.cc:3055] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Stepping down as leader of term 15
W20260709 13:10:16.112825 31455 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
W20260709 13:10:16.137225 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:16.140798 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:16.173653 31449 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:16.156838 25209 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:16.132126 31290 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:16.160291 31436 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:10:16.168154 31371 raft_consensus.cc:3055] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:10:16.172739 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2507) took 1256 ms. Trace:
I20260709 13:10:16.137377 31450 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:16.136938 31353 raft_consensus.cc:2764] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election decision vote started in defunct term 13: won
I20260709 13:10:16.088554 31419 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:16.178530 25284 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:16.178535 25943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 1194) took 1266 ms. Trace:
I20260709 13:10:16.142200 31395 raft_consensus.cc:2764] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 13:10:16.137935 31377 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:10:16.194010 25909 leader_election.cc:336] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 2.120s (SENT)
I20260709 13:10:16.132014 31318 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.047s
I20260709 13:10:16.156471 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:16.213457 25389 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:16.215368 31458 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:16.220063 30856 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:16.225210 31363 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:16.157194 31413 consensus_queue.cc:1059] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260709 13:10:16.325510 31290 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:16.193607 31409 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 13:10:16.212538 25938 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:16.263942 31461 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:16.329634 31371 raft_consensus.cc:740] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:16.180766 31448 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:16.273377 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 2188) took 1270 ms. Trace:
I20260709 13:10:16.280874 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:16.281628 25392 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:16.281889 25395 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:16.282749 25394 raft_consensus.cc:1217] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Deduplicated request from leader. Original: 13.10->[14.11-19.13]   Dedup: 14.11->[17.12-19.13]
I20260709 13:10:16.280939 31000 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:16.437523 31000 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:16.289626 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:16.441431 25245 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:16.290277 31438 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:16.291364 31389 consensus_meta.cc:287] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.529s	user 0.003s	sys 0.008s
I20260709 13:10:16.291447 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:16.291615 31223 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:16.292486 31340 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:16.457688 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2514) took 1178 ms. Trace:
I20260709 13:10:16.310213 25246 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:16.458429 30856 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:16.459229 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:16.460500 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:16.314002 25212 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:16.461575 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 147653us
I20260709 13:10:16.465523 25241 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: [15.11-16.12]
I20260709 13:10:16.320055 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:16.467391 25390 raft_consensus.cc:2368] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:10:16.468068 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:16.469376 31413 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:16.469555 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:16.470335 31353 consensus_queue.cc:1059] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:10:16.329952 25240 rpcz_store.cc:276] 0709 13:10:14.915997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:14.926099 (+ 10102us) service_pool.cc:224] Handling call
0709 13:10:14.926448 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:14.974721 (+ 48273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:16.172380 (+1197659us) spinlock_profiling.cc:243] Waited 45 ms on lock 0x7b54003aa7d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:16.172659 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":103551872}
I20260709 13:10:16.332548 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:16.317273 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 160495us
W20260709 13:10:16.342797 25192 auto_rebalancer.cc:629] Failed to schedule move for tablet 58ec1834a96f447f8fe74448e8480342: Illegal state: Replica a2a77ecf2c0941618b5bc79d4c04db61 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 12 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:16.313740 31390 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:16.292819 31386 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:16.268600 31242 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:16.318511 31113 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 13:10:16.492901 31113 consensus_queue.cc:1048] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20260709 13:10:16.316294 31321 consensus_meta.cc:287] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.547s	user 0.009s	sys 0.001s
I20260709 13:10:16.495824 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 1067) took 1563 ms. Trace:
W20260709 13:10:16.364660 31452 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.184s	user 0.001s	sys 0.000s
W20260709 13:10:16.366150 31457 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
I20260709 13:10:16.366786 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:16.371899 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 193501us
W20260709 13:10:16.511193 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:16.392819 25237 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:16.371414 25943 rpcz_store.cc:276] 0709 13:10:14.911771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:15.371172 (+459401us) service_pool.cc:224] Handling call
0709 13:10:15.371563 (+   391us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:15.371847 (+   284us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:15.371873 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:15.372608 (+   735us) log.cc:844] Serialized 26 byte log entry
0709 13:10:15.628767 (+256159us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:10:15.655729 (+ 26962us) log.cc:844] Serialized 14 byte log entry
0709 13:10:15.749380 (+ 93651us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:15.749473 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:15.768599 (+ 19126us) spinlock_profiling.cc:243] Waited 3.24 ms on lock 0x7b180036a1e0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:10:16.161725 (+393126us) raft_consensus.cc:1685] finished
0709 13:10:16.178242 (+ 16517us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:16.178450 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":28349,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":5545,"raft.queue_time_us":38630,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":17524,"spinlock_wait_cycles":7463296,"thread_start_us":349121,"threads_started":2,"wal-append.queue_time_us":308945}
I20260709 13:10:16.413393 25937 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
W20260709 13:10:16.415084 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 221113us
W20260709 13:10:16.514094 25909 leader_election.cc:336] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 2.126s (SENT)
I20260709 13:10:16.514627 25909 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:16.416064 25318 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:16.419950 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:16.516247 25319 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:16.421348 31458 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:16.420874 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:16.423760 25940 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: []
I20260709 13:10:16.523859 25389 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:16.185889 31272 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.054s
I20260709 13:10:16.329977 31151 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:16.268543 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:16.431980 25316 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:16.333077 31284 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:16.435192 25394 pending_rounds.cc:85] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 11
I20260709 13:10:16.411442 31436 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:16.434157 25388 rpcz_store.cc:276] 0709 13:10:15.002791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:15.011404 (+  8613us) service_pool.cc:224] Handling call
0709 13:10:16.273110 (+1261706us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x7b5400120cd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:16.273306 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1250645248}
I20260709 13:10:16.327237 31362 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.103s
I20260709 13:10:16.442368 31461 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:16.544124 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:16.458096 31371 consensus_queue.cc:260] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:16.469892 31413 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:16.470489 25313 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:16.339973 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:16.461091 25246 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:16.338296 31459 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:16.345343 31450 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:16.458158 25244 rpcz_store.cc:276] 0709 13:10:15.279349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:15.892739 (+613390us) service_pool.cc:224] Handling call
0709 13:10:15.893175 (+   436us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:15.961306 (+ 68131us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:15.961337 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:15.972988 (+ 11651us) log.cc:844] Serialized 26 byte log entry
0709 13:10:16.233836 (+260848us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:16.233887 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:16.233966 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:16.457471 (+223505us) raft_consensus.cc:1685] finished
0709 13:10:16.457494 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:16.457628 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82232,"raft.queue_time_us":68291,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":22765,"thread_start_us":197005,"threads_started":1,"wal-append.queue_time_us":435377}
I20260709 13:10:16.474104 31389 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:16.484179 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:16.485376 31401 consensus_queue.cc:1059] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
I20260709 13:10:16.488241 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:16.488718 31390 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:16.537262 31409 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:16.518129 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:16.552940 31413 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:16.496417 25242 rpcz_store.cc:276] 0709 13:10:14.932030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:15.812292 (+880262us) service_pool.cc:224] Handling call
0709 13:10:15.853614 (+ 41322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:15.860548 (+  6934us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:16.495635 (+635087us) spinlock_profiling.cc:243] Waited 40.9 ms on lock 0x7b54002b2350. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:16.495761 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":94005632}
W20260709 13:10:16.523314 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 391436us
I20260709 13:10:16.523932 25285 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:10:16.572258 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 48412.8us
I20260709 13:10:16.432091 31460 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:16.447973 31395 raft_consensus.cc:3055] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:10:16.579566 31395 raft_consensus.cc:740] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:16.579764 25240 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 13:10:16.580045 31461 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:16.580628 31436 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:16.580498 31395 consensus_queue.cc:260] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:16.581347 31436 raft_consensus.cc:697] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:16.581866 31395 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:16.582206 31436 consensus_queue.cc:237] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:16.546602 31318 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
I20260709 13:10:16.576994 25942 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:16.524966 31377 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:16.446214 31000 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:16.534112 25394 pending_rounds.cc:107] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 13:10:16.588786 31377 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:10:16.546103 25287 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.011s	sys 0.022s
I20260709 13:10:16.436048 31075 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:16.564075 31371 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:16.592460 25243 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:16.531325 30856 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:16.528924 25359 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:16.560073 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2583) took 1175 ms. Trace:
I20260709 13:10:16.606971 25244 rpcz_store.cc:276] 0709 13:10:15.384798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.559213 (+1174415us) service_pool.cc:224] Handling call
0709 13:10:16.559530 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:16.559691 (+   161us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:16.559728 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:16.559771 (+    43us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:10:16.559811 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:16.559872 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:16.559996 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:16.530171 31151 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:16.533535 31321 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 13:10:16.481807 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:16.491181 31386 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:16.533870 31284 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:16.564145 25246 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:16.616766 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:16.551901 31459 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:16.573691 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:16.526894 25909 leader_election.cc:336] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 2.221s (SENT)
W20260709 13:10:16.621703 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 94848.4us
I20260709 13:10:16.624509 25909 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:16.574940 31460 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:16.579134 31326 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:16.588884 25394 raft_consensus.cc:2889] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:10:16.592583 25287 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.012s	sys 0.023s
W20260709 13:10:16.656772 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 764215us
I20260709 13:10:16.658802 25314 raft_consensus.cc:1240] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:16.659718 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:16.660050 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:16.660389 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:16.660734 25314 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:16.594182 25908 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:16.661021 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:16.661531 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:16.661595 31312 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:16.661921 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:16.662186 25315 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:16.664431 31308 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:16.665167 31308 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:16.548141 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:16.580672 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2518) took 1135 ms. Trace:
I20260709 13:10:16.670331 25240 rpcz_store.cc:276] 0709 13:10:15.445171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.579078 (+1133907us) service_pool.cc:224] Handling call
0709 13:10:16.579475 (+   397us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:16.580327 (+   852us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:16.580601 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:16.551031 25386 raft_consensus.cc:2393] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 17: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:16.604614 25317 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:16.650471 31459 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:16.617686 25285 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:10:16.674496 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 56898.8us
I20260709 13:10:16.617676 25246 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:16.619405 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 90557.7us
I20260709 13:10:16.676775 25359 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:16.678382 31280 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:16.678862 31280 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:16.679316 31280 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:16.625620 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:16.680598 25937 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:16.620148 31321 consensus_queue.cc:1243] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 16.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:10:16.681697 31272 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:16.682269 31272 raft_consensus.cc:697] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:16.683156 31272 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:16.655447 31460 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:16.688668 31308 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:16.662227 31312 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:16.690935 31363 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:16.691838 31363 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:10:16.692305 31312 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:16.615490 31151 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:16.699067 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:16.701625 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:16.701934 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:16.702028 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:16.702847 31284 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:16.702244 25314 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 17
I20260709 13:10:16.704591 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:16.704613 25321 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:10:16.705922 31401 raft_consensus.cc:3055] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:16.705479 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:16.706488 31401 raft_consensus.cc:740] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:16.707057 25942 raft_consensus.cc:2410] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 13:10:16.708256 25321 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:10:16.707588 31401 consensus_queue.cc:260] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:16.711191 25314 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:16.711819 25246 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:10:16.712529 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2521) took 1265 ms. Trace:
W20260709 13:10:16.721668 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 127601us
I20260709 13:10:16.721974 25943 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:10:16.721834 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2515) took 1442 ms. Trace:
I20260709 13:10:16.722841 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2579) took 1518 ms. Trace:
I20260709 13:10:16.722795 25245 rpcz_store.cc:276] 0709 13:10:15.279577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:15.897847 (+618270us) service_pool.cc:224] Handling call
0709 13:10:15.898142 (+   295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:16.441816 (+543674us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:16.721594 (+279778us) spinlock_profiling.cc:243] Waited 543 ms on lock 0x7b54000a00d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:16.721774 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1249012608}
I20260709 13:10:16.723248 25241 rpcz_store.cc:276] 0709 13:10:15.204166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:15.872878 (+668712us) service_pool.cc:224] Handling call
0709 13:10:16.465227 (+592349us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:16.466067 (+   840us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:16.722654 (+256587us) spinlock_profiling.cc:243] Waited 592 ms on lock 0x7b5400422550. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:16.722787 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1361042176}
I20260709 13:10:16.724471 25245 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 13:10:16.710318 31458 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:16.726258 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2524) took 1106 ms. Trace:
I20260709 13:10:16.730927 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:16.726845 25245 rpcz_store.cc:276] 0709 13:10:15.619983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.723882 (+1103899us) service_pool.cc:224] Handling call
0709 13:10:16.724203 (+   321us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:16.724982 (+   779us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:16.726177 (+  1195us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:16.732398 25315 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:16.712951 25246 rpcz_store.cc:276] 0709 13:10:15.447330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.617190 (+1169860us) service_pool.cc:224] Handling call
0709 13:10:16.617541 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:16.712296 (+ 94755us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:16.712481 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:16.733739 25358 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:16.573067 31409 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:16.739125 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:16.742595 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:16.621876 31419 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:16.747339 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 122929us
I20260709 13:10:16.750479 25909 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:16.736773 31401 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:16.745765 25315 raft_consensus.cc:2368] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 18. Current term is 19.
I20260709 13:10:16.739090 31464 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:10:16.745741 31239 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:16.785280 25240 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 11. (term mismatch)
I20260709 13:10:16.799990 25240 pending_rounds.cc:85] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 10
I20260709 13:10:16.800251 25240 pending_rounds.cc:107] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260709 13:10:16.800486 25240 raft_consensus.cc:2889] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:16.785833 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:16.801723 25938 raft_consensus.cc:2410] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 13:10:16.785943 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:16.802466 25942 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:16.790078 25909 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:16.790735 31466 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:16.791512 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:16.792518 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:16.793489 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:16.795655 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:16.811553 25939 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:16.796589 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:16.798506 31447 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:16.808383 25909 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:16.724628 31326 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:16.768739 31095 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:16.831626 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:16.832324 25392 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 17
I20260709 13:10:16.767951 30856 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:16.815014 25909 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:16.816594 31395 raft_consensus.cc:3055] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:16.817037 25938 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:10:16.817500 31239 raft_consensus.cc:697] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:16.818441 31409 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:16.753697 31312 consensus_queue.cc:1059] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.062s
I20260709 13:10:16.821921 31284 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:16.822453 25315 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:10:16.825551 31464 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:16.827196 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:16.827970 31326 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:16.828524 25320 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:16.875602 31284 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:16.769555 31308 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.060s
I20260709 13:10:16.834414 25909 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:16.835064 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:16.874042 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:16.877805 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:16.878463 25315 raft_consensus.cc:2410] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 13:10:16.878593 25388 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:16.889276 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2513) took 1610 ms. Trace:
I20260709 13:10:16.847890 25358 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:16.889823 25239 rpcz_store.cc:276] 0709 13:10:15.279121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:15.883248 (+604127us) service_pool.cc:224] Handling call
0709 13:10:15.883582 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:15.883797 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:10:15.962564 (+ 78767us) log.cc:844] Serialized 14 byte log entry
0709 13:10:16.495088 (+532524us) spinlock_profiling.cc:243] Waited 3.78 ms on lock 0x7b18004702c0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f297685223a 00007f2976cbca2a
0709 13:10:16.888885 (+393797us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:16.888946 (+    61us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:16.888999 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:16.889070 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:16.889210 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":78594,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":4887,"spinlock_wait_cycles":8683776,"thread_start_us":393549,"threads_started":1}
I20260709 13:10:16.892912 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:16.838850 31458 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:16.919391 31458 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:16.919981 31458 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:16.904539 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 70227.1us
W20260709 13:10:16.906673 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 420155us
I20260709 13:10:16.921753 25209 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:16.907667 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 442699us
I20260709 13:10:16.839233 30856 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:16.919445 25239 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:16.925652 25285 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:16.929754 25940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1121) took 1072 ms. Trace:
I20260709 13:10:16.925916 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:16.930896 25209 leader_election.cc:400] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17.
I20260709 13:10:16.930299 25940 rpcz_store.cc:276] 0709 13:10:15.857425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:15.858837 (+  1412us) service_pool.cc:224] Handling call
0709 13:10:15.859119 (+   282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:16.524185 (+665066us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:16.929456 (+405271us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b540037ee54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:16.929680 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1298103552}
I20260709 13:10:16.931282 25209 leader_election.cc:403] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 13:10:16.878616 31464 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:16.837598 31395 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:16.846117 31467 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:16.904791 31447 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:16.942812 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:16.944284 25392 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:16.831355 31466 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:16.876036 31239 consensus_queue.cc:237] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:16.845697 31095 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:16.961941 25315 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
W20260709 13:10:16.963351 25312 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s	user 0.005s	sys 0.025s
W20260709 13:10:16.963750 25312 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.005s	sys 0.025s
W20260709 13:10:16.965493 25387 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.024s	sys 0.019s
W20260709 13:10:16.966971 25387 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.511s	user 0.024s	sys 0.019s
I20260709 13:10:16.952682 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:16.952965 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:16.926744 30856 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:16.953833 31467 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:16.954284 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:16.954425 31421 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:16.955498 31284 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:16.874609 31068 consensus_queue.cc:1059] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.065s
I20260709 13:10:16.958424 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2786) took 1071 ms. Trace:
I20260709 13:10:16.940232 31409 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:16.933658 30884 consensus_queue.cc:1059] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.015s
I20260709 13:10:16.940289 31403 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:16.995657 25239 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:10:16.998764 25386 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:16.940308 31464 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:16.999185 25937 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:16.977535 31353 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.004s	sys 0.034s
I20260709 13:10:17.006373 25320 rpcz_store.cc:276] 0709 13:10:15.887097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.620677 (+733580us) service_pool.cc:224] Handling call
0709 13:10:16.686280 (+ 65603us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:16.876187 (+189907us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:16.958126 (+ 81939us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b5400275454. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:16.958352 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":326552960}
I20260709 13:10:17.014302 31075 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:17.071465 25314 raft_consensus.cc:1275] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
W20260709 13:10:17.094012 25936 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.003s	sys 0.007s
W20260709 13:10:17.094449 25936 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.004s	sys 0.008s
W20260709 13:10:17.094928 31425 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.003s	sys 0.009s
W20260709 13:10:17.095274 31425 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.003s	sys 0.009s
I20260709 13:10:17.096112 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2526) took 1363 ms. Trace:
I20260709 13:10:17.096447 25245 rpcz_store.cc:276] 0709 13:10:15.732923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.743698 (+1010775us) service_pool.cc:224] Handling call
0709 13:10:16.770103 (+ 26405us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:16.770555 (+   452us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:16.770593 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:16.771299 (+   706us) log.cc:844] Serialized 52 byte log entry
0709 13:10:17.095874 (+324575us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:17.095901 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:17.095946 (+    45us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:17.095975 (+    29us) raft_consensus.cc:1685] finished
0709 13:10:17.095981 (+     6us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:17.096069 (+    88us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":229349,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":12115,"thread_start_us":281545,"threads_started":1,"wal-append.queue_time_us":296485}
W20260709 13:10:17.097959 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 173741us
I20260709 13:10:17.098544 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:17.100440 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:17.102315 25245 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:10:17.102916 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2534) took 1056 ms. Trace:
I20260709 13:10:17.103215 25245 rpcz_store.cc:276] 0709 13:10:16.046519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:17.101618 (+1055099us) service_pool.cc:224] Handling call
0709 13:10:17.101997 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:17.102682 (+   685us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:17.102858 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:10:17.106739 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 234110us
I20260709 13:10:17.020035 31386 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:17.115373 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 191814us
W20260709 13:10:17.055362 31353 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.005s	sys 0.035s
I20260709 13:10:17.016181 31409 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:17.120162 25936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 1138) took 1222 ms. Trace:
I20260709 13:10:17.141325 25936 rpcz_store.cc:276] 0709 13:10:15.897801 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:15.897931 (+   130us) service_pool.cc:224] Handling call
0709 13:10:15.898199 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:15.898383 (+   184us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:10:15.970985 (+ 72602us) log.cc:844] Serialized 14 byte log entry
0709 13:10:17.094771 (+1123786us) log.cc:844] Serialized 14 byte log entry
0709 13:10:17.119707 (+ 24936us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:17.119772 (+    65us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:10:17.119833 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:17.119924 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:17.120083 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24660,"prepare.queue_time_us":583684,"prepare.run_cpu_time_us":4746,"prepare.run_wall_time_us":31234,"raft.queue_time_us":317954,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":7966,"thread_start_us":1109534,"threads_started":3,"wal-append.queue_time_us":326215}
I20260709 13:10:17.024809 31284 raft_consensus.cc:3055] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:17.135782 25937 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:17.123144 31239 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:17.153257 31239 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:17.155151 25320 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:10:17.155743 25320 raft_consensus.cc:606] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:17.160058 25287 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:17.163511 25393 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.015s	sys 0.046s
W20260709 13:10:17.165000 25393 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.015s	sys 0.046s
I20260709 13:10:17.004904 31467 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:17.173491 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 325711us
I20260709 13:10:17.149329 31284 raft_consensus.cc:740] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:17.150213 31409 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:17.019127 31395 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:17.190342 25287 leader_election.cc:400] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 13. Current term is 14.
I20260709 13:10:17.193288 25312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2792) took 1184 ms. Trace:
I20260709 13:10:17.163002 31421 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:17.149744 31477 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:17.195578 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2783) took 1355 ms. Trace:
I20260709 13:10:17.295050 25316 rpcz_store.cc:276] 0709 13:10:15.840128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.281738 (+441610us) service_pool.cc:224] Handling call
0709 13:10:16.281990 (+   252us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:16.533589 (+251599us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:17.195299 (+661710us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b54000636d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:17.195518 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":344387456}
W20260709 13:10:17.283179 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 93568.8us
I20260709 13:10:17.283648 25312 rpcz_store.cc:276] 0709 13:10:16.009103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.085839 (+ 76736us) service_pool.cc:224] Handling call
0709 13:10:16.214130 (+128291us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:16.214504 (+   374us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:16.214540 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:16.230344 (+ 15804us) log.cc:844] Serialized 26 byte log entry
0709 13:10:16.964179 (+733835us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:10:16.964225 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:16.964287 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:17.193051 (+228764us) raft_consensus.cc:1685] finished
0709 13:10:17.193071 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:17.193213 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":633372,"threads_started":1,"wal-append.queue_time_us":756749}
I20260709 13:10:17.296923 25287 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:17.298724 25287 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:17.300017 25287 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:17.136390 31386 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:17.197047 31167 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:17.322065 31466 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:17.324244 31413 consensus_meta.cc:287] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.727s	user 0.006s	sys 0.002s
I20260709 13:10:17.160312 31371 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.125s
I20260709 13:10:17.168011 31478 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17.
W20260709 13:10:17.338992 31151 consensus_meta.cc:287] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.645s	user 0.010s	sys 0.000s
W20260709 13:10:17.341185 31460 consensus_meta.cc:287] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.654s	user 0.011s	sys 0.000s
I20260709 13:10:17.344007 31386 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:17.344628 31458 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:17.302352 31421 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:17.899295 31421 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:17.326646 31095 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:17.334285 31280 consensus_meta.cc:287] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.654s	user 0.012s	sys 0.000s
I20260709 13:10:17.900884 31280 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:17.903391 31280 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:17.906710 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 2210) took 1577 ms. Trace:
I20260709 13:10:17.907174 25393 rpcz_store.cc:276] 0709 13:10:16.328871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.479196 (+150325us) service_pool.cc:224] Handling call
0709 13:10:16.525878 (+ 46682us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:16.531994 (+  6116us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:16.532040 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:16.532654 (+   614us) log.cc:844] Serialized 26 byte log entry
0709 13:10:17.165534 (+632880us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:17.165593 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:17.165766 (+   173us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:17.906444 (+740678us) raft_consensus.cc:1685] finished
0709 13:10:17.906474 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:17.906642 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":934,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":79,"thread_start_us":600986,"threads_started":1,"wal-append.queue_time_us":910788}
I20260709 13:10:17.908268 31154 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:17.908978 31154 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:17.911549 31154 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:17.913677 31339 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:17.914482 31339 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:17.919301 31460 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:17.934844 31460 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:17.572631 31511 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:10:17.602983 31467 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:17.618911 31515 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 13:10:17.620056 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 320096us
I20260709 13:10:18.107622 25287 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:17.452772 31448 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:18.122757 25287 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:17.639907 30856 consensus_meta.cc:287] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.666s	user 0.008s	sys 0.001s
I20260709 13:10:17.646139 31390 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:17.752133 25358 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.060s	sys 0.047s
W20260709 13:10:18.135998 25358 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.960s	user 0.060s	sys 0.047s
W20260709 13:10:18.136375 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 960207us
W20260709 13:10:18.137104 25358 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.500s (SENT) [suppressed 17 similar messages]
I20260709 13:10:17.936784 25942 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
W20260709 13:10:17.942232 31290 consensus_meta.cc:287] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.508s	user 0.008s	sys 0.003s
I20260709 13:10:17.946172 31458 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:17.789515 31284 consensus_queue.cc:260] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:17.467640 31395 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:17.952288 31401 consensus_meta.cc:287] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.170s	user 0.006s	sys 0.005s
I20260709 13:10:17.952783 31339 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:17.955379 31095 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:17.959275 25386 consensus_meta.cc:287] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.905s	user 0.008s	sys 0.000s
W20260709 13:10:17.970048 31459 consensus_meta.cc:287] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 1.297s	user 0.012s	sys 0.000s
I20260709 13:10:17.790282 31151 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:17.442291 31413 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:17.940972 31377 consensus_meta.cc:287] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.351s	user 0.010s	sys 0.000s
W20260709 13:10:17.558295 25388 consensus_meta.cc:287] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.678s	user 0.005s	sys 0.005s
I20260709 13:10:18.179526 25388 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:18.180330 25388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 2219) took 1343 ms. Trace:
I20260709 13:10:18.180889 25388 rpcz_store.cc:276] 0709 13:10:16.836497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.836685 (+   188us) service_pool.cc:224] Handling call
0709 13:10:18.180247 (+1343562us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:10:17.983014 31464 consensus_meta.cc:287] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.923s	user 0.007s	sys 0.000s
I20260709 13:10:18.131942 25287 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:18.167141 25358 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:18.183187 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 51567us
I20260709 13:10:18.184201 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:18.184414 25358 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:18.172765 25386 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:18.165427 31290 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:18.189584 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.190449 25316 raft_consensus.cc:2410] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 16 because replica has last-logged OpId of term: 15 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 12.
I20260709 13:10:18.207312 25318 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:18.207108 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.190960 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.201721 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.207602 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.212477 25320 raft_consensus.cc:2410] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 15 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 13:10:18.136144 31312 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:18.135066 31511 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:10:18.208242 31290 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:18.214687 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.207361 31242 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:18.215838 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.216235 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:18.216468 25319 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:18.218832 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:18.223099 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.223522 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:18.224160 25321 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:10:18.225371 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.227757 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.207769 31515 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:18.191884 31167 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:18.208913 31154 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:18.208184 31284 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:18.240156 25391 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.753s	user 0.008s	sys 0.007s
W20260709 13:10:18.243909 25391 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.760s	user 0.008s	sys 0.007s
I20260709 13:10:18.174242 31421 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:18.223088 31409 consensus_meta.cc:287] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.922s	user 0.011s	sys 0.000s
I20260709 13:10:18.141965 30856 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:18.222615 31448 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:18.178596 31095 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:18.213594 31339 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.031s
I20260709 13:10:18.181568 31459 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:18.262667 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 1144) took 1477 ms. Trace:
I20260709 13:10:18.262957 30856 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:18.251752 31448 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:18.251835 31409 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:18.266639 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2704) took 1487 ms. Trace:
I20260709 13:10:18.281962 31409 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:18.210635 31464 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:18.270376 25937 rpcz_store.cc:276] 0709 13:10:16.785154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.792171 (+  7017us) service_pool.cc:224] Handling call
0709 13:10:18.262497 (+1470326us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b5400703854. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:18.262603 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":433203072}
I20260709 13:10:18.285064 25943 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: [16.15-16.15]
I20260709 13:10:18.276676 25315 rpcz_store.cc:276] 0709 13:10:16.779224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.905829 (+126605us) service_pool.cc:224] Handling call
0709 13:10:16.961528 (+ 55699us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:16.962488 (+   960us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:18.266297 (+1303809us) spinlock_profiling.cc:243] Waited 55.3 ms on lock 0x7b5400275450. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:18.266541 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":127206144}
I20260709 13:10:18.287035 25315 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:18.249341 31421 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:18.297298 31448 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:18.296584 25909 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.372s	user 0.034s	sys 0.067s
W20260709 13:10:18.299690 25909 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.375s	user 0.034s	sys 0.067s
I20260709 13:10:18.301990 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:10:18.302220 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 117883us
I20260709 13:10:18.302759 25936 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 18: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:18.305186 25358 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:18.307370 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:18.315882 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:18.313549 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:18.323033 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:18.323868 25936 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:18.324915 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:18.325727 25936 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:18.332052 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:18.311167 31154 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:18.311575 31339 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:18.313724 31458 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:18.313488 31460 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:18.336660 31151 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:18.332532 25286 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.926s	user 0.066s	sys 0.095s
I20260709 13:10:18.327554 31242 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:18.337280 31339 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:18.352906 31499 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 13:10:18.333551 31413 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:18.352939 25936 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
W20260709 13:10:18.357136 25286 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.951s	user 0.066s	sys 0.096s
I20260709 13:10:18.351670 31095 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:18.351681 31154 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:18.370441 25936 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:18.338095 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:18.380672 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:18.361900 31339 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:18.349902 31458 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:18.350769 31460 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:18.392043 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:18.379787 31242 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:10:18.400462 31511 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:18.401074 31511 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:18.403316 31511 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:18.407995 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2702) took 1661 ms. Trace:
I20260709 13:10:18.408577 25317 rpcz_store.cc:276] 0709 13:10:16.762263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.770451 (+  8188us) service_pool.cc:224] Handling call
0709 13:10:16.770910 (+   459us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:16.771379 (+   469us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:16.771419 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:16.785240 (+ 13821us) log.cc:844] Serialized 26 byte log entry
0709 13:10:17.166450 (+381210us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:10:17.166514 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:17.166605 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:18.407710 (+1241105us) raft_consensus.cc:1685] finished
0709 13:10:18.407741 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:18.407916 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6186,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":20524,"thread_start_us":320097,"threads_started":1,"wal-append.queue_time_us":1341573}
W20260709 13:10:18.427960 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 47392.4us
I20260709 13:10:18.428370 31532 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:18.429075 25358 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:18.383378 31095 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.015s
I20260709 13:10:18.395368 31459 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:18.387198 31523 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.043s
I20260709 13:10:18.424433 31421 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:18.399657 31464 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:18.429957 31409 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:18.442560 31242 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:18.463574 31532 raft_consensus.cc:697] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:18.464637 31532 consensus_queue.cc:237] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:18.470412 25358 leader_election.cc:400] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 18. Current term is 19.
I20260709 13:10:18.470830 25358 leader_election.cc:403] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:10:18.482342 31524 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 13:10:18.473253 31421 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 18. Current term is 19.
I20260709 13:10:18.470221 31448 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20260709 13:10:18.504797 31464 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:18.505219 31460 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:18.505476 31464 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:18.506150 25909 leader_election.cc:336] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 1.957s (SENT)
I20260709 13:10:18.507673 31464 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:18.509853 31458 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:18.511564 31312 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:18.479700 31095 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.033s
I20260709 13:10:18.455363 31409 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:18.515945 31409 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:18.531246 31154 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:18.531010 31339 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:18.546936 31339 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:18.552043 31318 raft_consensus.cc:3055] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:10:18.552559 31318 raft_consensus.cc:740] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:18.553452 31318 consensus_queue.cc:260] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:18.554554 31318 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:10:18.633898 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 160343us
I20260709 13:10:18.637326 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:18.638815 31532 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:18.639236 31532 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:18.639602 31532 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:18.640724 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.641657 25313 raft_consensus.cc:2410] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 13:10:18.641965 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.642397 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.643297 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.644150 25315 raft_consensus.cc:2393] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:18.644946 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.645961 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.642597 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:18.643333 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:18.649266 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:18.649894 25313 raft_consensus.cc:2376] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14. Re-sending same reply.
I20260709 13:10:18.651531 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.653065 25317 raft_consensus.cc:2393] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 20: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:18.653414 31095 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:18.654014 31095 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:18.656349 31095 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:18.646108 31339 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.011s
W20260709 13:10:18.660936 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 56688.3us
I20260709 13:10:18.661188 31541 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:18.688261 31450 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:18.698648 25245 raft_consensus.cc:3055] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:18.699158 25245 raft_consensus.cc:740] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:18.699843 25245 consensus_queue.cc:260] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:18.694819 31541 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:18.704845 31541 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:18.711041 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.714735 31532 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:18.717841 31532 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:18.719169 31520 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.364s	user 0.001s	sys 0.002s
I20260709 13:10:18.721042 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 2204) took 2400 ms. Trace:
I20260709 13:10:18.722110 25942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 1155) took 1933 ms. Trace:
I20260709 13:10:18.721465 25391 rpcz_store.cc:276] 0709 13:10:16.320783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.449738 (+128955us) service_pool.cc:224] Handling call
0709 13:10:16.450173 (+   435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:16.450666 (+   493us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:16.450705 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:17.167459 (+716754us) spinlock_profiling.cc:243] Waited 3.22 ms on lock 0x7b2c000e1828. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f29755aba11 00007f29755bb372 00007f2976ceda01 00007f2976cb243a 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2
0709 13:10:17.190210 (+ 22751us) log.cc:844] Serialized 26 byte log entry
0709 13:10:18.244398 (+1054188us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:18.244437 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:18.244558 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:18.718556 (+473998us) spinlock_profiling.cc:243] Waited 1.5 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:18.720786 (+  2230us) raft_consensus.cc:1685] finished
0709 13:10:18.720810 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:18.720957 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":128,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":104,"spinlock_wait_cycles":10853632,"thread_start_us":759931,"threads_started":1,"wal-append.queue_time_us":1013717}
I20260709 13:10:18.722681 25942 rpcz_store.cc:276] 0709 13:10:16.788650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:17.054803 (+266153us) service_pool.cc:224] Handling call
0709 13:10:17.119028 (+ 64225us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:18.165105 (+1046077us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:18.721872 (+556767us) spinlock_profiling.cc:243] Waited 817 ms on lock 0x7b540062a2d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:18.722054 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1880057216}
I20260709 13:10:18.740118 31239 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:18.738539 31532 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
W20260709 13:10:18.742538 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:18.743242 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:18.746152 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:18.752429 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2585) took 3332 ms. Trace:
I20260709 13:10:18.755877 25358 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:18.757516 31340 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:18.761624 25943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 1150) took 1974 ms. Trace:
I20260709 13:10:18.762094 25943 rpcz_store.cc:276] 0709 13:10:16.787016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.874120 (+ 87104us) service_pool.cc:224] Handling call
0709 13:10:16.875941 (+  1821us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:18.285519 (+1409578us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:18.761308 (+475789us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b5400014d54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:18.761550 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3240120192}
I20260709 13:10:18.752974 25242 rpcz_store.cc:276] 0709 13:10:15.419980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.571147 (+1151167us) service_pool.cc:224] Handling call
0709 13:10:16.601482 (+ 30335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:16.601872 (+   390us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:16.601912 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:16.928682 (+326770us) spinlock_profiling.cc:243] Waited 3.24 ms on lock 0x7b2c000014c8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f29755aba11 00007f29755bb372 00007f2976ceda01 00007f2976cb243a 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2
0709 13:10:16.929232 (+   550us) log.cc:844] Serialized 26 byte log entry
0709 13:10:17.114408 (+185176us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:17.114450 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:17.114524 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:18.752072 (+1637548us) raft_consensus.cc:1685] finished
0709 13:10:18.752103 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:18.752358 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":199253,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":7495,"spinlock_wait_cycles":7456000,"thread_start_us":159358,"threads_started":1,"wal-append.queue_time_us":211158,"wal-append.run_cpu_time_us":2035,"wal-append.run_wall_time_us":1160302}
I20260709 13:10:18.767020 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:18.767812 25242 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
W20260709 13:10:18.768553 25242 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32994 (request call id 2598) took 1659 ms (client timeout 2062 ms). Trace:
W20260709 13:10:18.769117 25242 rpcz_store.cc:269] 0709 13:10:17.109047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:18.766711 (+1657664us) service_pool.cc:224] Handling call
0709 13:10:18.768476 (+  1765us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:18.770975 25242 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
W20260709 13:10:18.771315 25239 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 2543) took 1940 ms (client timeout 1921 ms). Trace:
W20260709 13:10:18.774016 25239 rpcz_store.cc:269] 0709 13:10:16.831175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.925607 (+ 94432us) service_pool.cc:224] Handling call
0709 13:10:18.771117 (+1845510us) spinlock_profiling.cc:243] Waited 64.8 ms on lock 0x7b54003a9dd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:18.771225 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":148949888}
I20260709 13:10:18.775697 25239 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:18.776362 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2588) took 2473 ms. Trace:
I20260709 13:10:18.776680 25239 rpcz_store.cc:276] 0709 13:10:16.302588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:18.774937 (+2472349us) service_pool.cc:224] Handling call
0709 13:10:18.775239 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:18.776122 (+   883us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:18.776307 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:10:18.781685 31533 log.cc:927] Time spent T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.324s	user 0.002s	sys 0.001s
I20260709 13:10:18.786782 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2589) took 2484 ms. Trace:
I20260709 13:10:18.787498 25239 rpcz_store.cc:276] 0709 13:10:16.302850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:18.780110 (+2477260us) service_pool.cc:224] Handling call
0709 13:10:18.780396 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:18.780564 (+   168us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:18.780590 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:18.780627 (+    37us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:10:18.786260 (+  5633us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:18.786543 (+   283us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:18.786696 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:18.789830 25239 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:10:18.774204 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2586) took 2472 ms. Trace:
I20260709 13:10:18.791965 25242 rpcz_store.cc:276] 0709 13:10:16.302078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:18.770080 (+2468002us) service_pool.cc:224] Handling call
0709 13:10:18.770537 (+   457us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:18.773941 (+  3404us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:18.774149 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:18.772042 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:18.793692 25312 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:18.790671 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2591) took 2487 ms. Trace:
I20260709 13:10:18.798494 25239 rpcz_store.cc:276] 0709 13:10:16.303396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:18.788879 (+2485483us) service_pool.cc:224] Handling call
0709 13:10:18.789317 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:18.790337 (+  1020us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:18.790602 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:10:18.808682 25243 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 2510) took 3813 ms (client timeout 1563 ms). Trace:
W20260709 13:10:18.809226 25243 rpcz_store.cc:269] 0709 13:10:14.994951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:15.106031 (+111080us) service_pool.cc:224] Handling call
0709 13:10:18.808398 (+3702367us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b5400456354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:18.808582 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3142682496}
I20260709 13:10:18.810796 25243 pending_rounds.cc:85] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 9
I20260709 13:10:18.811092 25243 pending_rounds.cc:107] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 16.10
I20260709 13:10:18.811344 25243 raft_consensus.cc:2889] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:18.827086 31326 raft_consensus.cc:3055] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:18.828393 31326 raft_consensus.cc:740] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:18.829383 31326 consensus_queue.cc:260] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:18.830493 31326 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:18.834088 31469 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 1.830s	user 0.005s	sys 0.012s
I20260709 13:10:18.836158 25944 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 1208) took 2960 ms. Trace:
W20260709 13:10:18.837625 25386 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 2220) took 1991 ms (client timeout 2114 ms). Trace:
I20260709 13:10:18.836535 25944 rpcz_store.cc:276] 0709 13:10:15.875916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:15.876067 (+   151us) service_pool.cc:224] Handling call
0709 13:10:15.886979 (+ 10912us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:16.214658 (+327679us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:16.214692 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:16.222950 (+  8258us) log.cc:844] Serialized 52 byte log entry
0709 13:10:16.613313 (+390363us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:17.057967 (+444654us) log.cc:844] Serialized 14 byte log entry
0709 13:10:17.058292 (+   325us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:17.058398 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:18.833516 (+1775118us) spinlock_profiling.cc:243] Waited 7.75 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:18.835832 (+  2316us) raft_consensus.cc:1685] finished
0709 13:10:18.835940 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:18.836096 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10938,"prepare.queue_time_us":468116,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":248444,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":70233,"spinlock_wait_cycles":17832704,"thread_start_us":1092109,"threads_started":3,"wal-append.queue_time_us":713802}
W20260709 13:10:18.838024 25386 rpcz_store.cc:269] 0709 13:10:16.845832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.954009 (+108177us) service_pool.cc:224] Handling call
0709 13:10:18.837471 (+1883462us) spinlock_profiling.cc:243] Waited 3.27 ms on lock 0x7b5400481554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:18.837572 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7526784}
I20260709 13:10:18.844059 25312 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:10:18.860707 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:18.867791 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2525) took 3153 ms. Trace:
W20260709 13:10:18.869221 25909 leader_election.cc:336] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 1.946s (SENT)
I20260709 13:10:18.869704 25909 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:18.870368 31459 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:18.868268 25241 rpcz_store.cc:276] 0709 13:10:15.714103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.738388 (+1024285us) service_pool.cc:224] Handling call
0709 13:10:16.771957 (+ 33569us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:16.772372 (+   415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:16.772413 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:16.772949 (+   536us) log.cc:844] Serialized 26 byte log entry
0709 13:10:18.867177 (+2094228us) spinlock_profiling.cc:243] Waited 5.01 ms on lock 0x7b18003981a0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f297685a4bc 00007f2973ba848b 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f29768520d1
0709 13:10:18.867359 (+   182us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:10:18.867399 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:18.867476 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:18.867523 (+    47us) raft_consensus.cc:1685] finished
0709 13:10:18.867535 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:18.867731 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":74704,"raft.queue_time_us":251939,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":5608,"spinlock_wait_cycles":11524480,"thread_start_us":259112,"threads_started":1,"wal-append.queue_time_us":1160699}
I20260709 13:10:18.864042 31536 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:10:18.873809 25241 raft_consensus.cc:1275] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:10:18.874656 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2593) took 1996 ms. Trace:
I20260709 13:10:18.875034 25241 rpcz_store.cc:276] 0709 13:10:16.877778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:18.872432 (+1994654us) service_pool.cc:224] Handling call
0709 13:10:18.873432 (+  1000us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:18.874382 (+   950us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:18.874593 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:18.876578 25241 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 13:10:18.902879 31536 raft_consensus.cc:697] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:18.906208 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2594) took 2028 ms. Trace:
I20260709 13:10:18.906657 25241 rpcz_store.cc:276] 0709 13:10:16.878035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:18.875899 (+1997864us) service_pool.cc:224] Handling call
0709 13:10:18.876232 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:18.905889 (+ 29657us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:18.906146 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:18.906522 31536 consensus_queue.cc:237] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:18.911916 31383 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:18.912570 31383 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:18.914325 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2595) took 2036 ms. Trace:
I20260709 13:10:18.914757 25241 rpcz_store.cc:276] 0709 13:10:16.878288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:18.911237 (+2032949us) service_pool.cc:224] Handling call
0709 13:10:18.911626 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:18.911814 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:18.911842 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:18.911882 (+    40us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:10:18.911941 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:18.912001 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:18.914249 (+  2248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:18.917552 31383 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:18.919615 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.920025 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2684) took 2260 ms. Trace:
I20260709 13:10:18.920363 25318 rpcz_store.cc:276] 0709 13:10:16.659098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.663759 (+  4661us) service_pool.cc:224] Handling call
0709 13:10:16.664111 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:18.210265 (+1546154us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:18.919764 (+709499us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b54000631d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:18.919941 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3548539136}
I20260709 13:10:18.925973 31383 consensus_queue.cc:1059] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:10:18.926782 31536 raft_consensus.cc:3055] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:18.927245 31536 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:18.928344 31536 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.15, Last appended by leader: 15, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:18.929702 31536 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:18.931385 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:18.944500 31449 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:18.946820 31449 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:18.946175 25909 leader_election.cc:336] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 2.023s (SENT)
I20260709 13:10:18.949510 25909 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:18.951146 31459 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:10:18.954871 31449 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:18.955367 31472 log.cc:927] Time spent T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 2.007s	user 0.007s	sys 0.008s
I20260709 13:10:18.957337 25359 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:18.964370 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 1.86529e+06us
I20260709 13:10:18.966904 31280 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:18.967111 25245 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:18.981112 31449 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:18.982618 31449 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:18.985637 25244 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:18.986275 31449 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:18.987982 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2522) took 3540 ms. Trace:
I20260709 13:10:18.988404 25244 rpcz_store.cc:276] 0709 13:10:15.447979 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.713537 (+1265558us) service_pool.cc:224] Handling call
0709 13:10:16.713930 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:18.986131 (+2272201us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:18.987712 (+  1581us) spinlock_profiling.cc:243] Waited 2.27 s on lock 0x7b5400362c54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:18.987917 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5223949184}
I20260709 13:10:18.995545 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.995896 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.996240 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:18.994575 25245 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:10:18.997267 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2535) took 2821 ms. Trace:
I20260709 13:10:18.997772 25321 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 17: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:18.995759 31459 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:18.998075 25245 rpcz_store.cc:276] 0709 13:10:16.175723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:17.104436 (+928713us) service_pool.cc:224] Handling call
0709 13:10:17.104737 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:18.698548 (+1593811us) spinlock_profiling.cc:243] Waited 44.9 ms on lock 0x7b54000f58a0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976c29dc0 00007f2976cb41c0 00007f2976cb5231 00007f2976cb1d8f 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c
0709 13:10:18.966522 (+267974us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b5800512fd0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976c0e0aa 00007f2976c90447 00007f2976ca8fe7 00007f2976caa7de 00007f2976cb4b49 00007f2976cb5416 00007f2976cb1d8f 00007f2976cb183e 00007f297bc76419 00007f29751edbd4
0709 13:10:18.996982 (+ 30460us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:18.997212 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":710046208}
I20260709 13:10:18.999181 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:19.000365 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:19.002435 25908 connection.cc:256] RPC call timeout handler was delayed by -1.6701590529999066s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16000000000000003s for a response
W20260709 13:10:19.002782 25908 connection.cc:256] RPC call timeout handler was delayed by -1.3956590529999175s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1905s for a response
W20260709 13:10:19.003005 25908 connection.cc:256] RPC call timeout handler was delayed by -1.206659052999953s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21150000000000002s for a response
W20260709 13:10:19.003304 25908 connection.cc:256] RPC call timeout handler was delayed by -1.1067590529999052s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22260000000000002s for a response
I20260709 13:10:19.006366 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:19.007151 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:19.007187 25908 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:19.007823 25945 raft_consensus.cc:2376] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17. Re-sending same reply.
I20260709 13:10:19.008396 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:19.008651 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:19.000509 31362 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 19 current term: 19 is higher.
I20260709 13:10:19.009513 25321 raft_consensus.cc:2393] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 17: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:19.009011 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:19.009706 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:19.010210 25938 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:10:19.010340 25312 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:19.005892 31461 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:19.013657 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:19.014467 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:19.014361 31461 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:19.014278 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:19.015179 25942 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:19.012068 31541 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:19.013690 31518 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:19.017652 31541 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:18.998980 31459 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:19.018548 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 1.90204e+06us
I20260709 13:10:19.019778 31541 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:19.019853 25212 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:19.020218 31459 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:19.014118 31095 consensus_queue.cc:1059] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:10:19.022424 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:19.023370 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:19.023761 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:18.994894 31449 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:19.009909 31450 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:19.025228 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:19.022524 25321 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:10:19.013794 25937 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:10:19.012324 31154 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:19.022264 25908 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:19.025895 25393 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:19.029991 25908 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:19.030947 25908 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:19.031164 31450 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:19.037400 25386 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:10:19.038980 31450 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:19.039649 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:19.040360 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:19.040565 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:19.040683 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:19.041201 25395 raft_consensus.cc:1240] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 13:10:19.041203 25389 raft_consensus.cc:2410] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 17 because replica has last-logged OpId of term: 16 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 14.
I20260709 13:10:19.043709 25391 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 13:10:19.044279 25942 raft_consensus.cc:2468] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:19.046427 25321 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:10:19.045882 31450 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:10:19.042584 31154 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:19.048641 31239 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:10:19.048642 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:19.050513 25386 raft_consensus.cc:2393] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 15: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:19.066274 25393 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 13:10:19.052670 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:19.068176 25321 raft_consensus.cc:2393] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 17: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:19.041980 31541 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.004s
I20260709 13:10:19.073477 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:19.024528 31383 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:19.077775 25393 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:19.078578 25386 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:10:19.083474 25238 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:10:19.085246 31389 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:19.044718 31460 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:19.099710 25287 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:19.066362 31461 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:19.050531 31154 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:19.109846 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 64711us
I20260709 13:10:19.029775 31449 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:19.117789 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:19.047370 31518 consensus_queue.cc:1059] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 13:10:19.121809 25944 raft_consensus.cc:2393] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 13: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:19.124140 31537 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:19.123950 31510 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:19.101272 31383 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:19.144769 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:19.149207 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:19.142225 31564 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:19.151257 31564 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:19.139070 31223 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.005s
I20260709 13:10:19.156081 31564 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:10:19.128051 31519 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.020s
I20260709 13:10:19.147760 31510 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:19.161350 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 47828.1us
I20260709 13:10:19.171629 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:19.163290 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:19.170586 31510 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:19.175087 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:19.176062 25358 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:19.176767 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:19.176985 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:19.177299 31536 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:10:19.177472 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:19.178131 25937 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:10:19.197979 25942 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 19: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:19.181759 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:19.186511 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:19.187382 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:19.186574 31403 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:19.188288 31449 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:19.189318 31539 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:19.203770 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:19.190565 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:19.193430 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:19.205116 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:19.202894 25388 raft_consensus.cc:2410] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 17 because replica has last-logged OpId of term: 16 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 14.
I20260709 13:10:19.189464 25358 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:19.206910 31095 raft_consensus.cc:2804] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:19.207168 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:19.207922 25395 raft_consensus.cc:2393] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 16: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:19.208117 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:19.208388 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:19.208602 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:19.208959 25393 raft_consensus.cc:2410] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 13:10:19.210391 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:19.211339 25395 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:10:19.214022 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:19.218046 25911 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:19.219683 25911 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:19.223546 25911 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:19.228987 31569 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:19.251235 25911 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:19.251783 31571 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:19.253765 31571 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:19.254177 31571 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:19.228176 31564 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:10:19.225625 31389 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:10:19.265960 25942 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:45428
I20260709 13:10:19.266618 25942 raft_consensus.cc:606] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:19.266955 25942 raft_consensus.cc:614] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:10:19.267848 25939 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:10:19.268823 25940 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:19.269361 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:19.270216 25937 raft_consensus.cc:2393] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 17: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:19.270690 25943 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:10:19.271927 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 1077) took 3819 ms. Trace:
I20260709 13:10:19.272047 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:19.272372 25240 rpcz_store.cc:276] 0709 13:10:15.452552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.671479 (+1218927us) service_pool.cc:224] Handling call
0709 13:10:16.671933 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:16.800921 (+128988us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:19.271679 (+2470758us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b5400193054. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:19.271845 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":259799936}
I20260709 13:10:19.283490 25938 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
mode: GRACEFUL
new_leader_uuid: "2329e428ebbf484fa3ea9452133df567"
 from {username='slave'} at 127.0.0.1:45428
I20260709 13:10:19.281313 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:19.295372 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:19.296204 25388 raft_consensus.cc:2376] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19. Re-sending same reply.
I20260709 13:10:19.228914 31403 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.069s
W20260709 13:10:19.302274 25284 connection.cc:256] RPC call timeout handler was delayed by -1.1076634209999838s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1652s for a response
W20260709 13:10:19.302837 25284 connection.cc:256] RPC call timeout handler was delayed by -1.0201747910000449s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1582s for a response
I20260709 13:10:19.304951 31519 consensus_queue.cc:1059] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:10:19.302322 25940 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:10:19.304893 25284 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:19.325204 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:19.336642 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:19.344982 25943 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:19.346342 25284 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:19.347196 25287 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:19.350039 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 142767us
I20260709 13:10:19.297338 31449 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:19.350924 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:19.361619 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:19.362515 25388 raft_consensus.cc:2376] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20. Re-sending same reply.
I20260709 13:10:19.355312 31425 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:19.352634 31272 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:19.355171 31436 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:19.368821 25237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2584) took 3949 ms. Trace:
I20260709 13:10:19.500798 25237 rpcz_store.cc:276] 0709 13:10:15.419712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.568400 (+1148688us) service_pool.cc:224] Handling call
0709 13:10:16.568688 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:16.568881 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:10:17.057910 (+489029us) log.cc:844] Serialized 14 byte log entry
0709 13:10:19.368304 (+2310394us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:19.368380 (+    76us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:19.368455 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:19.368555 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:19.368732 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2149174,"prepare.queue_time_us":525220,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"thread_start_us":626582,"threads_started":3,"wal-append.queue_time_us":49121,"wal-append.run_cpu_time_us":921,"wal-append.run_wall_time_us":1014190}
I20260709 13:10:19.354930 31537 consensus_queue.cc:1059] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.143s
I20260709 13:10:19.370010 31425 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:19.542335 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 194315us
I20260709 13:10:19.347668 31095 raft_consensus.cc:697] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:19.497483 31272 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:19.537902 31436 raft_consensus.cc:697] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:19.605715 31425 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:19.541548 31519 consensus_queue.cc:1059] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.079s
W20260709 13:10:19.648000 31575 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 13:10:19.665607 31571 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:19.669467 31584 raft_consensus.cc:3055] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:19.670161 31584 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:19.671223 31584 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:19.681878 31584 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:19.676290 31571 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:19.698071 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:19.707490 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2577) took 4558 ms. Trace:
I20260709 13:10:19.707991 25238 rpcz_store.cc:276] 0709 13:10:15.148659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:15.821336 (+672677us) service_pool.cc:224] Handling call
0709 13:10:19.083131 (+3261795us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:19.095294 (+ 12163us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:19.707233 (+611939us) spinlock_profiling.cc:243] Waited 3.26 s on lock 0x7b5400456350. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:19.707416 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7500773120}
I20260709 13:10:19.722741 31095 consensus_queue.cc:237] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:19.765971 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:19.774832 31419 raft_consensus.cc:3055] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:10:19.787956 31419 raft_consensus.cc:740] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:19.789084 31419 consensus_queue.cc:260] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:19.614711 31272 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:19.790302 31419 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:10:19.785100 25938 raft_consensus.cc:606] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Received request to transfer leadership to TS 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:19.794919 25938 raft_consensus.cc:614] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:10:19.799439 31339 raft_consensus.cc:3055] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:19.799914 31339 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:19.801273 31272 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:19.803087 31272 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:19.822120 31425 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:19.824657 31425 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:19.844336 31510 consensus_meta.cc:287] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.639s	user 0.008s	sys 0.000s
I20260709 13:10:19.844810 31510 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:19.846803 31510 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:19.858014 31537 raft_consensus.cc:3055] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:19.858549 31537 raft_consensus.cc:740] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:19.859462 31537 consensus_queue.cc:260] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:19.864219 31537 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:19.867807 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2724) took 1187 ms. Trace:
I20260709 13:10:19.868407 25320 rpcz_store.cc:276] 0709 13:10:18.680122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:18.680260 (+   138us) service_pool.cc:224] Handling call
0709 13:10:18.680626 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:18.680996 (+   370us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:18.681024 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:18.684271 (+  3247us) log.cc:844] Serialized 26 byte log entry
0709 13:10:19.862945 (+1178674us) spinlock_profiling.cc:243] Waited 3.56 ms on lock 0x7b18002cdee0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f29768520d1 00007f2976c6da0f
0709 13:10:19.864108 (+  1163us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:10:19.864159 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:19.864264 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:19.867449 (+  3185us) raft_consensus.cc:1685] finished
0709 13:10:19.867473 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:19.867735 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56,"raft.queue_time_us":175,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":2873,"spinlock_wait_cycles":10002176,"thread_start_us":940,"threads_started":1,"wal-append.queue_time_us":1175498}
I20260709 13:10:19.884534 31460 raft_consensus.cc:3055] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:19.884977 31460 raft_consensus.cc:740] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:19.885856 31460 consensus_queue.cc:260] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:19.886725 31460 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:19.899713 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:19.915232 31436 consensus_queue.cc:237] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:19.931097 31436 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:19.931381 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:19.931819 31436 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:19.932390 31459 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:19.935925 31436 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:19.957607 25285 leader_election.cc:336] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 1.563s (SENT)
I20260709 13:10:19.958267 25285 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:19.965672 31436 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:19.971609 31460 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:19.972290 31460 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:19.974671 31460 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:19.972301 25287 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.045s	sys 0.038s
W20260709 13:10:19.976040 25287 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.047s	sys 0.039s
W20260709 13:10:19.977964 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 629214us
I20260709 13:10:19.982435 31459 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:19.982553 25320 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
W20260709 13:10:19.987226 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 636352us
I20260709 13:10:19.989274 31539 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:19.985255 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:19.999898 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:20.000602 25394 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 14
I20260709 13:10:20.006193 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:20.006598 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:20.010054 31339 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.011266 31339 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:20.046151 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:20.048259 25313 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:20.048720 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:20.049816 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 2233) took 1002 ms. Trace:
I20260709 13:10:20.050625 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:20.051815 25393 raft_consensus.cc:2410] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 18 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 13:10:20.049465 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:20.050360 25391 rpcz_store.cc:276] 0709 13:10:19.047366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:19.047505 (+   139us) service_pool.cc:224] Handling call
0709 13:10:19.047772 (+   267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:19.047962 (+   190us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:10:19.141411 (+ 93449us) log.cc:844] Serialized 14 byte log entry
0709 13:10:19.346386 (+204975us) log.cc:844] Serialized 14 byte log entry
0709 13:10:19.750151 (+403765us) log.cc:844] Serialized 14 byte log entry
0709 13:10:20.049370 (+299219us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:20.049442 (+    72us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:20.049513 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:20.049595 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:20.049739 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":725248,"prepare.queue_time_us":36715,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":116,"raft.queue_time_us":401928,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"thread_start_us":267363,"threads_started":2,"wal-append.queue_time_us":356326}
I20260709 13:10:20.052783 31539 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:20.056919 31459 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:20.059511 25390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 2229) took 1033 ms. Trace:
I20260709 13:10:20.059957 25390 rpcz_store.cc:276] 0709 13:10:19.026560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:19.026702 (+   142us) service_pool.cc:224] Handling call
0709 13:10:19.026971 (+   269us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:19.179470 (+152499us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:10:19.192849 (+ 13379us) log.cc:844] Serialized 14 byte log entry
0709 13:10:19.282341 (+ 89492us) log.cc:844] Serialized 14 byte log entry
0709 13:10:19.742724 (+460383us) spinlock_profiling.cc:243] Waited 29.4 ms on lock 0x7b1800384340. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b3fc0a 00007f297d27f219 00007f2973bc7a40 00007f2973bc24e4 00007f2973bc2455 00007f2976caa060 00007f2976cbc716 00007f2976ccd287 00007f2976ccd208
0709 13:10:19.749615 (+  6891us) log.cc:844] Serialized 14 byte log entry
0709 13:10:19.767841 (+ 18226us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b1800384340. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:10:20.043988 (+276147us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:20.044053 (+    65us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:20.044113 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:20.059250 (+ 15137us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b5400282dd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:20.059310 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:20.059447 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":594403,"prepare.queue_time_us":21919,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":105,"raft.queue_time_us":770589,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":196,"spinlock_wait_cycles":455430912,"thread_start_us":88889,"threads_started":1,"wal-append.queue_time_us":168284}
I20260709 13:10:20.063474 25392 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:20.072705 31523 consensus_queue.cc:1059] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.004s
I20260709 13:10:20.085124 31523 raft_consensus.cc:3055] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:20.085665 31523 raft_consensus.cc:740] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:20.086740 31523 consensus_queue.cc:260] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.087316 25938 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:45428
I20260709 13:10:20.089854 25911 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:20.091110 25911 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:20.099849 25911 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:20.100016 25937 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:20.109746 31461 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:20.102016 31450 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:10:20.107581 31154 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:20.102272 31449 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:20.127880 25320 raft_consensus.cc:1275] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:10:20.121991 31403 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:20.134379 31403 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.135059 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2540) took 3546 ms. Trace:
I20260709 13:10:20.135607 25239 rpcz_store.cc:276] 0709 13:10:16.588319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:18.812722 (+2224403us) service_pool.cc:224] Handling call
0709 13:10:18.813050 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:18.813326 (+   276us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:10:18.827745 (+ 14419us) log.cc:844] Serialized 14 byte log entry
0709 13:10:20.133494 (+1305749us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:20.134740 (+  1246us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:20.134799 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:20.134884 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:20.135001 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1085041,"prepare.queue_time_us":9946,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"thread_start_us":234341,"threads_started":3,"wal-append.queue_time_us":81375,"wal-append.run_cpu_time_us":1454,"wal-append.run_wall_time_us":1003940}
I20260709 13:10:20.136729 31403 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:20.137744 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:20.117093 31571 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:20.140302 25940 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:20.149932 25394 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:20.136746 31523 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:20.137428 31151 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:20.155510 25911 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:20.173511 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:20.139159 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2597) took 3208 ms. Trace:
I20260709 13:10:20.174968 25320 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: []
W20260709 13:10:20.177728 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:20.178194 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:20.177888 25241 rpcz_store.cc:276] 0709 13:10:16.930359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:18.916911 (+1986552us) service_pool.cc:224] Handling call
0709 13:10:18.924662 (+  7751us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:18.924852 (+   190us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:10:18.926264 (+  1412us) log.cc:844] Serialized 14 byte log entry
0709 13:10:20.138792 (+1212528us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:20.138852 (+    60us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:20.138925 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:20.139002 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:20.139108 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1003503,"prepare.queue_time_us":1007,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"thread_start_us":208554,"threads_started":2,"wal-append.queue_time_us":19723,"wal-append.run_cpu_time_us":1142,"wal-append.run_wall_time_us":1002123}
I20260709 13:10:20.130615 31154 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.155400 31386 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:20.199117 25358 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:20.199601 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:20.200950 25284 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.003s	sys 0.003s
I20260709 13:10:20.191654 31449 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:20.154361 31571 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.199806 31386 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:20.204725 25284 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.003s	sys 0.003s
W20260709 13:10:20.205168 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 611230us
I20260709 13:10:20.206548 31386 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:20.209587 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:20.210271 25394 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:20.211143 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.211831 25239 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:20.203395 31000 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:20.197109 31403 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.018s
I20260709 13:10:20.214228 31580 consensus_queue.cc:1059] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.002s
I20260709 13:10:20.216378 31580 raft_consensus.cc:3055] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:20.214408 31000 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:20.216720 31580 raft_consensus.cc:740] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:20.217231 31000 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:20.217747 31580 consensus_queue.cc:260] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.219014 31580 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:20.213590 31449 raft_consensus.cc:697] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:20.222153 31449 consensus_queue.cc:237] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.231820 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.213255 31537 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.018s
I20260709 13:10:20.213353 31167 consensus_queue.cc:1059] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.023s
W20260709 13:10:20.236778 31611 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
W20260709 13:10:20.236922 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:20.213949 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2530) took 4399 ms. Trace:
I20260709 13:10:20.238317 25246 rpcz_store.cc:276] 0709 13:10:15.814213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:16.755241 (+941028us) service_pool.cc:224] Handling call
0709 13:10:16.755665 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:16.755932 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:10:17.073691 (+317759us) log.cc:844] Serialized 14 byte log entry
0709 13:10:20.213594 (+3139903us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:20.213651 (+    57us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:20.213704 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:20.213775 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:20.213894 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2795724,"prepare.queue_time_us":338415,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"thread_start_us":637802,"threads_started":3,"wal-append.queue_time_us":245267,"wal-append.run_cpu_time_us":1833,"wal-append.run_wall_time_us":1007029}
I20260709 13:10:20.240782 25246 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:20.240743 25237 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:10:20.245436 31167 consensus_queue.cc:1243] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 15.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:10:20.246378 31321 raft_consensus.cc:3055] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:20.246874 31321 raft_consensus.cc:740] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:20.247021 31154 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:20.247761 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2537) took 3823 ms. Trace:
I20260709 13:10:20.247886 31321 consensus_queue.cc:260] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:20.248162 25242 rpcz_store.cc:276] 0709 13:10:16.424328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:18.792948 (+2368620us) service_pool.cc:224] Handling call
0709 13:10:18.812400 (+ 19452us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:20.155548 (+1343148us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:20.155596 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:20.158168 (+  2572us) log.cc:844] Serialized 26 byte log entry
0709 13:10:20.197090 (+ 38922us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:20.197151 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:20.197226 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:20.247543 (+ 50317us) raft_consensus.cc:1685] finished
0709 13:10:20.247564 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:20.247705 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1131189,"thread_start_us":247041,"threads_started":2,"wal-append.queue_time_us":87174}
I20260709 13:10:20.250710 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:20.252096 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:20.252935 25314 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:20.254163 25237 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:20.237844 31000 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.257506 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:20.258553 25941 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:20.260888 25284 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:20.264828 25242 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:10:20.265398 31000 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:20.266268 31321 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:20.266489 25237 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:20.272202 25358 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:20.272588 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:20.273269 25941 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 17
W20260709 13:10:20.275748 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 107169us
I20260709 13:10:20.278079 25237 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 19 index: 9. (index mismatch)
I20260709 13:10:20.279546 31000 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:20.279915 31386 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:20.279978 31000 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:20.280824 31000 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:20.281395 31610 log.cc:927] Time spent T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 13:10:20.283257 25941 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
W20260709 13:10:20.284186 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:20.284758 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:20.284214 31590 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:20.286785 25358 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:20.276124 31585 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:20.303283 25284 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:20.297808 31339 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:20.304638 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.306420 25143 catalog_manager.cc:5697] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 14 to 18, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 18 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:20.313918 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.314677 25239 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:10:20.298928 31577 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:20.319046 25941 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 11. Current term is 12. Ops: []
I20260709 13:10:20.321841 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:20.274220 31272 consensus_queue.cc:1059] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.035s
I20260709 13:10:20.326354 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:20.331950 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:20.327852 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:20.332785 25940 raft_consensus.cc:2410] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 18 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 13:10:20.285773 31590 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.313841 31339 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.341881 31339 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:20.333578 31583 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.012s
I20260709 13:10:20.330381 31537 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:20.345981 31537 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:20.346372 31537 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:20.348229 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.349080 25238 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:20.353183 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:20.353873 25937 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:10:20.353183 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:20.329384 31571 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:20.362661 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:20.330117 31577 raft_consensus.cc:697] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:20.369676 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:20.312662 31419 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 13:10:20.344759 31000 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.362285 31280 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:20.377665 31280 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.380417 31280 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:20.383766 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:20.376734 31000 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:20.384442 25321 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:20.397185 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:20.398090 25937 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:20.399335 31583 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:20.399974 31583 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:20.402510 31583 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:20.402817 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.404018 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.404620 25245 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:20.404700 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:20.381850 31539 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:20.405432 25394 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 12.
I20260709 13:10:20.360981 31459 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.405509 31539 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:20.406415 31539 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:20.406507 31590 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:20.407508 31459 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:20.410162 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:20.410858 25394 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:20.411913 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:20.412690 25937 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 18: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:20.413447 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:20.413993 31583 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:10:20.414268 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:20.414825 25320 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:20.415577 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.405503 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.416152 25238 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:20.416376 25245 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:20.413388 31321 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:10:20.417011 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.417402 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:20.417986 25939 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 18
I20260709 13:10:20.421828 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:20.422717 25239 raft_consensus.cc:2376] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17. Re-sending same reply.
I20260709 13:10:20.423024 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:20.424022 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.424918 25244 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:20.313084 31585 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.427919 25285 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:20.395346 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:20.429116 25285 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:20.429620 25395 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:20.431407 25911 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:20.432708 31459 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:20.433012 31577 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.433748 31459 raft_consensus.cc:697] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 19 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:20.435194 31459 consensus_queue.cc:237] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:20.436307 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:20.437687 31151 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:20.438287 31151 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:20.441334 31419 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:20.442699 31510 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:20.443150 31510 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:20.443543 31510 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:20.444026 31383 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:20.444851 25909 leader_election.cc:336] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 1.681s (SENT)
I20260709 13:10:20.445216 31167 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:10:20.445555 25909 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:20.447160 31571 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:10:20.435716 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 1.46998e+06us
I20260709 13:10:20.451372 25287 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:20.429553 25321 raft_consensus.cc:2393] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 19: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:20.465464 25314 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:10:20.482990 25939 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:20.440481 31539 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.499495 31413 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:20.500114 31413 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.501142 31436 raft_consensus.cc:3055] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:20.501575 31436 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:20.502449 31436 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.503755 31436 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:10:20.507753 31604 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.296s	user 0.002s	sys 0.000s
I20260709 13:10:20.509017 31585 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:20.519044 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:20.520347 25939 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:20.471918 31000 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:20.476300 31419 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:10:20.528550 25938 raft_consensus.cc:606] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:20.529171 25938 raft_consensus.cc:614] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:10:20.538643 25939 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:20.522218 31539 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:20.524046 31000 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:20.540131 31419 raft_consensus.cc:697] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 20 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:20.545369 31000 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:20.550297 31413 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:20.551424 31419 consensus_queue.cc:237] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:20.553804 31401 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:20.554359 31401 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.606343 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2541) took 3918 ms. Trace:
I20260709 13:10:20.610368 25287 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:20.614693 25287 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:20.607697 25243 rpcz_store.cc:276] 0709 13:10:16.687376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:18.810124 (+2122748us) service_pool.cc:224] Handling call
0709 13:10:18.810542 (+   418us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:19.973385 (+1162843us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:19.973432 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:19.974125 (+   693us) log.cc:844] Serialized 52 byte log entry
0709 13:10:20.147519 (+173394us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:10:20.147577 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:20.147673 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:20.507160 (+359487us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:20.604760 (+ 97600us) raft_consensus.cc:1685] finished
0709 13:10:20.604854 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:20.605780 (+   926us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1131278,"spinlock_wait_cycles":3419008,"thread_start_us":202503,"threads_started":2,"wal-append.queue_time_us":207084}
W20260709 13:10:20.620750 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:20.634020 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 143367us
I20260709 13:10:20.639312 31167 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:20.640511 31167 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:20.645969 25267 debug-util.cc:398] Leaking SignalData structure 0x7b080030b180 after lost signal to thread 24116
I20260709 13:10:20.647639 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 13:10:20.648500 31223 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:20.650070 31223 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:20.645704 31095 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:20.650933 31223 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:20.651532 31095 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:20.648998 25267 debug-util.cc:398] Leaking SignalData structure 0x7b08003bc040 after lost signal to thread 25178
I20260709 13:10:20.655874 31223 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:20.656344 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:20.618062 25287 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:20.681926 31340 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:20.665344 31403 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:20.662052 31386 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:10:20.704169 31095 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:20.711337 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:20.712333 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:20.718422 25267 debug-util.cc:398] Leaking SignalData structure 0x7b08002ecd00 after lost signal to thread 25270
W20260709 13:10:20.719552 25267 debug-util.cc:398] Leaking SignalData structure 0x7b08001bfbe0 after lost signal to thread 25271
I20260709 13:10:20.569900 31537 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.730216 31459 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:20.731666 31459 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.740152 31340 raft_consensus.cc:697] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:10:20.740747 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 122980us
I20260709 13:10:20.742573 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:20.709545 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.749660 31585 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:20.743407 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:20.751219 31580 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:20.761953 25939 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 14: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:20.752835 31167 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:20.742578 31340 consensus_queue.cc:237] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:20.772684 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 321759us
I20260709 13:10:20.774328 25209 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:20.684763 31403 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:20.783340 25285 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:20.784564 31403 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:20.786621 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 44324.8us
I20260709 13:10:20.801156 31459 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:20.802850 25395 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:10:20.800923 31353 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:20.805299 31537 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:20.805400 31580 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:20.805826 31353 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.809549 31459 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:10:20.815692 25246 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:20.824141 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:20.825079 25939 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:20.829433 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 55498.8us
I20260709 13:10:20.831379 25209 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:20.835002 31638 log.cc:927] Time spent T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
W20260709 13:10:20.839367 25267 debug-util.cc:398] Leaking SignalData structure 0x7b080018f600 after lost signal to thread 25345
W20260709 13:10:20.839876 25267 debug-util.cc:398] Leaking SignalData structure 0x7b08000481a0 after lost signal to thread 25346
W20260709 13:10:20.841888 25267 debug-util.cc:398] Leaking SignalData structure 0x7b08002e9740 after lost signal to thread 25419
W20260709 13:10:20.844117 25267 debug-util.cc:398] Leaking SignalData structure 0x7b08003d12e0 after lost signal to thread 25969
W20260709 13:10:20.844542 25267 debug-util.cc:398] Leaking SignalData structure 0x7b08003cbd80 after lost signal to thread 25970
W20260709 13:10:20.844911 25267 debug-util.cc:398] Leaking SignalData structure 0x7b08003ac8e0 after lost signal to thread 30884
I20260709 13:10:20.846460 31362 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:20.847708 31363 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:20.848218 31353 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:20.848505 31371 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:20.850221 31377 consensus_queue.cc:1059] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:10:20.865036 31371 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.865478 31362 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:20.874038 31363 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:20.875648 31362 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:20.876566 31363 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:20.865104 31583 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:20.878382 31583 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:20.878894 31583 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:20.890554 31577 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:10:20.891755 31239 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:10:20.894565 31605 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:10:20.895191 25285 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:20.894827 31000 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.892740 31577 raft_consensus.cc:3055] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:20.897522 31537 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:20.897595 31577 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:20.898311 31537 raft_consensus.cc:697] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:20.898173 31605 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:10:20.899473 31577 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:20.899762 31537 consensus_queue.cc:237] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.900139 31095 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:10:20.901445 31401 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:20.913885 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.914746 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.915187 25242 raft_consensus.cc:2410] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 13:10:20.916682 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:20.918161 25243 raft_consensus.cc:2393] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 15: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:20.918229 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.916759 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.920102 25287 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:20.920966 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.921559 31580 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:20.924065 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.927130 31571 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:20.929293 31371 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:20.930321 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:20.928745 31571 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:20.931488 25285 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:20.934031 31478 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:20.965709 25394 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:10:20.966488 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:20.967118 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:20.967284 25243 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 11.
I20260709 13:10:20.928617 31390 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:20.885099 31151 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:20.943073 25267 debug-util.cc:398] Leaking SignalData structure 0x7b08002adac0 after lost signal to thread 31308
I20260709 13:10:20.944594 31272 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:20.945607 25287 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:20.947454 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
W20260709 13:10:20.948455 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 117283us
I20260709 13:10:20.949411 31353 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:20.952987 31395 raft_consensus.cc:3055] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:10:20.987445 31395 raft_consensus.cc:740] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:20.987712 25243 raft_consensus.cc:3055] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:20.988211 25243 raft_consensus.cc:740] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:20.989305 25243 consensus_queue.cc:260] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:20.955209 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.990387 25239 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:20.991477 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.982584 31272 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:20.957074 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:20.992709 31272 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:20.992976 25238 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:20.994982 31353 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:20.995556 31353 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:20.958492 25143 catalog_manager.cc:5697] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 9 to 11, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 11 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:20.960649 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.998801 25240 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:21.000089 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:20.964605 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 61866.9us
I20260709 13:10:20.966259 31478 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:20.948561 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:20.937112 31000 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:20.981340 25267 debug-util.cc:398] Leaking SignalData structure 0x7b08002ca660 after lost signal to thread 31384
I20260709 13:10:20.989054 31395 consensus_queue.cc:260] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:20.992671 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:20.953716 31577 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:21.009521 25943 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:10:21.001633 31643 consensus_queue.cc:1059] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.016s
I20260709 13:10:21.011219 31478 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:21.020563 31449 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:21.020431 31371 consensus_queue.cc:1059] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:21.021528 31167 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:21.021256 31449 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.021942 31167 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:21.022356 31167 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:21.023670 31449 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:20.980014 31390 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:21.026620 31390 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:21.002732 31459 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 13:10:21.031112 31395 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:21.042515 31154 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:10:21.043254 25359 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:21.044238 31340 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:21.046145 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.046330 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:21.046643 25238 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:21.046959 25394 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:21.046692 31154 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.002s
I20260709 13:10:21.048863 25911 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:21.049944 31459 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:21.050446 31459 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:21.046959 25239 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:21.008628 25939 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:21.007954 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.012374 25143 catalog_manager.cc:5697] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 12 to 14, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 14 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:21.053680 25359 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:21.055444 25359 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:21.053655 25240 raft_consensus.cc:2393] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 15: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:21.054752 25143 catalog_manager.cc:5697] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 4 to 10, term changed from 12 to 17, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to <none>, VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) evicted, no longer has a pending config: VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65), VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66), VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 17 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:21.051131 31459 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:21.018455 25320 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:21.063478 25938 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:21.064769 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.065349 31223 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:21.065622 25240 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:21.065913 31326 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:21.066444 31326 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:21.066067 25143 catalog_manager.cc:5697] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 11 to 12, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 12 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:21.066804 31326 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:21.067710 25143 catalog_manager.cc:5697] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:21.068446 31643 raft_consensus.cc:3055] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:21.068871 31643 raft_consensus.cc:740] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:21.065332 25359 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:21.066720 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.072862 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.069252 31154 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:21.062515 31340 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:21.065558 31642 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:21.083122 31340 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:21.094743 25240 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:21.109170 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.107388 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.121507 31449 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:21.122195 31449 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.124526 31449 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:21.111231 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 46027.7us
I20260709 13:10:21.121275 25284 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:21.131340 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 234372us
W20260709 13:10:21.132603 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 187236us
W20260709 13:10:21.134445 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 54152.4us
I20260709 13:10:21.135596 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:21.124557 31340 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:21.137341 25394 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:21.137548 31537 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:21.175784 31537 raft_consensus.cc:697] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:21.176642 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:21.177678 31449 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:21.178143 31449 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:21.178709 31449 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:21.146045 31151 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:21.146576 31436 consensus_meta.cc:287] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.642s	user 0.021s	sys 0.000s
I20260709 13:10:21.147212 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.181154 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.182845 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.184296 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.185818 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.134629 31642 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:21.152215 31510 consensus_meta.cc:287] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.708s	user 0.016s	sys 0.006s
I20260709 13:10:21.159037 25358 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:21.186065 25243 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:21.164305 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.194259 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:21.194834 25241 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:21.196028 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.196805 25241 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:21.198783 31154 raft_consensus.cc:2804] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:21.199792 31154 raft_consensus.cc:697] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:21.205431 31154 consensus_queue.cc:237] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.110419 31353 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.164994 31583 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.165583 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.167173 31571 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:21.168812 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:21.177069 31537 consensus_queue.cc:237] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.138193 31643 consensus_queue.cc:260] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.140069 31272 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.151063 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.192543 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:21.204566 25143 catalog_manager.cc:5697] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 12 to 17. New cstate: current_term: 17 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:21.146095 25359 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:21.191861 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.224105 25285 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:21.235126 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:21.233323 31362 consensus_queue.cc:1059] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.005s
I20260709 13:10:21.236431 25285 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:21.189379 31510 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.239408 31510 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:21.191614 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:21.240106 31583 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:21.188848 31642 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:21.242003 31643 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:10:21.235925 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 89916.3us
I20260709 13:10:21.244308 31588 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:21.244827 31588 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:21.244903 25359 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:21.245270 31223 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:21.245769 31223 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:21.245885 25237 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:21.246280 31223 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:21.246563 25359 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:21.246982 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.247627 25237 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:21.248051 31239 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:10:21.193190 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:21.248553 31239 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:21.249060 31239 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:10:21.249303 25390 raft_consensus.cc:2410] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 20 because replica has last-logged OpId of term: 19 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 6.
I20260709 13:10:21.252704 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.255820 25237 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:21.191720 31371 consensus_queue.cc:1059] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20260709 13:10:21.235954 25386 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:21.259366 25938 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:21.262305 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.264120 31371 raft_consensus.cc:3055] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:21.264627 31371 raft_consensus.cc:740] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:21.264748 31577 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:21.265358 31510 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:21.265799 31371 consensus_queue.cc:260] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.266132 31510 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.267176 31371 raft_consensus.cc:3060] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:21.229725 31580 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:21.269590 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.270423 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.245263 31588 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:21.276376 25243 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 11. (term mismatch)
I20260709 13:10:21.275859 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:21.278330 25390 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:21.278584 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.278134 25243 pending_rounds.cc:85] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 10
I20260709 13:10:21.280206 25238 raft_consensus.cc:2393] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 17: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:21.288487 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:21.292263 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:21.279312 25243 pending_rounds.cc:107] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 18.11
W20260709 13:10:21.288853 25909 leader_election.cc:336] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 2.124s (SENT)
I20260709 13:10:21.299417 25243 raft_consensus.cc:2889] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:21.304054 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.302927 25909 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:21.304177 25240 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
W20260709 13:10:21.305454 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 58989.7us
I20260709 13:10:21.272889 31459 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.312393 25239 raft_consensus.cc:2368] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17.
I20260709 13:10:21.314113 25391 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:21.274890 31167 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.316512 25359 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:21.318588 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.327601 25241 raft_consensus.cc:2410] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
W20260709 13:10:21.329077 31648 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260709 13:10:21.329607 31539 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:21.330324 31459 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:21.312430 31095 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:21.332058 25238 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:21.335809 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.309648 31571 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:21.349061 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.351457 25239 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:21.361685 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.332439 31000 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:21.315341 31580 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:21.365840 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 110910us
I20260709 13:10:21.365815 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.348274 31095 raft_consensus.cc:697] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:21.379431 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.381287 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.382099 25238 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:21.383335 25388 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 14. Ops: []
I20260709 13:10:21.384382 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.385205 25244 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:21.388032 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.388821 25240 raft_consensus.cc:3060] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:21.393141 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 86900.4us
I20260709 13:10:21.364081 31390 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.373046 31510 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:21.376729 31272 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:21.402513 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:21.382442 31353 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:21.409550 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.411580 25387 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:21.418640 25212 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:21.419427 31661 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:21.420241 31661 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.422528 31661 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:21.425331 31395 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:21.425822 31395 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:21.426213 31395 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:10:21.419513 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 94627us
I20260709 13:10:21.419041 31095 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.441677 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:21.441876 25244 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:21.446429 25238 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:21.448050 25320 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:21.389463 31326 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.454993 25314 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:10:21.455260 31326 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:21.456604 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:21.473510 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:21.475024 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:21.457408 31390 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:21.462167 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:21.467034 31167 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:21.472391 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:21.473330 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:21.471355 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:21.474969 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:21.487200 25237 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
W20260709 13:10:21.496515 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:21.496464 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.497200 25237 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:21.473774 31223 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:21.480759 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 69576.7us
I20260709 13:10:21.490599 25390 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 19.
I20260709 13:10:21.501433 25387 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:21.501350 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.495374 25314 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:21.506371 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:21.508206 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:21.508884 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:21.512604 25313 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:21.513384 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.514099 25244 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:21.520380 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.533978 25238 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:21.535509 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.538304 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:21.539216 25320 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:21.542022 25238 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:21.542553 25318 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:21.545400 25359 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:21.491416 31436 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
W20260709 13:10:21.551014 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:21.551743 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.555589 25237 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:21.556921 31654 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:21.557219 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.557327 31450 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:21.558463 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.543190 31353 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:21.559477 25237 raft_consensus.cc:2368] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17.
I20260709 13:10:21.550025 25321 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:10:21.557682 31654 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.541003 31239 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.558001 31450 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.564677 25285 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:21.565850 31670 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:21.566349 31670 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:21.566721 31670 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:21.566768 31239 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:21.584594 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:21.585650 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:21.586704 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 48611.7us
I20260709 13:10:21.587648 25359 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:21.588685 31403 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:21.589344 31403 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:21.589582 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:21.589972 31403 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:21.590505 25313 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 16: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:21.566831 31450 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:21.593271 31223 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:21.594777 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 138340us
I20260709 13:10:21.595774 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:21.586726 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.597265 25241 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:21.601647 31580 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:21.602540 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.597795 31362 raft_consensus.cc:2764] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election decision vote started in defunct term 17: won
I20260709 13:10:21.603257 25241 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:21.608093 31588 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.600561 25359 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:21.612726 31580 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:21.618340 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:21.619941 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:21.620690 25314 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:21.622898 31577 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:21.631268 25244 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:21.633652 25285 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:21.640353 25240 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:21.640702 25314 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:21.642187 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.642882 25237 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:21.645416 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:21.653162 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.653409 25241 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:21.655531 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.656594 25244 raft_consensus.cc:2393] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 19: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:21.651211 31450 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:21.658768 31450 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.659658 31403 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.661504 25209 leader_election.cc:400] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17.
I20260709 13:10:21.661965 25209 leader_election.cc:403] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 13:10:21.662639 31643 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17.
I20260709 13:10:21.663506 25209 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:21.664788 31403 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:21.665992 31649 log.cc:927] Time spent T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.407s	user 0.004s	sys 0.000s
I20260709 13:10:21.637260 31654 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:21.664402 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.654637 25285 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:21.675782 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.677376 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.677975 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.678732 25241 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:21.647365 31223 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:21.666141 31670 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.697604 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:21.698359 31670 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:21.685669 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:21.690815 25908 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:21.700886 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 53743us
I20260709 13:10:21.707255 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:21.708182 25321 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:21.703261 25908 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:21.713316 25241 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:21.709599 31223 raft_consensus.cc:697] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:21.716109 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.715934 31223 consensus_queue.cc:237] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.716902 25239 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 17: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:21.677564 31390 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:10:21.677363 31583 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:21.718428 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.718060 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.712842 25318 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:21.723212 25939 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:21.726943 25241 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:10:21.727275 25239 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:21.727176 25238 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:21.729197 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:21.717873 31571 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:21.730006 25313 raft_consensus.cc:2393] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 15: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:21.729948 31571 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:21.730508 31571 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:21.731257 25285 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:21.732383 25285 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:21.738274 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.739153 25244 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:21.725446 25242 raft_consensus.cc:2376] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17. Re-sending same reply.
I20260709 13:10:21.745004 25241 raft_consensus.cc:1275] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260709 13:10:21.748169 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 84779.1us
I20260709 13:10:21.726155 31459 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:21.750452 25237 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:10:21.751914 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 255580us
W20260709 13:10:21.781507 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 49222.4us
I20260709 13:10:21.718088 31654 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 13:10:21.745291 31670 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:21.785099 25945 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:10:21.755617 31450 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:21.786070 25319 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:10:21.787467 25212 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:21.787648 31577 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:21.788393 31353 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:21.788368 31577 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.789167 31353 raft_consensus.cc:697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 19 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:21.790092 31353 consensus_queue.cc:237] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.791013 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:21.791215 31577 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:21.791932 31577 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:21.792518 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:21.793361 31537 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:21.793442 31670 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:21.794193 31537 raft_consensus.cc:697] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:21.794691 31670 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.795447 31537 consensus_queue.cc:237] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.757908 31660 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:21.793334 31436 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:21.799265 31660 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.799576 31436 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.802477 31478 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:21.802799 31436 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:21.806392 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:21.793311 25944 raft_consensus.cc:2368] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19.
I20260709 13:10:21.752827 31583 raft_consensus.cc:697] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:21.810410 31583 consensus_queue.cc:237] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.753640 31459 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:21.805008 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:21.818900 31459 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:21.826699 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:21.826941 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:21.827991 25241 raft_consensus.cc:2393] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 17: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:21.792218 31340 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:21.829612 31340 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.831935 31340 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:21.833284 31154 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:21.833403 31395 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:21.833841 31154 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.835418 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:21.835633 31395 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:21.807569 31449 consensus_meta.cc:287] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.576s	user 0.012s	sys 0.000s
I20260709 13:10:21.836109 25390 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:21.836438 31449 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.833841 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:21.792611 31577 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.835920 31154 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:21.838488 25939 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:21.843304 25237 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 12.
I20260709 13:10:21.844810 25941 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 20: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:21.845476 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:21.828949 31390 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:10:21.851099 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 88763.7us
I20260709 13:10:21.852653 25391 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 11. Current term is 12. Ops: []
I20260709 13:10:21.854651 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:21.852106 25944 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:21.803022 31478 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.858747 25284 leader_election.cc:400] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19.
I20260709 13:10:21.859153 25284 leader_election.cc:403] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 13:10:21.876435 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:21.877313 25240 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:21.857327 31371 consensus_meta.cc:287] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.584s	user 0.009s	sys 0.003s
I20260709 13:10:21.881188 25938 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:10:21.839334 31661 consensus_queue.cc:1059] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.030s
I20260709 13:10:21.822000 31436 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 13:10:21.886008 25243 raft_consensus.cc:2368] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15.
I20260709 13:10:21.893880 25244 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:21.878623 31588 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:21.898191 25244 raft_consensus.cc:3055] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:10:21.924883 25244 raft_consensus.cc:740] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:21.933306 31326 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:21.943104 25209 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:21.949874 25321 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:21.952028 25939 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:21.929760 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:21.961525 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:21.961894 31395 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:21.962368 31395 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:21.962783 31395 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:21.970510 25209 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:21.972422 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:21.973285 31362 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:21.973950 31362 raft_consensus.cc:2764] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 13:10:21.975381 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:21.976099 31151 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:21.988873 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2572) took 1028 ms. Trace:
W20260709 13:10:22.011106 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 88562.6us
I20260709 13:10:22.014856 31419 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19.
I20260709 13:10:22.050446 25240 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:21.979954 25244 consensus_queue.cc:260] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:22.084841 25244 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 14
I20260709 13:10:21.980582 31326 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:21.989641 25245 rpcz_store.cc:276] 0709 13:10:20.950784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:20.961200 (+ 10416us) service_pool.cc:224] Handling call
0709 13:10:20.963989 (+  2789us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:21.064504 (+100515us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:10:21.207322 (+142818us) log.cc:844] Serialized 14 byte log entry
0709 13:10:21.736544 (+529222us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:21.801309 (+ 64765us) log.cc:844] Serialized 26 byte log entry
0709 13:10:21.801474 (+   165us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:21.801741 (+   267us) log.cc:844] Serialized 14 byte log entry
0709 13:10:21.801940 (+   199us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:21.802060 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:21.979282 (+177222us) raft_consensus.cc:1685] finished
0709 13:10:21.979311 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:21.979499 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44553,"prepare.queue_time_us":159413,"prepare.run_cpu_time_us":127,"prepare.run_wall_time_us":2145,"raft.queue_time_us":49088,"raft.run_cpu_time_us":411,"raft.run_wall_time_us":9957,"thread_start_us":591945,"threads_started":2,"wal-append.queue_time_us":623569}
I20260709 13:10:22.089955 31154 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:22.090785 31154 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:22.093945 25245 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:22.098737 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:22.099386 25386 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:22.103740 25358 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:22.106501 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:22.107307 25387 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:22.066219 31095 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:22.011894 31459 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:22.109930 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:22.086400 31371 consensus_queue.cc:1175] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f646dfd532974dceadfe00005d2bbf59" responder_term: 13 status { last_received { term: 12 index: 11 } last_committed_idx: 9 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 12. Current term is 13. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:10:22.115653 31577 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:22.141810 31154 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:22.142925 31326 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:22.143213 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:22.143375 31459 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:22.110641 31095 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:22.144837 31095 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:22.148000 31571 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:22.149520 31395 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:22.152348 31395 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:22.115797 31449 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:22.154428 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:22.155397 25395 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 13:10:22.157528 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:22.158146 25388 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:22.158563 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:22.136618 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2571) took 1186 ms. Trace:
I20260709 13:10:22.163961 25386 raft_consensus.cc:2368] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19.
I20260709 13:10:22.168429 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:22.160179 25246 rpcz_store.cc:276] 0709 13:10:20.950443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:20.961801 (+ 11358us) service_pool.cc:224] Handling call
0709 13:10:20.962198 (+   397us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:20.962784 (+   586us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:10:20.963470 (+   686us) log.cc:844] Serialized 14 byte log entry
0709 13:10:21.281388 (+317918us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:21.301435 (+ 20047us) log.cc:844] Serialized 26 byte log entry
0709 13:10:21.301610 (+   175us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:21.312440 (+ 10830us) log.cc:844] Serialized 14 byte log entry
0709 13:10:21.316396 (+  3956us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:21.316479 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:21.826718 (+510239us) spinlock_profiling.cc:243] Waited 1.9 ms on lock 0x7b1800431f20. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc5522 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:10:22.134079 (+307361us) spinlock_profiling.cc:243] Waited 4.93 ms on lock 0x7b1800431f20. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:10:22.136391 (+  2312us) raft_consensus.cc:1685] finished
0709 13:10:22.136416 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:22.136553 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10529,"prepare.queue_time_us":12856,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":79,"raft.queue_time_us":9918,"raft.run_cpu_time_us":512,"raft.run_wall_time_us":8858,"spinlock_wait_cycles":15702656,"thread_start_us":316898,"threads_started":1,"wal-append.queue_time_us":361125}
I20260709 13:10:22.111694 31674 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:10:22.170818 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 310095us
I20260709 13:10:22.173933 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:22.174630 25387 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:22.175750 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:22.176415 25394 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:22.138489 31667 raft_consensus.cc:3055] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:22.179914 25391 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 13:10:22.192519 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:22.198585 25313 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:22.199102 25318 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:22.203209 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:22.205933 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:22.206691 25394 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:22.219123 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:22.220793 25246 raft_consensus.cc:2368] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18.
I20260709 13:10:22.225244 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:22.226001 25393 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:22.229846 31571 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:22.230207 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:22.183758 31667 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:10:22.231926 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 67191us
I20260709 13:10:22.239132 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:22.229830 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:22.247900 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:22.249059 25320 raft_consensus.cc:2410] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 13:10:22.247203 25387 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:22.252964 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:22.253424 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:10:22.256623 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 398932us
I20260709 13:10:22.258888 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:22.260638 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:22.279300 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:22.280617 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:22.281270 25246 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:22.284683 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:22.287158 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:22.294816 31669 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.492s	user 0.002s	sys 0.000s
I20260709 13:10:22.296844 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:22.306463 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 1084) took 1285 ms. Trace:
W20260709 13:10:22.307286 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 74342.5us
I20260709 13:10:22.307070 25243 rpcz_store.cc:276] 0709 13:10:21.022070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:21.325430 (+303360us) service_pool.cc:224] Handling call
0709 13:10:21.473335 (+147905us) spinlock_profiling.cc:243] Waited 2.05 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:10:22.306146 (+832811us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b5400192654. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:22.306379 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":917756416}
I20260709 13:10:22.309890 25244 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:10:22.290992 25358 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:22.312096 25388 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:22.314205 25393 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:22.315811 25359 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:22.303040 25285 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:22.328364 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:22.329195 25244 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 13:10:22.331403 25359 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:22.333758 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:22.328583 25285 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:22.337610 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:22.340788 25244 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 13:10:22.341310 31670 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:22.306515 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:22.354161 25315 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:22.345935 31537 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:22.338363 31605 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:22.348333 25239 raft_consensus.cc:1275] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 11. (term mismatch)
I20260709 13:10:22.358938 25239 pending_rounds.cc:85] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 10
I20260709 13:10:22.359244 25239 pending_rounds.cc:107] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260709 13:10:22.359535 25239 raft_consensus.cc:2889] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:10:22.351085 25909 leader_election.cc:336] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 1.614s (SENT)
I20260709 13:10:22.361272 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:22.362380 25394 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
W20260709 13:10:22.365590 25911 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.019s	sys 0.031s
W20260709 13:10:22.366530 25911 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.019s	sys 0.031s
W20260709 13:10:22.366863 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 529746us
I20260709 13:10:22.367532 25911 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:22.368340 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:22.382406 25911 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:22.362300 31670 raft_consensus.cc:697] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:22.363696 31537 raft_consensus.cc:697] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:22.388093 25240 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:10:22.358526 31403 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
I20260709 13:10:22.349046 31667 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:22.390683 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:22.425144 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 45066.3us
I20260709 13:10:22.434363 25318 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 15: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:22.434909 25394 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 13:10:22.442461 25245 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:22.445007 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:22.388805 31675 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:22.458855 31571 consensus_queue.cc:1059] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:22.396272 31688 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:22.397266 31670 consensus_queue.cc:237] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:22.465445 25240 raft_consensus.cc:2393] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 18: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:22.465502 31571 raft_consensus.cc:3055] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:10:22.466089 31571 raft_consensus.cc:740] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:22.466997 31571 consensus_queue.cc:260] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:22.378669 31154 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.077s
I20260709 13:10:22.468380 31571 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:10:22.469566 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:22.474787 25315 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
W20260709 13:10:22.485911 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 172909us
I20260709 13:10:22.486474 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2622) took 1495 ms. Trace:
W20260709 13:10:22.486650 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:22.486855 25239 rpcz_store.cc:276] 0709 13:10:20.991283 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:21.737280 (+745997us) service_pool.cc:224] Handling call
0709 13:10:22.308437 (+571157us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:22.364203 (+ 55766us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:22.486268 (+122065us) spinlock_profiling.cc:243] Waited 565 ms on lock 0x7b5400192650. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:22.486410 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1298772864}
I20260709 13:10:22.488152 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:22.490417 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:22.491053 25314 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:22.491361 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:22.494374 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:22.497578 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:22.500587 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:22.501875 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:22.547757 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:22.551915 25317 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14. Ops: []
W20260709 13:10:22.555564 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:22.557494 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:22.558568 25242 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 11. Current term is 12. Ops: []
I20260709 13:10:22.428821 31654 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:22.560962 25945 raft_consensus.cc:1240] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:22.385555 31167 consensus_queue.cc:1059] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 7, Time since last communication: 0.059s
I20260709 13:10:22.561905 25287 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:22.562131 25313 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 20.
I20260709 13:10:22.396777 31680 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:22.583189 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:22.583909 25314 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:10:22.585906 25941 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:10:22.586848 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:22.587874 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:22.589147 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:22.589861 25386 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:22.591017 25387 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 11 FOLLOWER]: Advancing to term 13
W20260709 13:10:22.592039 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 205951us
I20260709 13:10:22.592432 25313 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:10:22.592065 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:22.594802 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:22.595513 25939 raft_consensus.cc:2368] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 12. Current term is 13.
I20260709 13:10:22.504379 31660 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:22.642148 25245 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:22.647667 25315 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:22.577005 31654 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:22.425858 31537 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:22.681406 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 89236.2us
I20260709 13:10:22.704425 25387 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:10:22.647598 31167 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:10:22.662243 31654 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:22.720400 25245 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:10:22.730542 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2593) took 1025 ms. Trace:
I20260709 13:10:22.741721 25245 rpcz_store.cc:276] 0709 13:10:21.705468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:22.690064 (+984596us) service_pool.cc:224] Handling call
0709 13:10:22.713947 (+ 23883us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:22.721034 (+  7087us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:22.730442 (+  9408us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:22.690698 31688 raft_consensus.cc:697] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 19 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:10:22.745329 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 152445us
I20260709 13:10:22.766260 25359 leader_election.cc:400] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18.
I20260709 13:10:22.766774 25359 leader_election.cc:403] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 13:10:22.770705 25359 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:22.773928 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 212331us
I20260709 13:10:22.775269 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:22.780570 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:22.781327 31272 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:10:22.782589 25941 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:10:22.784269 25945 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 13:10:22.785022 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:22.785174 31000 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:10:22.774068 31577 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:22.786098 31577 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:22.786532 31577 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:10:22.787168 31683 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
W20260709 13:10:22.788638 31695 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.212s	user 0.000s	sys 0.001s
I20260709 13:10:22.790472 25245 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 11. (term mismatch)
I20260709 13:10:22.790980 25245 pending_rounds.cc:85] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 10
I20260709 13:10:22.791252 25245 pending_rounds.cc:107] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 19.11
I20260709 13:10:22.791527 25245 raft_consensus.cc:2889] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:22.792119 31580 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:10:22.792421 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2594) took 1086 ms. Trace:
I20260709 13:10:22.785877 31605 consensus_queue.cc:1059] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20260709 13:10:22.798225 31674 consensus_meta.cc:287] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.599s	user 0.009s	sys 0.001s
I20260709 13:10:22.798733 31674 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:22.777473 31154 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 13:10:22.807003 31688 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:22.809327 31450 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:22.811513 31450 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:10:22.817413 31095 consensus_meta.cc:287] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.672s	user 0.013s	sys 0.002s
I20260709 13:10:22.818056 31095 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:22.792965 25245 rpcz_store.cc:276] 0709 13:10:21.705820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:22.760433 (+1054613us) service_pool.cc:224] Handling call
0709 13:10:22.789973 (+ 29540us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:22.792113 (+  2140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:22.792353 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:10:22.774664 31679 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18.
I20260709 13:10:22.818992 25319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2773) took 1175 ms. Trace:
I20260709 13:10:22.820415 31095 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:22.820456 25319 rpcz_store.cc:276] 0709 13:10:21.619016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:21.622459 (+  3443us) service_pool.cc:224] Handling call
0709 13:10:21.726377 (+103918us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:21.786451 (+ 60074us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:22.776285 (+989834us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54000f9e50. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:22.794029 (+ 17744us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":238246784}
W20260709 13:10:22.819691 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 55921.8us
I20260709 13:10:22.822371 25245 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:10:22.822661 25909 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:22.823585 25245 raft_consensus.cc:1240] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:22.824359 25909 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:22.824870 25245 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:10:22.825263 31711 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:22.825721 31711 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:22.826130 31239 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:22.827011 31691 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:10:22.826905 31239 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:22.828119 25287 leader_election.cc:400] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19.
I20260709 13:10:22.829083 25287 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:22.829536 31239 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:22.830155 31239 raft_consensus.cc:3055] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:10:22.830442 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:22.830549 31239 raft_consensus.cc:740] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:22.832038 31662 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:22.831763 31239 consensus_queue.cc:260] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:22.827143 31478 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:22.833344 31239 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:22.834055 31605 raft_consensus.cc:3055] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:22.834488 31605 raft_consensus.cc:740] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:22.835264 31605 consensus_queue.cc:260] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:22.837471 31583 raft_consensus.cc:2764] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 13:10:22.838110 25909 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:22.839640 25909 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:22.840219 31675 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:22.840456 31571 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:22.840782 31675 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:22.841296 31571 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:22.826090 31711 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:22.832974 31654 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:22.851264 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 1082) took 1850 ms. Trace:
I20260709 13:10:22.851452 31654 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:22.851720 25242 rpcz_store.cc:276] 0709 13:10:21.000911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:21.743412 (+742501us) service_pool.cc:224] Handling call
0709 13:10:22.554041 (+810629us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:22.573795 (+ 19754us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:22.851040 (+277245us) spinlock_profiling.cc:243] Waited 798 ms on lock 0x7b54001f63d0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:22.851198 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1836521344}
I20260709 13:10:22.850251 25287 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:22.826285 31580 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:22.840993 25909 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:22.774026 31321 raft_consensus.cc:3055] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:22.862751 25908 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:22.872881 25908 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:22.877437 31654 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:10:22.877580 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:22.877844 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:22.878422 25315 raft_consensus.cc:2393] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 15: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:22.866904 31436 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:22.878963 31691 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:10:22.875264 31571 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 13:10:22.886596 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 45674.6us
I20260709 13:10:22.887665 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:22.888616 31691 raft_consensus.cc:697] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:22.890098 25909 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:22.870515 31321 raft_consensus.cc:740] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:22.892506 25386 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 20.
I20260709 13:10:22.850260 31326 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 13:10:22.906495 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:22.894119 31571 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:22.908066 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:22.898737 31436 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:22.871269 31580 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:22.915879 25237 raft_consensus.cc:1275] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:10:22.916814 25393 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:10:22.917832 25388 raft_consensus.cc:1275] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:10:22.918251 25387 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:22.918793 25390 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:10:22.921574 25395 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:10:22.922542 25239 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 13:10:22.928781 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:22.931227 31675 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:22.937459 31510 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:22.938057 31510 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:22.926043 31436 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:22.940526 31510 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:22.940836 31436 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:22.934975 31690 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.539s	user 0.004s	sys 0.000s
I20260709 13:10:22.936451 31605 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:22.932415 31667 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:22.926180 31450 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:22.943641 25394 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:22.953485 31577 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:22.928699 25285 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:22.910982 31571 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:22.965179 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 331408us
I20260709 13:10:22.974937 25358 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:22.976459 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:22.980485 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 51805.2us
I20260709 13:10:22.989586 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:22.144315 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:10:22.973557 31571 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:22.998162 25391 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 13:10:22.965188 31585 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 13:10:22.984371 31095 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:22.992121 31688 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:22.995011 31588 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:22.997704 31710 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:22.999671 31436 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:23.027678 25315 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 13:10:23.034963 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:23.030542 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 141227us
I20260709 13:10:23.017894 31326 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.028s
W20260709 13:10:23.053834 31706 log.cc:927] Time spent T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.154s	user 0.000s	sys 0.001s
I20260709 13:10:23.030913 31654 raft_consensus.cc:697] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 20 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:22.986233 31691 consensus_queue.cc:237] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:23.043421 31719 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.004s
I20260709 13:10:23.071725 31716 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.053s
I20260709 13:10:23.072286 31711 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:23.045557 25909 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:23.030408 31661 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.017s
I20260709 13:10:23.045485 31419 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:23.086704 31419 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:23.087469 31711 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:23.088013 31577 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:23.023744 31588 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:23.088763 31419 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:23.041337 31718 consensus_queue.cc:1059] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.011s
I20260709 13:10:23.067694 31510 consensus_queue.cc:1048] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:10:23.028163 31154 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:23.089845 31154 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:23.090233 31718 raft_consensus.cc:3055] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:23.090809 31718 raft_consensus.cc:740] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:23.091898 31154 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:23.091825 31718 consensus_queue.cc:260] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:23.092553 31711 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:10:23.092881 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 56224.3us
I20260709 13:10:23.093734 25312 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:23.073858 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:23.031495 31710 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:23.073020 31654 consensus_queue.cc:237] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:23.096098 31710 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:23.040788 31438 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:23.094282 25909 leader_election.cc:400] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17.
I20260709 13:10:23.078593 31095 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:10:23.097020 25909 leader_election.cc:403] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:10:23.099104 31727 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17.
I20260709 13:10:23.100751 25909 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:23.101056 31662 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:23.097040 31438 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:23.074275 31436 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:23.081393 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:23.041998 31658 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 13:10:23.093093 31711 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:23.017707 31660 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 13:10:23.079726 25358 leader_election.cc:400] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 12. Current term is 13.
W20260709 13:10:23.104651 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 195149us
W20260709 13:10:23.124045 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 230907us
I20260709 13:10:23.124900 25358 leader_election.cc:403] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:10:23.089023 31588 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:10:23.137981 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 58412.7us
I20260709 13:10:23.140079 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:23.143044 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:23.143852 25312 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:23.027508 31688 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:23.129766 31711 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:23.129657 31727 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:23.157927 31727 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:23.158381 31727 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:23.125535 31718 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:23.166510 25909 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:23.144536 31719 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 12. Current term is 13.
I20260709 13:10:23.169413 31729 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:23.169916 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:23.170085 31729 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:23.174117 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:23.178496 31095 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:10:23.181078 31577 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:10:23.182829 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:23.183614 31436 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:23.184036 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:23.181389 31438 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:23.184489 31353 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:23.185030 31353 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:23.187196 31353 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:23.177655 31719 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:23.190613 31675 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:23.191388 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 48285.9us
I20260709 13:10:23.191560 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:23.192186 25237 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:23.192039 31321 consensus_queue.cc:260] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:23.193092 31723 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:23.193197 31321 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:23.193615 31723 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:23.194108 31723 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:23.194028 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:23.195147 31450 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:23.196749 31395 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:23.197636 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2630) took 1573 ms. Trace:
I20260709 13:10:23.197882 25388 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:10:23.196996 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2631) took 1571 ms. Trace:
I20260709 13:10:23.198942 25238 rpcz_store.cc:276] 0709 13:10:21.624693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:22.122988 (+498295us) service_pool.cc:224] Handling call
0709 13:10:22.123330 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:22.159537 (+ 36207us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:10:22.449332 (+289795us) log.cc:844] Serialized 14 byte log entry
0709 13:10:23.197322 (+747990us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:23.197386 (+    64us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:23.197427 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:23.197487 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:23.197588 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":501944,"prepare.queue_time_us":316781,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"thread_start_us":482273,"threads_started":2,"wal-append.queue_time_us":345067}
I20260709 13:10:23.193471 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:23.178376 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:23.206348 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:23.212005 25209 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:23.217028 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 770634us
I20260709 13:10:23.202293 25246 rpcz_store.cc:276] 0709 13:10:21.625157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:22.281990 (+656833us) service_pool.cc:224] Handling call
0709 13:10:22.282367 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:22.282820 (+   453us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:22.282861 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:22.338380 (+ 55519us) log.cc:844] Serialized 26 byte log entry
0709 13:10:22.587333 (+248953us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:10:22.660712 (+ 73379us) log.cc:844] Serialized 14 byte log entry
0709 13:10:23.196685 (+535973us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:23.196759 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:23.196799 (+    40us) raft_consensus.cc:1685] finished
0709 13:10:23.196817 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:23.196939 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":551254,"prepare.queue_time_us":101829,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":4779,"raft.queue_time_us":540498,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":14030,"thread_start_us":218522,"threads_started":1,"wal-append.queue_time_us":414571}
I20260709 13:10:23.183744 31571 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:23.227361 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 49069.4us
I20260709 13:10:23.227521 25941 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:10:23.225610 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:23.226737 25945 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:23.238442 25209 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:23.229928 31719 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:23.229343 31395 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:23.248886 25209 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:23.251055 25945 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:23.252586 25389 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:23.253649 31733 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:23.254300 31733 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:23.257846 31688 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:23.260083 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:23.248184 31654 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:23.261675 31654 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:23.263954 31730 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:23.264536 31730 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:23.205293 31478 consensus_queue.cc:1059] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20260709 13:10:23.247330 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:23.268477 25143 catalog_manager.cc:5697] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 7 to 15, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 15 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:23.269555 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:23.274583 25359 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:23.243455 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2582) took 1998 ms. Trace:
I20260709 13:10:23.283979 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:23.285336 25315 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:23.288448 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2633) took 1662 ms. Trace:
I20260709 13:10:23.291404 25312 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:23.296978 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:23.300561 25242 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 17: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:23.280313 25241 rpcz_store.cc:276] 0709 13:10:21.244303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:21.828571 (+584268us) service_pool.cc:224] Handling call
0709 13:10:21.828865 (+   294us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:21.829069 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:21.829094 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:21.840664 (+ 11570us) log.cc:844] Serialized 26 byte log entry
0709 13:10:22.172862 (+332198us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:22.370514 (+197652us) log.cc:844] Serialized 14 byte log entry
0709 13:10:23.183808 (+813294us) spinlock_profiling.cc:243] Waited 2.49 ms on lock 0x7b1800474700. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b3fc0a 00007f297d27f219 00007f2973bc7a40 00007f2973bc24e4 00007f2973bc2455 00007f2976caa060 00007f2976cbc716 00007f2976ccd287 00007f2976ccd208
0709 13:10:23.238056 (+ 54248us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:23.238124 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:23.238164 (+    40us) raft_consensus.cc:1685] finished
0709 13:10:23.243020 (+  4856us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:23.243160 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31748,"prepare.queue_time_us":309533,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":816960,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"spinlock_wait_cycles":5718272,"thread_start_us":515664,"threads_started":3,"wal-append.queue_time_us":528440}
I20260709 13:10:23.304615 31571 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:23.286410 31605 consensus_queue.cc:1048] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.019s
I20260709 13:10:23.286520 31577 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:10:23.308347 31711 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:23.309327 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 60522.4us
I20260709 13:10:23.310117 25209 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:23.311107 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:23.313446 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:23.308625 31395 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:23.289466 25243 rpcz_store.cc:276] 0709 13:10:21.626006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:22.313968 (+687962us) service_pool.cc:224] Handling call
0709 13:10:22.339278 (+ 25310us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:22.339715 (+   437us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:22.339749 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:22.340515 (+   766us) log.cc:844] Serialized 26 byte log entry
0709 13:10:22.594579 (+254064us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:22.697336 (+102757us) log.cc:844] Serialized 14 byte log entry
0709 13:10:23.288041 (+590705us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:23.288144 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:23.288203 (+    59us) raft_consensus.cc:1685] finished
0709 13:10:23.288223 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:23.288374 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":584617,"prepare.queue_time_us":114804,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":55,"raft.queue_time_us":520396,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":4736,"spinlock_wait_cycles":583552,"thread_start_us":343565,"threads_started":2,"wal-append.queue_time_us":378304}
I20260709 13:10:23.319737 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:23.314196 31588 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:23.274856 31733 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:23.286355 31390 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:23.329854 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:23.327684 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2635) took 1701 ms. Trace:
I20260709 13:10:23.328310 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:23.273777 31643 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.055s
I20260709 13:10:23.333494 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:23.315007 31154 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:23.342502 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:23.343709 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 307227us
I20260709 13:10:23.311065 31729 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:23.325361 31362 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:23.330968 25244 rpcz_store.cc:276] 0709 13:10:21.626638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:22.350159 (+723521us) service_pool.cc:224] Handling call
0709 13:10:22.350486 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:22.370886 (+ 20400us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:22.370933 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:22.379972 (+  9039us) log.cc:844] Serialized 26 byte log entry
0709 13:10:22.715068 (+335096us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:22.754680 (+ 39612us) log.cc:844] Serialized 14 byte log entry
0709 13:10:23.256879 (+502199us) spinlock_profiling.cc:243] Waited 1.98 ms on lock 0x7b4800080058. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297bbdbaa1 00007f297bbd2b37 00007f297bbd29ca 00007f297bcca26f 00007f297bcc9f37 00007f297bcde55c 00007f297bcde508 00007f297bcde488 00007f297bcde443 00007f297bcdd6ec 00007f297ac07a96
0709 13:10:23.323509 (+ 66630us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:23.323617 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:23.323673 (+    56us) raft_consensus.cc:1685] finished
0709 13:10:23.327325 (+  3652us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:23.327500 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1268119,"prepare.queue_time_us":43417,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":1104,"raft.queue_time_us":464152,"raft.run_cpu_time_us":3005,"raft.run_wall_time_us":48261,"spinlock_wait_cycles":4553600,"thread_start_us":442841,"threads_started":2,"wal-append.queue_time_us":427732}
I20260709 13:10:23.329492 31151 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:23.327049 31577 raft_consensus.cc:697] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:23.326718 31478 consensus_queue.cc:1243] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer 2329e428ebbf484fa3ea9452133df567 log is divergent from this leader: its last log entry 14.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:10:23.378690 25241 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:23.380632 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:23.382424 25320 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 19.
I20260709 13:10:23.384373 25245 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:10:23.340406 31716 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 13:10:23.402951 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:23.405170 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:23.405824 25243 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:23.370685 31675 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:23.346058 31679 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 13:10:23.376283 31478 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:23.359872 31658 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 13:10:23.376086 31395 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:23.353587 31154 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:23.419305 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:23.426854 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:23.370823 31151 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:23.434609 31721 consensus_queue.cc:1059] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.075s
I20260709 13:10:23.436401 31721 raft_consensus.cc:3055] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:23.425554 31658 consensus_queue.cc:1243] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer f646dfd532974dceadfe00005d2bbf59 log is divergent from this leader: its last log entry 14.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:10:23.438532 31735 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:23.438974 31326 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:10:23.439085 31735 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:23.425796 31340 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:23.409454 31740 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:23.441563 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:23.436848 31151 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:23.444904 31577 consensus_queue.cc:237] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:23.428471 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 2260) took 1326 ms. Trace:
I20260709 13:10:23.430620 25243 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:10:23.432884 31154 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:23.481215 25943 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:10:23.420109 31580 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:23.482785 31735 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:23.436883 31721 raft_consensus.cc:740] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:23.482321 25945 pending_rounds.cc:85] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 10
I20260709 13:10:23.450173 25392 rpcz_store.cc:276] 0709 13:10:22.096878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:22.099039 (+  2161us) service_pool.cc:224] Handling call
0709 13:10:22.099295 (+   256us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:22.099525 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:10:22.359515 (+259990us) log.cc:844] Serialized 14 byte log entry
0709 13:10:23.409614 (+1050099us) spinlock_profiling.cc:243] Waited 4.48 ms on lock 0x7b18000e8000. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f297685a4bc 00007f2973ba848b 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc2455 00007f2976caa060 00007f2976cbc716 00007f2976ccd287
0709 13:10:23.409707 (+    93us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:23.409758 (+    51us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:23.422866 (+ 13108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:23.422983 (+   117us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:23.423144 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8,"prepare.queue_time_us":589699,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":47369,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":2788,"spinlock_wait_cycles":10321920,"thread_start_us":629280,"threads_started":3,"wal-append.queue_time_us":451339}
I20260709 13:10:23.565645 25244 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:10:23.569432 31167 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 13:10:23.574071 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 264022us
I20260709 13:10:23.576623 25945 pending_rounds.cc:107] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260709 13:10:23.450132 31478 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:23.576702 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 209587us
I20260709 13:10:23.582628 25945 raft_consensus.cc:2889] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:23.583498 25209 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:23.483675 31723 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:23.589172 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 183009us
I20260709 13:10:23.594666 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:23.597563 25209 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:23.585150 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:23.600500 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:23.600512 25936 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:23.601344 25243 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 20
I20260709 13:10:23.584877 31362 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:23.587689 31395 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:23.604537 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:23.586371 31711 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:23.651983 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 54515.5us
I20260709 13:10:23.611157 25941 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:23.662112 25284 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:23.743106 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 148519us
W20260709 13:10:23.743829 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 81833.6us
I20260709 13:10:23.744231 25209 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:23.592214 31675 raft_consensus.cc:697] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:23.740962 31362 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:23.606285 31588 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:23.749329 31660 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:23.605928 31742 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:23.970221 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 226087us
I20260709 13:10:23.592025 31321 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.151s
I20260709 13:10:23.758436 31748 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:23.749498 31674 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:23.976151 25943 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:23.977895 25243 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
W20260709 13:10:23.974299 25909 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.65:45313 timed out after 2.005s (SENT) [suppressed 29 similar messages]
I20260709 13:10:23.988963 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:23.989789 25209 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:23.993259 31743 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:23.988775 31660 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:23.994731 31742 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:23.665522 31721 consensus_queue.cc:260] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:23.995992 31746 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.491s	user 0.001s	sys 0.000s
W20260709 13:10:23.997860 31759 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.247s	user 0.000s	sys 0.000s
I20260709 13:10:23.974382 31151 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:23.979700 31691 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:24.023455 25285 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:23.995239 31362 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:24.034847 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:23.978912 31718 consensus_queue.cc:1059] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.049s
I20260709 13:10:24.030755 31660 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:23.982260 31395 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.163797 31272 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:10:24.164630 31272 raft_consensus.cc:697] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 20 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:10:24.165673 31737 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.577s	user 0.003s	sys 0.004s
W20260709 13:10:24.166481 31751 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.203s	user 0.001s	sys 0.000s
I20260709 13:10:24.166952 31744 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:24.142927 31741 log.cc:927] Time spent T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.568s	user 0.001s	sys 0.000s
W20260709 13:10:24.187139 31577 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.033s	sys 0.032s
I20260709 13:10:24.146620 31386 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:24.172103 31675 consensus_queue.cc:237] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.142956 31748 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 13:10:24.143427 31754 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 13:10:24.002257 31167 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:24.191327 25943 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
W20260709 13:10:24.189173 31755 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.201s	user 0.002s	sys 0.000s
W20260709 13:10:24.196036 31577 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.747s	user 0.034s	sys 0.032s
I20260709 13:10:24.228372 25939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 1218) took 1050 ms. Trace:
I20260709 13:10:24.233951 25939 rpcz_store.cc:276] 0709 13:10:23.163894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:23.165019 (+  1125us) service_pool.cc:224] Handling call
0709 13:10:23.165887 (+   868us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:23.185312 (+ 19425us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:23.185340 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:23.185830 (+   490us) log.cc:844] Serialized 26 byte log entry
0709 13:10:23.334070 (+148240us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:23.334110 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:23.334192 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:24.214043 (+879851us) raft_consensus.cc:1685] finished
0709 13:10:24.214066 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:24.214222 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18959,"raft.queue_time_us":22995,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"spinlock_wait_cycles":64384,"thread_start_us":147850,"threads_started":1,"wal-append.queue_time_us":236167}
I20260709 13:10:24.196218 31718 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:24.277578 25238 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
W20260709 13:10:24.322736 31764 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
W20260709 13:10:24.324640 31757 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
I20260709 13:10:24.186439 31744 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.141235 31691 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.254066 31723 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:24.354455 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 2288) took 1417 ms. Trace:
I20260709 13:10:24.355012 25393 rpcz_store.cc:276] 0709 13:10:22.937352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:23.001841 (+ 64489us) service_pool.cc:224] Handling call
0709 13:10:23.017910 (+ 16069us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:23.022386 (+  4476us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:23.022417 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:23.027930 (+  5513us) log.cc:844] Serialized 26 byte log entry
0709 13:10:23.285283 (+257353us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:23.285349 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:23.285440 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:24.354121 (+1068681us) raft_consensus.cc:1685] finished
0709 13:10:24.354156 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:24.354389 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22420,"raft.queue_time_us":7175,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":11176,"thread_start_us":206533,"threads_started":1,"wal-append.queue_time_us":301944}
I20260709 13:10:24.360330 25944 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 1225) took 1072 ms. Trace:
I20260709 13:10:24.360455 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2596) took 2575 ms. Trace:
I20260709 13:10:24.200661 31386 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.223623 31167 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.361074 25240 rpcz_store.cc:276] 0709 13:10:21.784715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:22.760178 (+975463us) service_pool.cc:224] Handling call
0709 13:10:22.768087 (+  7909us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:23.391658 (+623571us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:23.391686 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:23.392131 (+   445us) log.cc:844] Serialized 26 byte log entry
0709 13:10:23.580146 (+188015us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:23.580183 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:23.580255 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:24.360194 (+779939us) raft_consensus.cc:1685] finished
0709 13:10:24.360219 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:24.360381 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":429800,"raft.queue_time_us":461546,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":2322,"thread_start_us":341676,"threads_started":2,"wal-append.queue_time_us":332325}
I20260709 13:10:24.360937 25944 rpcz_store.cc:276] 0709 13:10:23.287841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:23.287981 (+   140us) service_pool.cc:224] Handling call
0709 13:10:23.288390 (+   409us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:23.327084 (+ 38694us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:23.327120 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:23.357778 (+ 30658us) log.cc:844] Serialized 52 byte log entry
0709 13:10:23.606873 (+249095us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:10:23.606918 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:23.606988 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:24.360026 (+753038us) raft_consensus.cc:1685] finished
0709 13:10:24.360053 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:24.360258 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2717,"raft.queue_time_us":11804,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":8917,"thread_start_us":224893,"threads_started":1,"wal-append.queue_time_us":632161}
I20260709 13:10:24.317315 31272 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:24.389307 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 399586us
I20260709 13:10:24.399726 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 1226) took 1030 ms. Trace:
I20260709 13:10:24.402531 25209 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:24.366739 31588 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:24.406271 25209 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:24.391891 25241 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.903s	user 0.003s	sys 0.027s
I20260709 13:10:24.320888 31718 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:24.407507 25241 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.919s	user 0.003s	sys 0.027s
W20260709 13:10:24.409189 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 385826us
I20260709 13:10:24.373222 31723 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:24.411907 25209 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:24.405614 31371 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:24.415073 25313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2796) took 1125 ms. Trace:
I20260709 13:10:24.415454 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:24.416287 25240 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:24.402375 25937 rpcz_store.cc:276] 0709 13:10:23.297768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:23.301349 (+  3581us) service_pool.cc:224] Handling call
0709 13:10:23.301708 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:23.322494 (+ 20786us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:10:23.322524 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:23.339406 (+ 16882us) log.cc:844] Serialized 26 byte log entry
0709 13:10:23.489943 (+150537us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:23.489986 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:23.490055 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:24.327388 (+837333us) raft_consensus.cc:1685] finished
0709 13:10:24.327414 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:24.327593 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":20433,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":3977,"thread_start_us":101432,"threads_started":1,"wal-append.queue_time_us":220439}
I20260709 13:10:24.415510 25313 rpcz_store.cc:276] 0709 13:10:23.289732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:23.313198 (+ 23466us) service_pool.cc:224] Handling call
0709 13:10:23.313530 (+   332us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:23.313925 (+   395us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:10:23.313964 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:23.314552 (+   588us) log.cc:844] Serialized 52 byte log entry
0709 13:10:23.752568 (+438016us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:23.752616 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:23.752703 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:24.414841 (+662138us) raft_consensus.cc:1685] finished
0709 13:10:24.414863 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:24.415011 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7855,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":2877,"thread_start_us":437633,"threads_started":1,"wal-append.queue_time_us":924253}
I20260709 13:10:24.388527 31742 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:24.424954 31371 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:24.432112 31718 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:24.433377 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:24.427930 31762 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:24.434043 25240 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:24.434235 31762 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.435120 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:24.427604 31605 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:24.437595 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:24.409148 31768 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:10:24.432149 31691 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:24.442201 31691 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:24.442804 31691 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.443462 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:24.444833 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:24.445227 31691 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:24.446776 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:24.441561 31769 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:24.449335 31769 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.451681 31769 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:24.452661 31769 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:24.438261 31742 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:24.455231 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2649) took 1179 ms. Trace:
I20260709 13:10:24.458504 25246 rpcz_store.cc:276] 0709 13:10:23.276163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:23.276296 (+   133us) service_pool.cc:224] Handling call
0709 13:10:23.276679 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:23.287018 (+ 10339us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:23.287054 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:23.296560 (+  9506us) log.cc:844] Serialized 26 byte log entry
0709 13:10:23.578303 (+281743us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:23.747295 (+168992us) log.cc:844] Serialized 14 byte log entry
0709 13:10:23.747532 (+   237us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:23.747606 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:24.454913 (+707307us) raft_consensus.cc:1685] finished
0709 13:10:24.454943 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:24.455123 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16103,"prepare.queue_time_us":460139,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":3377,"raft.queue_time_us":67633,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":47357,"thread_start_us":361397,"threads_started":2,"wal-append.queue_time_us":1099354}
I20260709 13:10:24.464273 31386 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:24.465029 31386 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:24.465667 31386 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:24.469854 31756 log.cc:927] Time spent T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 13:10:24.470247 25192 auto_rebalancer.cc:629] Failed to schedule move for tablet 6f616cb6823f424a847dceaa5900dfc0: Illegal state: Replica 46de6919f85341c6a9b574dff112e35a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 19 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:24.472599 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2793) took 1231 ms. Trace:
I20260709 13:10:24.458210 31605 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.473104 25317 rpcz_store.cc:276] 0709 13:10:23.240765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:23.240891 (+   126us) service_pool.cc:224] Handling call
0709 13:10:23.241204 (+   313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:23.250781 (+  9577us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:10:23.250813 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:23.284853 (+ 34040us) log.cc:844] Serialized 26 byte log entry
0709 13:10:23.577117 (+292264us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:23.577175 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:23.577245 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:24.472243 (+894998us) raft_consensus.cc:1685] finished
0709 13:10:24.472332 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:24.472519 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1811,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":18717,"thread_start_us":262318,"threads_started":1,"wal-append.queue_time_us":1042076}
I20260709 13:10:24.475857 31605 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:24.475972 31744 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:24.476639 31742 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.409148 31588 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.479326 31742 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:24.479545 31588 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:24.421849 31723 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.436522 31762 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:24.489055 31723 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:24.462646 31727 consensus_meta.cc:287] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 1.304s	user 0.017s	sys 0.000s
I20260709 13:10:24.493563 31727 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.495852 31727 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:24.453305 31769 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.497875 31167 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:24.499171 31769 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:24.453786 31371 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:24.501475 25945 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1199) took 1050 ms. Trace:
I20260709 13:10:24.502197 31371 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.501914 25945 rpcz_store.cc:276] 0709 13:10:23.451577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:23.451730 (+   153us) service_pool.cc:224] Handling call
0709 13:10:23.481991 (+ 30261us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:10:23.651566 (+169575us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:23.651600 (+    34us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:10:23.668546 (+ 16946us) log.cc:844] Serialized 78 byte log entry
0709 13:10:24.330915 (+662369us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:24.330966 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:24.331050 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:24.501206 (+170156us) raft_consensus.cc:1685] finished
0709 13:10:24.501230 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:24.501414 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":46077,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":86543,"thread_start_us":302759,"threads_started":1,"wal-append.queue_time_us":487462}
I20260709 13:10:24.507771 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:24.517263 25358 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.007s	sys 0.034s
W20260709 13:10:24.517771 25358 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.008s	sys 0.034s
I20260709 13:10:24.528640 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:24.535723 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:24.537261 31771 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:24.537863 31771 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.509451 31722 raft_consensus.cc:3055] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:24.550515 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:24.551472 25393 raft_consensus.cc:2410] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 13:10:24.558526 25911 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
W20260709 13:10:24.564132 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:24.564913 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:24.571240 25237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2648) took 1295 ms. Trace:
I20260709 13:10:24.543816 31722 raft_consensus.cc:740] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:24.586777 25945 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:24.551985 31730 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:24.588027 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 162771us
W20260709 13:10:24.597502 25909 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.013s	sys 0.019s
W20260709 13:10:24.598187 25909 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.013s	sys 0.019s
I20260709 13:10:24.597195 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:24.569212 31577 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:24.601444 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:24.602430 25394 raft_consensus.cc:2410] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 13:10:24.588276 31732 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:24.571812 25237 rpcz_store.cc:276] 0709 13:10:23.275582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:23.275770 (+   188us) service_pool.cc:224] Handling call
0709 13:10:23.298271 (+ 22501us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:23.365668 (+ 67397us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:23.365715 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:23.373790 (+  8075us) log.cc:844] Serialized 26 byte log entry
0709 13:10:23.603014 (+229224us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:24.009290 (+406276us) log.cc:844] Serialized 14 byte log entry
0709 13:10:24.200965 (+191675us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:24.555958 (+354993us) spinlock_profiling.cc:243] Waited 52.8 ms on lock 0x7b5400363154. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:24.556031 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:24.556124 (+    93us) raft_consensus.cc:1685] finished
0709 13:10:24.570915 (+ 14791us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:24.571161 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":154042,"prepare.queue_time_us":568557,"prepare.run_cpu_time_us":829,"prepare.run_wall_time_us":12535,"raft.queue_time_us":64666,"raft.run_cpu_time_us":325,"raft.run_wall_time_us":107197,"spinlock_wait_cycles":121473408,"thread_start_us":603133,"threads_started":3,"wal-append.queue_time_us":399285}
W20260709 13:10:24.598590 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 624281us
I20260709 13:10:24.623461 25359 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:24.625227 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:24.637094 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:24.647360 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:24.645444 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:24.604228 31776 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:24.646536 25359 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:24.611359 31753 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:24.611208 31715 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:24.655542 25909 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:24.622545 31588 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:24.673799 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:24.611595 31748 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:24.675251 25209 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:24.667762 25359 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:24.624895 31732 raft_consensus.cc:697] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:24.682968 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:24.683868 25237 raft_consensus.cc:2410] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 19 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
W20260709 13:10:24.687937 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 124929us
I20260709 13:10:24.605360 31577 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.667388 31272 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:24.668510 31743 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:24.691267 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 50941.1us
I20260709 13:10:24.692121 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:24.668938 31769 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:24.690882 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:24.649868 31730 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:24.699785 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:24.679764 31742 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:24.639092 31660 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.686388 31744 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:24.712422 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:24.723482 25209 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:24.724239 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 83910.3us
I20260709 13:10:24.673287 31776 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.713747 31362 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:24.726011 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:24.688567 31588 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.727403 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:24.729835 25237 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:24.730397 25209 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:24.733446 25209 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:24.722466 31718 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:24.681666 31753 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.710824 31769 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.722827 31727 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:24.698345 31272 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.716609 31742 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.687074 31715 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.742331 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:24.760797 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:24.727743 31744 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:10:24.713488 31730 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.743748 31353 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:24.735764 31390 raft_consensus.cc:3055] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:24.765834 31771 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:24.766032 31390 raft_consensus.cc:740] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:24.767015 31793 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:24.767277 31789 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:24.747642 25939 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:24.767069 31390 consensus_queue.cc:260] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.767788 31789 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:24.767607 31793 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.749675 31386 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:24.768232 31789 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:24.726258 31167 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.018s
I20260709 13:10:24.737574 31718 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:24.762703 25936 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:24.734745 31371 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:24.749351 31783 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:24.765534 31772 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:24.746702 31782 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:24.749306 31775 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 13:10:24.714475 31743 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.771423 25937 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 13:10:24.688799 31748 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.737936 31768 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:24.739789 31762 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:24.686221 31722 consensus_queue.cc:260] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.775617 31718 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:24.779151 31371 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:24.780193 31660 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:24.801731 31722 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:24.740499 31732 consensus_queue.cc:237] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.750154 31784 consensus_queue.cc:1059] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 13:10:24.786000 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:24.768244 31390 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:24.757764 31727 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:24.796300 31779 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
W20260709 13:10:24.797406 31790 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 13:10:24.799198 31510 consensus_queue.cc:1048] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:10:24.800493 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2617) took 1417 ms. Trace:
I20260709 13:10:24.801579 31660 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 13:10:24.784072 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 56213.6us
I20260709 13:10:24.809476 25943 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 16: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:24.814129 25237 raft_consensus.cc:2393] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 18: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:24.818181 25241 rpcz_store.cc:276] 0709 13:10:23.382635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:23.392474 (+  9839us) service_pool.cc:224] Handling call
0709 13:10:23.392817 (+   343us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:23.406423 (+ 13606us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:23.406474 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:23.411378 (+  4904us) log.cc:844] Serialized 52 byte log entry
0709 13:10:24.409196 (+997818us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:24.422641 (+ 13445us) log.cc:844] Serialized 14 byte log entry
0709 13:10:24.425366 (+  2725us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:24.425446 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:24.539392 (+113946us) spinlock_profiling.cc:243] Waited 1.79 ms on lock 0x7b180041f920. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:10:24.800139 (+260747us) raft_consensus.cc:1685] finished
0709 13:10:24.800252 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:24.800424 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7193,"prepare.queue_time_us":16235,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":19919,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":17309,"spinlock_wait_cycles":4125056,"thread_start_us":920146,"threads_started":1,"wal-append.queue_time_us":1098604}
I20260709 13:10:24.821768 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:24.822381 25241 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:24.823302 31419 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:24.823786 31419 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.825819 31419 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:24.829473 25241 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:24.830327 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:24.833689 31654 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:24.829109 25287 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:24.839811 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 117300us
I20260709 13:10:24.840955 31167 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:24.842149 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:24.842759 25237 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 12.
I20260709 13:10:24.843894 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:24.806130 31371 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.821512 31776 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:24.835193 31772 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:24.850791 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:24.847361 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:24.786110 31782 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.837498 31762 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:24.854920 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2630) took 1359 ms. Trace:
I20260709 13:10:24.853869 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:24.860076 25241 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:24.860369 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:24.861768 31774 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:24.862366 31774 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:24.864071 31777 log.cc:927] Time spent T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260709 13:10:24.860177 31340 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:24.865303 31801 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:24.865682 31340 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.865902 31801 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.866497 31727 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:24.866799 31796 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:24.867342 31796 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.867853 31340 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:24.868088 31801 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:24.868572 31272 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:24.868646 31801 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:24.869441 31796 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:24.869625 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2625) took 1439 ms. Trace:
I20260709 13:10:24.870941 31748 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:24.871793 31762 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.876796 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:24.851106 31577 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:24.876996 31419 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:24.891017 31419 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.893671 31730 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:24.894425 31419 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:24.894806 31769 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:24.880403 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 69482.3us
I20260709 13:10:24.878562 25243 rpcz_store.cc:276] 0709 13:10:23.430064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:24.202772 (+772708us) service_pool.cc:224] Handling call
0709 13:10:24.230282 (+ 27510us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:24.309927 (+ 79645us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:10:24.310245 (+   318us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:24.347586 (+ 37341us) log.cc:844] Serialized 52 byte log entry
0709 13:10:24.605527 (+257941us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:24.605588 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:24.605683 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:24.869405 (+263722us) raft_consensus.cc:1685] finished
0709 13:10:24.869424 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:24.869564 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":138,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":20469,"thread_start_us":218719,"threads_started":1,"wal-append.queue_time_us":368882}
I20260709 13:10:24.855799 25242 rpcz_store.cc:276] 0709 13:10:23.495460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:24.379724 (+884264us) service_pool.cc:224] Handling call
0709 13:10:24.399146 (+ 19422us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:24.407938 (+  8792us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:24.407968 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:24.412157 (+  4189us) log.cc:844] Serialized 26 byte log entry
0709 13:10:24.560969 (+148812us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:10:24.561032 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:24.561144 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:24.854696 (+293552us) raft_consensus.cc:1685] finished
0709 13:10:24.854717 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:24.854867 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2601,"raft.queue_time_us":334,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":8782,"spinlock_wait_cycles":1760000,"thread_start_us":137459,"threads_started":1,"wal-append.queue_time_us":225018}
I20260709 13:10:24.882798 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:24.883857 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:24.873445 31782 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:24.876525 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:24.896512 31588 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:24.885560 25237 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:24.889514 31743 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:24.890036 31762 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:24.896396 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:24.899178 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:24.899439 25287 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:24.900138 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:24.909845 25243 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:24.910144 25237 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:24.910492 25287 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:24.915282 25241 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:24.913590 31419 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:10:24.898051 31801 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.919139 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:24.919683 31371 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:24.897490 31789 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.922384 25313 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 13:10:24.924027 31789 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:24.925249 31801 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:24.929376 25287 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:24.897610 31710 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:10:24.934422 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 168980us
I20260709 13:10:24.897775 31795 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:24.936695 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:24.938218 25209 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:24.897547 31715 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:24.939067 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:24.941329 25393 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:24.942256 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:24.943617 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:24.856024 31772 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:24.947163 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:24.947103 25209 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:24.949579 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:24.950198 25237 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:24.948944 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:24.952332 31390 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:24.942891 31151 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:24.953774 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:24.972366 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:24.974332 25243 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:24.934114 31419 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.014s
I20260709 13:10:24.978615 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:24.976786 25909 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:24.980486 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:24.941509 31753 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:24.981616 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:24.983112 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:24.948427 31371 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:24.968155 31798 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:24.984620 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:24.953680 31710 raft_consensus.cc:2764] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 13:10:24.988565 25317 raft_consensus.cc:2410] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 13:10:24.986084 25287 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:24.990967 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:24.992074 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:24.993834 25936 raft_consensus.cc:2393] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 17: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:24.940246 31795 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:24.995038 31769 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:24.996531 25319 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 13:10:24.982758 31727 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:10:24.967926 31803 consensus_queue.cc:1059] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260709 13:10:24.978338 31762 consensus_queue.cc:1059] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 13:10:25.012321 25237 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:25.040902 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:25.014750 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:25.016651 31807 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:25.016966 31395 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:25.017439 31799 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:25.029711 31771 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:25.034291 31787 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.285s	user 0.002s	sys 0.000s
W20260709 13:10:25.005887 25320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2812) took 2709 ms (client timeout 2110 ms). Trace:
I20260709 13:10:24.989792 31390 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:25.010052 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:25.047011 25315 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:25.008818 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:24.992484 31806 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:25.039808 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:24.981249 31801 consensus_queue.cc:1059] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.021s
W20260709 13:10:25.045377 25320 rpcz_store.cc:269] 0709 13:10:22.286327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:22.286872 (+   545us) service_pool.cc:224] Handling call
0709 13:10:24.995859 (+2708987us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54000f8f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:24.995966 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2379587456}
I20260709 13:10:25.051793 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:25.006306 31727 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:25.049736 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:25.052614 25938 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 17: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:25.052490 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:25.050273 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2615) took 1671 ms. Trace:
I20260709 13:10:25.050026 25313 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:25.054034 25318 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:25.056535 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:25.057782 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:25.066682 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:25.065877 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:25.068300 25237 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:24.995718 31769 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:25.073668 25387 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:10:25.073689 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:25.075882 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:25.077633 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:25.054780 25239 rpcz_store.cc:276] 0709 13:10:23.378679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:23.380036 (+  1357us) service_pool.cc:224] Handling call
0709 13:10:23.380309 (+   273us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:23.441832 (+ 61523us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:10:23.565398 (+123566us) log.cc:844] Serialized 14 byte log entry
0709 13:10:23.973397 (+407999us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:23.980750 (+  7353us) log.cc:844] Serialized 26 byte log entry
0709 13:10:23.980916 (+   166us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:23.980961 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:24.503280 (+522319us) spinlock_profiling.cc:243] Waited 7.48 ms on lock 0x7b1800390160. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc5522 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:10:25.049874 (+546594us) spinlock_profiling.cc:243] Waited 61.1 ms on lock 0x7b5400317ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:25.049979 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:25.050035 (+    56us) raft_consensus.cc:1685] finished
0709 13:10:25.050053 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:25.050193 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":300327,"prepare.run_cpu_time_us":6086,"prepare.run_wall_time_us":218947,"raft.queue_time_us":109788,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":6220,"spinlock_wait_cycles":157644928,"thread_start_us":398429,"threads_started":2,"wal-append.queue_time_us":466839}
I20260709 13:10:25.079527 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:25.080292 25239 raft_consensus.cc:2410] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 19 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 13:10:25.082852 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2611) took 1705 ms. Trace:
I20260709 13:10:25.083295 25244 rpcz_store.cc:276] 0709 13:10:23.377064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:23.377206 (+   142us) service_pool.cc:224] Handling call
0709 13:10:23.377511 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:23.576183 (+198672us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:25.082642 (+1506459us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b54000a00d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:25.082797 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":293413632}
W20260709 13:10:25.093961 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 55280.9us
I20260709 13:10:25.048790 31794 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:25.095098 25909 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:25.096911 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:25.066883 31727 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:25.072157 31782 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:25.108909 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:25.124346 25316 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:25.053057 31771 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:25.110616 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 58637.7us
I20260709 13:10:25.134256 25359 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:25.126865 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:25.130697 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:25.133214 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:25.048687 31799 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:25.125166 31811 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.137s
I20260709 13:10:25.154110 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:25.150876 31802 log.cc:927] Time spent T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.200s	user 0.001s	sys 0.000s
I20260709 13:10:25.163071 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:25.101733 31808 consensus_queue.cc:1059] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.218s
I20260709 13:10:25.165163 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:25.049033 31807 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:25.166805 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:25.166726 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:25.167719 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:25.166126 31789 raft_consensus.cc:3055] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:25.171700 25315 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:25.171954 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:25.172839 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:25.174201 25320 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:25.176322 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1189) took 1945 ms. Trace:
I20260709 13:10:25.172663 31772 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:25.178321 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:25.182016 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:25.182775 31809 log.cc:927] Time spent T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260709 13:10:25.183405 31808 raft_consensus.cc:3055] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:25.183800 31808 raft_consensus.cc:740] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:25.184904 31827 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:25.184614 31808 consensus_queue.cc:260] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:25.185086 25240 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:25.176693 25941 rpcz_store.cc:276] 0709 13:10:23.231345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:23.232389 (+  1044us) service_pool.cc:224] Handling call
0709 13:10:23.239780 (+  7391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:23.744863 (+505083us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:25.176104 (+1431241us) spinlock_profiling.cc:243] Waited 371 ms on lock 0x7b5400702bd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:25.176268 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":853479296}
I20260709 13:10:25.188288 25314 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:25.188650 25392 raft_consensus.cc:2393] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 16: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:25.190969 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:25.191567 25391 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:25.192242 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:25.192847 25387 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:25.193683 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:25.194016 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:25.194629 25390 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:10:25.194900 31771 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:25.173593 31789 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:25.193571 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:25.196429 25388 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:25.142823 31782 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:25.202528 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:25.203260 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:25.203904 25318 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:25.204190 25390 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:10:25.204427 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:25.185999 31808 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:25.205017 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:25.204188 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:25.187891 31795 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:25.225930 25391 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:25.233608 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:25.238014 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:25.205534 31798 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:25.251704 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:25.253497 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:25.254853 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:25.256664 25318 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:10:25.256686 25315 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:10:25.261586 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:25.262262 25390 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:25.262797 31807 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:25.227622 31789 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:25.263523 25315 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:10:25.267491 25318 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:10:25.270305 31718 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:25.272398 31718 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:25.273855 31769 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:25.279793 25393 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
W20260709 13:10:25.295987 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 97389.3us
W20260709 13:10:25.303176 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 208178us
I20260709 13:10:25.336864 25909 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:25.340677 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:25.336772 31771 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.045s
W20260709 13:10:25.381803 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:25.339555 31730 consensus_queue.cc:1059] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 13:10:25.385526 25390 raft_consensus.cc:1240] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 13:10:25.390331 31727 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:25.392364 31782 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:25.397895 31832 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:10:25.402477 31820 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:25.340809 31691 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:10:25.425009 31751 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 13:10:25.446112 25941 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 12. Current term is 13. Ops: [12.12-12.12]
I20260709 13:10:25.441296 31691 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:25.457052 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 111450us
I20260709 13:10:25.446657 31820 raft_consensus.cc:697] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:25.441411 31730 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.015s
W20260709 13:10:25.465375 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 187319us
I20260709 13:10:25.459250 31691 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:25.471105 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:25.461153 31799 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:25.501830 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 165180us
I20260709 13:10:25.503049 25909 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:25.508438 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 48229.1us
I20260709 13:10:25.469342 31840 raft_consensus.cc:3055] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:25.515928 25285 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:25.521620 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 178045us
I20260709 13:10:25.514528 31840 raft_consensus.cc:740] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:10:25.537039 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 67364.2us
I20260709 13:10:25.540695 25909 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:25.546883 31846 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:10:25.548774 31846 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:25.530520 31773 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:25.549880 31846 raft_consensus.cc:697] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:10:25.555985 31835 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260709 13:10:25.555792 31820 consensus_queue.cc:237] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:25.552884 31773 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:25.552474 31807 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
W20260709 13:10:25.568311 31838 log.cc:927] Time spent T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:10:25.560384 31769 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:25.581668 31769 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:25.584074 31769 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:25.563776 31773 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:25.586171 31727 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:25.590720 31478 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:25.593966 31841 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:25.558427 31840 consensus_queue.cc:260] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:25.556183 31846 consensus_queue.cc:237] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:25.595376 31840 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:25.597546 31769 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:25.598273 31390 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:25.598194 31769 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:25.599021 31390 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:25.604009 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 81207.1us
W20260709 13:10:25.605713 31852 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 13:10:25.609402 31691 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:25.611038 31390 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:25.613997 31789 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:25.616576 25319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2817) took 2181 ms. Trace:
I20260709 13:10:25.606953 25911 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:25.625835 31769 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:25.632774 25239 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:25.634421 31742 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:25.624866 31773 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:25.644575 25319 rpcz_store.cc:276] 0709 13:10:23.435069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:23.440931 (+  5862us) service_pool.cc:224] Handling call
0709 13:10:24.996124 (+1555193us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:24.997018 (+   894us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:25.616314 (+619296us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b54000f8f50. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:25.616479 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3575760768}
W20260709 13:10:25.650406 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 43565.2us
I20260709 13:10:25.651829 25911 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:25.666324 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 127492us
I20260709 13:10:25.641192 31223 raft_consensus.cc:3055] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:25.676683 25944 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:25.681425 31223 raft_consensus.cc:740] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:25.677692 31841 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:25.673664 31842 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:25.702225 31223 consensus_queue.cc:260] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:25.703367 31223 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:25.707461 31742 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:25.696359 31840 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:25.708070 31742 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:25.717657 31769 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:25.721511 31769 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:25.722473 31774 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:25.723661 31769 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:25.727677 31793 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:25.730295 31773 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:25.737963 31691 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:25.709097 31841 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 13:10:25.742494 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:25.742630 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:25.743592 25243 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:25.749701 25245 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:10:25.749961 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:25.750339 25944 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1205) took 1061 ms. Trace:
I20260709 13:10:25.748279 25285 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:25.750916 25944 rpcz_store.cc:276] 0709 13:10:24.688471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:24.688584 (+   113us) service_pool.cc:224] Handling call
0709 13:10:25.602064 (+913480us) spinlock_profiling.cc:243] Waited 3.24 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:10:25.602348 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:25.681153 (+ 78805us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:25.750023 (+ 68870us) spinlock_profiling.cc:243] Waited 72.7 ms on lock 0x7b540062e3d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:25.750263 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":174755584}
I20260709 13:10:25.752318 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:25.753968 25241 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:25.750574 25244 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:25.752806 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:25.755419 25237 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:25.758329 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:25.758880 25243 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:25.750898 31769 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:25.718611 31840 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:25.719077 31842 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
W20260709 13:10:25.789497 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 248852us
I20260709 13:10:25.761965 31803 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:25.792680 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:25.799016 25909 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:25.802220 31840 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:25.792436 31803 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:25.808429 25284 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:25.808938 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:25.794950 31846 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:10:25.810063 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 111270us
I20260709 13:10:25.810665 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:25.814464 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:25.816524 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:25.817387 25941 raft_consensus.cc:2393] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 13: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:25.822759 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:25.792565 31769 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:25.824793 25285 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:25.810613 31803 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:25.830850 25285 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:25.831126 31807 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:25.831508 31807 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:25.831897 31807 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:25.831794 25285 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:25.818681 31846 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:25.819773 31390 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:25.821316 31735 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:25.813423 31774 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:25.837471 31771 consensus_queue.cc:1059] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:10:25.832446 31799 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:25.827307 31588 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.015s
I20260709 13:10:25.844851 31805 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:25.846563 25284 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:25.848783 31799 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:25.836551 31846 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:25.852705 31805 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:25.857504 31223 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:25.857717 31799 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:25.866787 31805 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:25.883239 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:25.872776 31390 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:25.879163 31742 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:25.885634 31390 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:25.887866 31390 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:25.884097 25245 raft_consensus.cc:2376] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16. Re-sending same reply.
I20260709 13:10:25.890118 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:25.891181 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:25.891708 25245 raft_consensus.cc:2376] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17. Re-sending same reply.
I20260709 13:10:25.892385 31223 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:25.892549 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:25.893239 25245 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:25.868050 31735 consensus_queue.cc:1059] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 13:10:25.895967 31223 raft_consensus.cc:3055] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:25.896512 31223 raft_consensus.cc:740] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:25.897619 31223 consensus_queue.cc:260] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:25.899101 31223 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:25.901924 31721 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:25.910704 31768 raft_consensus.cc:3055] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:25.911199 31768 raft_consensus.cc:740] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:25.912145 31768 consensus_queue.cc:260] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:25.930064 31768 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:25.937281 31732 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:25.937927 31793 raft_consensus.cc:3055] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:25.937786 31803 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:25.938409 31793 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:25.937888 31732 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:25.939312 31793 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:25.940017 31803 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:25.941054 31732 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:25.942745 31762 raft_consensus.cc:3055] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:25.943192 31762 raft_consensus.cc:740] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:25.943863 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:25.944029 31762 consensus_queue.cc:260] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:25.950109 31769 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:25.953413 25941 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:10:25.954362 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:25.954653 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:25.955150 25246 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:25.956499 31769 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:10:25.960911 31827 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:25.961411 31827 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:25.961742 31827 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:25.962961 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:25.963080 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 681710us
I20260709 13:10:25.963814 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 1130) took 1221 ms. Trace:
I20260709 13:10:25.963862 25358 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:25.964295 25242 rpcz_store.cc:276] 0709 13:10:24.742526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:24.896067 (+153541us) service_pool.cc:224] Handling call
0709 13:10:25.962856 (+1066789us) spinlock_profiling.cc:243] Waited 3.59 ms on lock 0x7b5800002798. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8deee 0000000000375939 00007f297bc011e0 00007f297bcca0c4 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:10:25.963740 (+   884us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8287232}
I20260709 13:10:25.965332 31813 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:25.914613 31862 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:25.965868 31813 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:25.966361 31813 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:25.966708 25358 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:25.967499 31577 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:25.969565 31827 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:25.970769 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2662) took 1370 ms. Trace:
I20260709 13:10:25.970901 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:25.971136 25239 rpcz_store.cc:276] 0709 13:10:24.600339 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:25.164265 (+563926us) service_pool.cc:224] Handling call
0709 13:10:25.632570 (+468305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:25.645507 (+ 12937us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:25.970603 (+325096us) spinlock_profiling.cc:243] Waited 468 ms on lock 0x7b54001f6dd0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:25.970719 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1076312320}
I20260709 13:10:25.974165 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:25.971442 25393 raft_consensus.cc:2376] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15. Re-sending same reply.
I20260709 13:10:25.928272 31735 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:25.975867 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:25.976081 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 336391us
I20260709 13:10:25.983333 25237 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 20.
I20260709 13:10:25.983441 25358 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:25.973448 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:25.978343 25143 catalog_manager.cc:5697] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 10 to 11, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 11 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:25.976366 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:25.981890 25911 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:25.985459 25209 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:10:25.962962 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:25.982089 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:10:25.976792 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:25.972926 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:10:25.989300 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:25.990703 31223 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:25.991034 31571 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:10:25.972046 31827 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:25.993239 31839 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.440s	user 0.003s	sys 0.000s
W20260709 13:10:25.995699 31781 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 1.247s	user 0.002s	sys 0.003s
I20260709 13:10:25.996940 25393 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 13:10:25.940877 31793 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:26.000195 25387 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:10:26.000756 25911 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:26.004396 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:26.004715 31571 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:26.005642 31390 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:26.006039 31390 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:26.006409 31390 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:25.972462 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:26.009932 25239 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:26.012408 31807 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:25.994189 31735 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:26.010047 25911 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:26.014714 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:26.015497 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:26.016322 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:26.017840 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:26.019410 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:26.020251 25239 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:26.021320 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:26.026813 25358 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:26.028075 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:26.028477 25287 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:26.029831 25358 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:25.998143 31862 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:26.031211 25243 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:26.003026 31643 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:26.032871 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:26.032783 31643 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:26.033731 25242 raft_consensus.cc:2410] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 13:10:26.034540 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:26.034991 31643 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:26.035529 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:26.036590 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:26.037109 25242 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:26.045471 31792 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:26.045899 31792 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:26.046077 31789 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:26.046258 31792 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:26.046289 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:26.046654 31154 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:26.047056 25242 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:26.047581 31789 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:26.044449 31735 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:26.038594 31772 consensus_queue.cc:1059] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.010s
I20260709 13:10:26.049250 31772 raft_consensus.cc:3055] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:26.047994 31789 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:26.049594 31772 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:26.050606 25239 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:26.050740 31772 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:26.051275 25246 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:26.051795 31772 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:26.052564 31643 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:26.037766 31810 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:26.053395 31643 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:26.038905 25911 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:26.055171 31810 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:26.043051 31846 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:26.058228 31853 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:10:26.059128 31846 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:26.058274 31808 consensus_queue.cc:1059] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
I20260709 13:10:26.060352 31643 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:26.060352 31808 raft_consensus.cc:3055] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:10:26.061220 31808 raft_consensus.cc:740] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:26.061417 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:26.062493 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:26.062263 31808 consensus_queue.cc:260] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:26.063061 25246 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:26.063625 31808 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:10:26.064471 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:26.026183 31762 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:26.065169 25246 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:26.066452 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:26.066517 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:26.070097 25287 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:26.072570 25287 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:26.073627 31782 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:26.073899 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:26.074191 31782 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:26.072392 31822 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:26.047951 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:26.076524 31782 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:26.082895 31845 log.cc:927] Time spent T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.537s	user 0.002s	sys 0.000s
I20260709 13:10:26.084774 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 1124) took 1347 ms. Trace:
I20260709 13:10:26.046388 31768 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:26.085168 25238 rpcz_store.cc:276] 0709 13:10:24.738076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:25.199356 (+461280us) service_pool.cc:224] Handling call
0709 13:10:25.206165 (+  6809us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:25.206494 (+   329us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:25.206523 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:25.218721 (+ 12198us) log.cc:844] Serialized 26 byte log entry
0709 13:10:25.516865 (+298144us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:10:25.540670 (+ 23805us) log.cc:844] Serialized 14 byte log entry
0709 13:10:25.540905 (+   235us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:25.540977 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:26.082206 (+541229us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:26.084478 (+  2272us) raft_consensus.cc:1685] finished
0709 13:10:26.084571 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:26.084693 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":22975,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":3402,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":23217,"spinlock_wait_cycles":33379584,"thread_start_us":275540,"threads_started":2,"wal-append.queue_time_us":307616}
I20260709 13:10:26.093448 31807 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:26.093693 25192 auto_rebalancer.cc:629] Failed to schedule move for tablet 9dac81b348b746fd8ecd3937028554ea: Illegal state: Replica 46de6919f85341c6a9b574dff112e35a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 16 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
W20260709 13:10:26.097016 31836 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.555s	user 0.002s	sys 0.000s
W20260709 13:10:26.100288 31837 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.546s	user 0.000s	sys 0.001s
I20260709 13:10:26.102380 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 1122) took 1366 ms. Trace:
I20260709 13:10:26.078222 25242 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:10:26.106341 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:26.107928 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:26.108623 25313 raft_consensus.cc:2376] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 19. Re-sending same reply.
I20260709 13:10:26.109386 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:26.107126 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:26.113484 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:26.107112 25287 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:26.130578 25909 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:26.131080 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 58002.3us
I20260709 13:10:26.132874 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:26.134598 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:26.135532 25243 raft_consensus.cc:2368] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 19. Current term is 20.
I20260709 13:10:26.102813 25240 rpcz_store.cc:276] 0709 13:10:24.736435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:25.188846 (+452411us) service_pool.cc:224] Handling call
0709 13:10:25.202178 (+ 13332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:25.207912 (+  5734us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:25.207946 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:25.208366 (+   420us) log.cc:844] Serialized 26 byte log entry
0709 13:10:25.456767 (+248401us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:25.539441 (+ 82674us) log.cc:844] Serialized 14 byte log entry
0709 13:10:25.541232 (+  1791us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:25.541304 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:25.950169 (+408865us) spinlock_profiling.cc:243] Waited 1.19 ms on lock 0x7b4800080058. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297bbdbaa1 00007f297bbd2b37 00007f297bbd29ca 00007f297bcca26f 00007f297bcc9f37 00007f297bcde55c 00007f297bcde508 00007f297bcde488 00007f297bcde443 00007f297bcdd6ec 00007f297ac07a96
0709 13:10:26.099455 (+149286us) spinlock_profiling.cc:243] Waited 2.31 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:26.102106 (+  2651us) raft_consensus.cc:1685] finished
0709 13:10:26.102134 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:26.102299 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2612,"prepare.queue_time_us":82911,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":14514,"raft.run_cpu_time_us":2756,"raft.run_wall_time_us":399633,"spinlock_wait_cycles":8065280,"thread_start_us":251870,"threads_started":2,"wal-append.queue_time_us":291803}
I20260709 13:10:26.137391 25313 raft_consensus.cc:2393] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 15: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:26.142339 25314 raft_consensus.cc:1240] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:26.145038 25287 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:26.069231 31825 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:26.148923 25909 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:26.151248 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:26.152525 25320 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:10:26.154038 25287 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:26.156068 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:26.157234 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:26.157996 25318 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 20.
I20260709 13:10:26.160570 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:26.167578 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:26.060844 31805 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:26.137234 31403 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:26.211433 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:26.141719 31798 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:26.173887 25394 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:26.217497 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:26.105259 31799 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:26.205731 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:26.224915 25243 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:26.150702 31825 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:26.158775 31588 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:26.159227 31735 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:26.171336 31842 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:26.209088 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:26.214056 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:26.155826 31459 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:26.209434 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:26.037609 31813 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:26.163555 31771 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:26.220837 25945 raft_consensus.cc:2410] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 20 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
W20260709 13:10:26.221252 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:26.168849 25359 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:26.254740 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 85974.2us
I20260709 13:10:26.132956 31768 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:26.231334 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:26.233392 25312 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 18: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:10:26.226078 31787 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 13:10:26.267028 25391 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:26.261183 25390 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:26.220430 31571 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:26.271332 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:26.274013 25359 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:26.223819 31827 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:26.275535 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:26.276612 25389 raft_consensus.cc:1240] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:26.278438 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:26.266455 31810 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:26.281006 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:26.281383 31868 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 13:10:26.281667 25314 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:26.281939 31789 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:26.232407 31478 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:26.288779 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:26.285087 25359 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:26.265424 25209 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:26.315248 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:26.223991 31403 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:26.264346 31721 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:26.226401 31825 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:26.319747 25359 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:26.336272 25359 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:26.337376 25909 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:26.340773 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 166279us
I20260709 13:10:26.341774 25909 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:26.311169 25243 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:26.287998 31674 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:26.346885 25359 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:26.287320 31571 raft_consensus.cc:697] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:26.355523 25909 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:26.383675 25909 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:26.361961 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:26.370645 31798 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:26.385113 25909 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:26.384326 25359 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:26.326117 31154 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:26.391578 25359 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:26.392206 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:26.397382 31800 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:26.397518 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:26.398031 31800 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:26.398259 31791 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:26.398451 31800 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:26.398762 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:26.355620 31340 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:26.349730 31796 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:26.354833 31848 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 13:10:26.405612 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:26.399396 25390 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:26.314515 31390 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:26.360954 31730 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:26.399914 31810 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:26.316958 31478 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:26.291692 31827 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:26.419045 25245 consensus_meta.cc:287] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.524s	user 0.006s	sys 0.003s
I20260709 13:10:26.423782 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:26.435254 25245 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13.
I20260709 13:10:26.426054 25359 leader_election.cc:400] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 19. Current term is 20.
I20260709 13:10:26.423507 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:26.393383 31798 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:26.428509 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:26.407323 31842 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:10:26.437265 25939 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:26.430130 31577 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:26.438333 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:26.438012 31577 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:26.439463 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:26.440024 25390 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:26.440182 31403 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:26.442262 25238 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 12. Current term is 13. Ops: []
I20260709 13:10:26.442610 25359 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:26.437950 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:26.394096 31853 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:26.400882 31674 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:26.417904 31340 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:26.440768 31577 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:26.459725 31718 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:26.460865 31390 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:26.517765 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 75242.1us
I20260709 13:10:26.463550 31829 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.033s
I20260709 13:10:26.481279 31340 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:26.431092 31810 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:26.495915 31805 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:26.616271 31810 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:26.495609 31789 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:26.594692 25394 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41830 (request call id 1330) took 1339 ms (client timeout 1519 ms). Trace:
W20260709 13:10:26.617249 25394 rpcz_store.cc:269] 0709 13:10:25.164580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:25.166487 (+  1907us) service_pool.cc:224] Handling call
0709 13:10:26.503799 (+1337312us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54001211d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:26.503968 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2303848704}
I20260709 13:10:26.618541 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:26.619683 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:26.600684 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:26.607352 25386 raft_consensus.cc:2410] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 13 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260709 13:10:26.518600 31718 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:26.621485 31718 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:26.629375 31829 consensus_queue.cc:1243] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 15.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:10:26.621260 31732 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:26.494590 31571 consensus_queue.cc:237] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:26.492786 31768 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:26.637537 25319 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.792s	user 0.020s	sys 0.080s
I20260709 13:10:26.449216 31674 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:26.646347 25319 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.801s	user 0.020s	sys 0.080s
I20260709 13:10:26.453295 31799 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:26.647073 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:26.659807 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:26.670545 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:26.671453 25246 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:26.675123 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:26.689059 25394 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 20: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:26.693158 25143 catalog_manager.cc:5697] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 10 to 12, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 12 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:25.540314 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:10:26.655082 31674 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:26.695976 25243 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:26.662949 31386 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:26.664336 31813 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:26.700608 25143 catalog_manager.cc:5697] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 13 to 17. New cstate: current_term: 17 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:26.704001 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:26.719578 25386 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:10:26.684458 31478 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:26.653005 31732 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:26.690508 31768 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:26.724965 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:26.730144 25238 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:26.768608 25143 catalog_manager.cc:5697] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 6 to 12, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 12 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:26.770282 25143 catalog_manager.cc:5697] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 12 to 18, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 18 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:10:26.782122 25395 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.816s	user 0.034s	sys 0.097s
W20260709 13:10:26.782793 25395 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.817s	user 0.034s	sys 0.097s
I20260709 13:10:26.783576 25143 catalog_manager.cc:5697] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 11 to 15, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 15 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:26.812383 25246 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
W20260709 13:10:26.814790 25241 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.836s	user 0.012s	sys 0.031s
W20260709 13:10:26.868333 25241 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.889s	user 0.012s	sys 0.031s
W20260709 13:10:26.817620 25244 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.994s	user 0.024s	sys 0.067s
I20260709 13:10:26.709594 31848 consensus_queue.cc:1059] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.064s
I20260709 13:10:26.834093 25245 pending_rounds.cc:85] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 9
I20260709 13:10:26.869596 25245 pending_rounds.cc:107] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 15.10
I20260709 13:10:26.869958 25245 raft_consensus.cc:2889] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:26.838532 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:26.771405 31478 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:26.765511 31753 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.803s	user 0.017s	sys 0.030s
I20260709 13:10:26.842095 25386 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:26.773869 31390 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:26.846987 25393 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.840s	user 0.030s	sys 0.082s
I20260709 13:10:26.814107 31722 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:10:26.872632 25244 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.049s	user 0.025s	sys 0.067s
W20260709 13:10:26.873224 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:26.903863 25316 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.877s	user 0.026s	sys 0.049s
W20260709 13:10:26.925020 25316 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.898s	user 0.026s	sys 0.050s
W20260709 13:10:26.842809 31151 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.706s	user 0.025s	sys 0.021s
W20260709 13:10:26.918936 25393 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.912s	user 0.030s	sys 0.082s
W20260709 13:10:26.903434 31753 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.941s	user 0.018s	sys 0.031s
I20260709 13:10:26.816632 31768 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:26.924335 31862 consensus_meta.cc:287] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.862s	user 0.010s	sys 0.001s
I20260709 13:10:26.955500 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:26.927613 31151 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.025s	sys 0.022s
W20260709 13:10:26.951332 31793 consensus_meta.cc:287] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.944s	user 0.008s	sys 0.004s
I20260709 13:10:26.964812 25359 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:27.061288 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 96564us
W20260709 13:10:27.038203 25941 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.042s	user 0.005s	sys 0.031s
W20260709 13:10:27.062685 25941 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.066s	user 0.005s	sys 0.031s
I20260709 13:10:26.927467 31478 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:26.955679 31792 consensus_meta.cc:287] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.909s	user 0.003s	sys 0.011s
I20260709 13:10:26.938306 31223 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:10:26.992525 25141 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39572 (request call id 291) took 59086 ms (client timeout 15000 ms). Trace:
I20260709 13:10:26.996560 25319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 1114) took 1180 ms. Trace:
I20260709 13:10:27.062592 25359 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:26.927636 31390 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.067772 25939 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:27.069623 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:27.087513 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:10:27.036155 31588 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.765s	user 0.010s	sys 0.015s
W20260709 13:10:27.008600 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:27.084941 25212 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:27.094448 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:27.001974 31732 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:27.101359 25212 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:27.103775 25212 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:27.101493 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:27.063535 31862 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:27.077930 31813 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:27.084237 25319 rpcz_store.cc:276] 0709 13:10:25.816575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:25.816793 (+   218us) service_pool.cc:224] Handling call
0709 13:10:25.817109 (+   316us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:25.826137 (+  9028us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:25.826168 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:25.834965 (+  8797us) log.cc:844] Serialized 26 byte log entry
0709 13:10:26.652749 (+817784us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:26.652804 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:26.652904 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:26.996234 (+343330us) raft_consensus.cc:1685] finished
0709 13:10:26.996267 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:26.996480 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14627,"raft.queue_time_us":12293,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":3046,"thread_start_us":806660,"threads_started":1,"wal-append.queue_time_us":1092971}
I20260709 13:10:27.114143 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:26.963470 31340 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:27.095113 31588 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.824s	user 0.011s	sys 0.015s
I20260709 13:10:27.115037 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:27.027865 31810 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:26.993803 31782 consensus_queue.cc:1059] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.102s
I20260709 13:10:27.080087 31796 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:27.121743 25940 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:27.122100 25212 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:27.116541 31872 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.234s	user 0.000s	sys 0.001s
W20260709 13:10:27.129335 31823 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.839s	user 0.030s	sys 0.066s
I20260709 13:10:27.099799 31577 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 13:10:27.116348 31873 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260709 13:10:27.115943 31869 log.cc:927] Time spent T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.301s	user 0.001s	sys 0.000s
I20260709 13:10:27.111122 31742 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:27.109953 31151 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:27.166857 31151 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:27.167891 31151 raft_consensus.cc:2764] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 13:10:27.090677 31792 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:27.114965 31813 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:27.169994 25387 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 16: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:10:27.171238 25285 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.939s	user 0.024s	sys 0.037s
W20260709 13:10:27.172243 25285 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.940s	user 0.024s	sys 0.037s
I20260709 13:10:27.137604 25386 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 19.
I20260709 13:10:27.100279 31861 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 13:10:27.124152 31862 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:27.129581 31660 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:27.139369 25287 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.891s	user 0.001s	sys 0.034s
W20260709 13:10:27.192391 25287 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.944s	user 0.002s	sys 0.035s
I20260709 13:10:27.132165 31789 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:27.118526 31774 consensus_queue.cc:1059] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.022s
I20260709 13:10:27.134259 31588 raft_consensus.cc:697] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:27.152089 25212 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:27.150018 25944 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:45428
I20260709 13:10:27.156688 25395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 2321) took 1911 ms. Trace:
W20260709 13:10:27.159941 31823 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.869s	user 0.030s	sys 0.066s
W20260709 13:10:27.192812 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 959585us
I20260709 13:10:27.157653 31796 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.196759 25395 rpcz_store.cc:276] 0709 13:10:25.249361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:25.250671 (+  1310us) service_pool.cc:224] Handling call
0709 13:10:25.251124 (+   453us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:25.961446 (+710322us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:25.961477 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:25.965641 (+  4164us) log.cc:844] Serialized 52 byte log entry
0709 13:10:26.783381 (+817740us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:10:26.783420 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:26.783490 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:27.156370 (+372880us) raft_consensus.cc:1685] finished
0709 13:10:27.156404 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:27.156609 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":709880,"raft.queue_time_us":710333,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":125,"thread_start_us":817273,"threads_started":1,"wal-append.queue_time_us":1188763}
I20260709 13:10:27.213022 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2651) took 1464 ms. Trace:
W20260709 13:10:27.202239 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 50252.5us
I20260709 13:10:27.215334 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 2329) took 1172 ms. Trace:
W20260709 13:10:27.084080 25141 rpcz_store.cc:269] 0709 13:09:27.764586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:27.765354 (+   768us) service_pool.cc:224] Handling call
0709 13:10:26.842316 (+59076962us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:10:12.110906 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:10:12.143348 (+ 32442us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:10:12.143421 (+    73us) write_op.cc:435] Acquired schema lock
0709 13:10:12.143441 (+    20us) tablet.cc:662] Decoding operations
0709 13:10:12.144376 (+   935us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:10:12.144485 (+   109us) write_op.cc:641] Partition lock acquired for write op
0709 13:10:12.144518 (+    33us) tablet.cc:685] Acquiring locks for 7 operations
0709 13:10:12.255042 (+110524us) tablet.cc:701] Row locks acquired
0709 13:10:12.304135 (+ 49093us) write_op.cc:260] PREPARE: finished
0709 13:10:12.331714 (+ 27579us) write_op.cc:270] Start()
0709 13:10:12.344347 (+ 12633us) write_op.cc:276] Timestamp: P: 1783602612323684 usec, L: 0
0709 13:10:12.361195 (+ 16848us) op_driver.cc:348] REPLICATION: starting
0709 13:10:12.366938 (+  5743us) log.cc:844] Serialized 3111 byte log entry
0709 13:10:13.219260 (+852322us) op_driver.cc:464] REPLICATION: finished
0709 13:10:13.923349 (+704089us) write_op.cc:301] APPLY: starting
0709 13:10:13.933874 (+ 10525us) tablet.cc:1367] starting BulkCheckPresence
0709 13:10:14.132468 (+198594us) tablet.cc:1370] finished BulkCheckPresence
0709 13:10:14.132590 (+   122us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:10:14.779616 (+647026us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b18002ac100. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f29769f10ec 00007f2976a3dc0a 00007f2976a3d4f7 00007f2976a3d490 00007f29769f4f48 00007f297a7091bb 00007f297a7505d4 00007f297a5fc8da 00007f297a5ff3cb
0709 13:10:14.784307 (+  4691us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:10:14.784723 (+   416us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=14,key_file_lookups=14,delta_file_lookups=14,mrs_lookups=0
0709 13:10:14.784795 (+    72us) write_op.cc:312] APPLY: finished
0709 13:10:14.792964 (+  8169us) log.cc:844] Serialized 73 byte log entry
0709 13:10:14.896778 (+103814us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:10:14.896905 (+   127us) write_op.cc:454] Released schema lock
0709 13:10:14.896975 (+    70us) write_op.cc:341] FINISH: Updating metrics
0709 13:10:14.943215 (+ 46240us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:10:14.943252 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":612198,"reported_tablets":46,"child_traces":[["op",{"apply.queue_time_us":580451,"apply.run_cpu_time_us":36448,"apply.run_wall_time_us":1044110,"cfile_cache_hit":30,"cfile_cache_hit_bytes":33359,"num_ops":7,"prepare.queue_time_us":225617,"prepare.run_cpu_time_us":31919,"prepare.run_wall_time_us":611262,"raft.queue_time_us":334333,"raft.run_cpu_time_us":22942,"raft.run_wall_time_us":387757,"replication_time_us":857886,"spinlock_wait_cycles":30666496,"thread_start_us":1054009,"threads_started":5,"wal-append.queue_time_us":863342,"wal-append.run_cpu_time_us":23244,"wal-append.run_wall_time_us":2496518}]]}
I20260709 13:10:27.199434 31580 raft_consensus.cc:3055] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:27.206674 31774 raft_consensus.cc:2764] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 13:10:27.172513 31813 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:27.208202 31151 raft_consensus.cc:3055] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:27.171344 31577 raft_consensus.cc:697] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:10:27.208302 31718 consensus_meta.cc:287] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.586s	user 0.009s	sys 0.000s
I20260709 13:10:27.233731 25241 rpcz_store.cc:276] 0709 13:10:25.748532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:25.754693 (+  6161us) service_pool.cc:224] Handling call
0709 13:10:25.754910 (+   217us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:25.836974 (+ 82064us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:10:25.978874 (+141900us) log.cc:844] Serialized 14 byte log entry
0709 13:10:26.891680 (+912806us) log.cc:844] Serialized 14 byte log entry
0709 13:10:26.891944 (+   264us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:26.892011 (+    67us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:10:26.892075 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:27.212780 (+320705us) spinlock_profiling.cc:243] Waited 81.8 ms on lock 0x7b54002b2354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:27.212832 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:27.212960 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":141578,"prepare.queue_time_us":157286,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":74,"raft.queue_time_us":71912,"raft.run_cpu_time_us":272,"raft.run_wall_time_us":101690,"spinlock_wait_cycles":198687872,"thread_start_us":889749,"threads_started":1,"wal-append.queue_time_us":1113539}
I20260709 13:10:27.220644 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:27.239020 25212 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:27.217779 31459 raft_consensus.cc:3055] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:27.238032 25387 rpcz_store.cc:276] 0709 13:10:26.042890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:26.132549 (+ 89659us) service_pool.cc:224] Handling call
0709 13:10:27.215061 (+1082512us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400234954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:27.215255 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2382560128}
I20260709 13:10:27.225023 25285 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:27.261797 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 1117) took 1439 ms. Trace:
I20260709 13:10:27.269711 25315 rpcz_store.cc:276] 0709 13:10:25.822680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:25.822910 (+   230us) service_pool.cc:224] Handling call
0709 13:10:25.823287 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:25.824680 (+  1393us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:25.824707 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:25.828217 (+  3510us) log.cc:844] Serialized 26 byte log entry
0709 13:10:26.133390 (+305173us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:10:26.133427 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:26.133487 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:27.261496 (+1128009us) raft_consensus.cc:1685] finished
0709 13:10:27.261526 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:27.261726 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":963,"raft.queue_time_us":2521,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":3987,"thread_start_us":280408,"threads_started":1,"wal-append.queue_time_us":838172}
I20260709 13:10:27.263125 31792 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:27.263239 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:27.263813 31877 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260709 13:10:27.183881 31674 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:27.195962 31789 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:27.199916 31660 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:27.272308 31798 consensus_meta.cc:287] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.833s	user 0.003s	sys 0.014s
W20260709 13:10:27.268304 25237 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.254s	user 0.040s	sys 0.102s
I20260709 13:10:27.270929 31643 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:10:27.268006 31718 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:27.239527 31580 raft_consensus.cc:740] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:10:27.276381 25911 leader_election.cc:336] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.534s (SENT)
I20260709 13:10:27.247786 31151 raft_consensus.cc:740] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:27.196599 31862 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.295126 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:27.297611 25313 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:10:27.240919 31762 consensus_meta.cc:287] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.175s	user 0.012s	sys 0.002s
I20260709 13:10:27.269095 31459 raft_consensus.cc:740] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:10:27.281934 25237 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.268s	user 0.040s	sys 0.102s
I20260709 13:10:27.268603 31768 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:27.273838 31478 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:27.306984 25394 raft_consensus.cc:2393] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 20: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:27.307687 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:27.308526 25317 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:27.207326 31588 consensus_queue.cc:237] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.293758 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:27.279385 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 56713.7us
I20260709 13:10:27.314194 25312 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:27.279048 31799 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:27.281422 31806 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:10:27.280164 31774 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20260709 13:10:27.304015 25315 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260709 13:10:27.287737 31800 consensus_meta.cc:287] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.889s	user 0.014s	sys 0.002s
I20260709 13:10:27.306727 31792 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:27.283142 31803 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:27.315459 25314 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.968s	user 0.006s	sys 0.058s
I20260709 13:10:27.315886 25944 raft_consensus.cc:606] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:27.326927 25314 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.979s	user 0.007s	sys 0.058s
I20260709 13:10:27.327021 25944 raft_consensus.cc:614] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:10:27.287204 31340 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:27.321993 25285 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:27.316447 31459 consensus_queue.cc:260] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.310654 31768 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:27.322655 31580 consensus_queue.cc:260] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:27.324048 31718 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:27.284549 31798 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:27.317337 31874 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.174s	user 0.001s	sys 0.001s
I20260709 13:10:27.335847 25321 raft_consensus.cc:1240] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:27.335167 31459 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:27.319087 31822 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:27.330403 31806 raft_consensus.cc:697] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 20 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:27.325213 31794 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:27.340961 25240 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.991s	user 0.027s	sys 0.040s
I20260709 13:10:27.326572 31810 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:27.347339 25313 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
W20260709 13:10:27.352399 25239 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.932s	user 0.036s	sys 0.043s
I20260709 13:10:27.330175 31803 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:27.353948 25389 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.007s	sys 0.043s
W20260709 13:10:27.355063 25389 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.007s	sys 0.043s
W20260709 13:10:27.355201 25239 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.935s	user 0.036s	sys 0.043s
I20260709 13:10:27.360872 31803 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:27.361744 31803 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:27.361923 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:27.362200 31803 raft_consensus.cc:697] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:27.363077 31803 consensus_queue.cc:237] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:27.330287 31792 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.365268 25312 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 13:10:27.336365 31768 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:27.367422 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2667) took 1213 ms. Trace:
I20260709 13:10:27.367614 31768 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:27.369007 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:27.369801 31879 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 13:10:27.369990 25319 raft_consensus.cc:2393] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 18: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:27.367985 25239 rpcz_store.cc:276] 0709 13:10:26.154522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:26.392766 (+238244us) service_pool.cc:224] Handling call
0709 13:10:26.393058 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:26.393301 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:10:26.419946 (+ 26645us) log.cc:844] Serialized 14 byte log entry
0709 13:10:27.367028 (+947082us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:27.367095 (+    67us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:27.367153 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:27.367217 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:27.367354 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":30018,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":102,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":77,"thread_start_us":937509,"threads_started":1}
I20260709 13:10:27.371066 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 1345) took 1372 ms. Trace:
I20260709 13:10:27.371532 25393 rpcz_store.cc:276] 0709 13:10:25.998561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:25.999258 (+   697us) service_pool.cc:224] Handling call
0709 13:10:25.999486 (+   228us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:26.006150 (+  6664us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:26.006180 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:26.006662 (+   482us) log.cc:844] Serialized 26 byte log entry
0709 13:10:26.928306 (+921644us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:26.928361 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:26.928452 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:27.370797 (+442345us) raft_consensus.cc:1685] finished
0709 13:10:27.370834 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:27.370998 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2934,"raft.queue_time_us":11968,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":12149,"thread_start_us":921245,"threads_started":1,"wal-append.queue_time_us":1192601}
I20260709 13:10:27.376021 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:27.377279 31742 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:27.379127 31792 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:27.381394 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:27.383025 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 1121) took 1384 ms. Trace:
I20260709 13:10:27.383435 25316 rpcz_store.cc:276] 0709 13:10:25.998998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:25.999114 (+   116us) service_pool.cc:224] Handling call
0709 13:10:25.999409 (+   295us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:25.999700 (+   291us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:25.999728 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:26.008196 (+  8468us) log.cc:844] Serialized 26 byte log entry
0709 13:10:26.926400 (+918204us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:26.926457 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:26.926550 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:27.382814 (+456264us) raft_consensus.cc:1685] finished
0709 13:10:27.382841 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:27.382970 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3992,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":6569,"thread_start_us":899214,"threads_started":1,"wal-append.queue_time_us":1315182}
W20260709 13:10:27.345779 25240 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.003s	user 0.027s	sys 0.040s
I20260709 13:10:27.326265 31800 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:27.315653 31577 consensus_queue.cc:237] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:27.423552 25908 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:27.425724 31861 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:27.426187 31861 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:27.426537 31861 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:27.429931 31732 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:10:27.431501 31798 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:27.440109 31660 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:27.398277 25390 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.957s	user 0.027s	sys 0.043s
I20260709 13:10:27.325912 31799 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:27.442233 31882 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:10:27.442364 31386 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:27.416208 31580 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:27.443070 31386 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:27.444128 31799 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:27.445065 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:27.446290 25390 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.005s	user 0.027s	sys 0.043s
I20260709 13:10:27.457564 31390 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:27.457823 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:27.347879 31822 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.424979 31827 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:27.349154 31794 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:27.479508 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:27.467206 31459 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:27.465083 31796 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:27.467890 31674 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:27.497025 31459 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.499676 31459 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:27.493273 25392 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:27.473737 31711 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:27.544355 31643 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:27.562194 31459 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:27.588742 31889 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:27.588801 31459 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.589704 31889 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:27.570639 31883 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:27.592607 31643 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.593452 31883 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.513021 31768 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:27.450462 31806 consensus_queue.cc:237] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:27.520762 31711 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.573010 31478 raft_consensus.cc:3055] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:27.631857 25940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 1230) took 1525 ms. Trace:
I20260709 13:10:27.581883 31774 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:27.649729 31887 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 13:10:27.599223 31889 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:27.585686 31742 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:27.620702 31862 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:27.642895 25940 rpcz_store.cc:276] 0709 13:10:26.106343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:26.106988 (+   645us) service_pool.cc:224] Handling call
0709 13:10:26.107263 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:27.149845 (+1042582us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:27.631458 (+481613us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54006905d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:27.631724 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2331890432}
W20260709 13:10:27.642305 31825 consensus_meta.cc:287] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.272s	user 0.007s	sys 0.007s
I20260709 13:10:27.642699 31478 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:27.653596 31800 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:27.690153 25245 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.731s	user 0.016s	sys 0.010s
W20260709 13:10:27.690600 25245 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.016s	sys 0.010s
W20260709 13:10:27.697892 25941 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.634s	user 0.007s	sys 0.020s
W20260709 13:10:27.698411 25941 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.007s	sys 0.020s
I20260709 13:10:27.702698 31846 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:27.703794 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1228) took 1709 ms. Trace:
I20260709 13:10:27.676676 31808 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:27.679930 31889 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:27.632465 31768 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:27.687037 31730 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.234s	user 0.017s	sys 0.037s
W20260709 13:10:27.706821 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 375818us
I20260709 13:10:27.719079 31478 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:27.754822 25359 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.032s	sys 0.039s
I20260709 13:10:27.675563 31813 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:27.714669 25360 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.014s	sys 0.032s
W20260709 13:10:27.756019 25360 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.014s	sys 0.032s
I20260709 13:10:27.760562 25386 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:27.760803 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:27.762248 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:27.762614 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:27.743438 25941 rpcz_store.cc:276] 0709 13:10:25.994840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:25.994982 (+   142us) service_pool.cc:224] Handling call
0709 13:10:25.995400 (+   418us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:25.995879 (+   479us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:10:25.996246 (+   367us) log.cc:844] Serialized 14 byte log entry
0709 13:10:27.698896 (+1702650us) log.cc:844] Serialized 14 byte log entry
0709 13:10:27.700205 (+  1309us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:27.700828 (+   623us) log.cc:844] Serialized 26 byte log entry
0709 13:10:27.702408 (+  1580us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:27.702461 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:27.703465 (+  1004us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:27.703531 (+    66us) raft_consensus.cc:1685] finished
0709 13:10:27.703549 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:27.703722 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3975,"prepare.queue_time_us":5626,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":1129,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":939,"thread_start_us":1701978,"threads_started":2,"wal-append.queue_time_us":1431853}
I20260709 13:10:27.763391 25241 raft_consensus.cc:2393] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 18: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:10:27.708949 31753 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.008s	sys 0.017s
I20260709 13:10:27.677428 31774 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.681504 31825 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:27.679013 31742 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.712787 31794 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:27.755481 31730 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.304s	user 0.017s	sys 0.038s
W20260709 13:10:27.728013 25244 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.798s	user 0.009s	sys 0.021s
I20260709 13:10:27.714665 31223 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:27.739531 25209 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.421s	user 0.024s	sys 0.029s
W20260709 13:10:27.798056 25209 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.480s	user 0.024s	sys 0.029s
W20260709 13:10:27.798759 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 1.53331e+06us
W20260709 13:10:27.733338 25143 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.018s	sys 0.003s
W20260709 13:10:27.799623 25143 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.019s	sys 0.003s
I20260709 13:10:27.728080 31643 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:27.742106 31891 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 13:10:27.685447 31390 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:27.696863 31880 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.153s	user 0.000s	sys 0.001s
W20260709 13:10:27.755637 25359 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.033s	sys 0.040s
I20260709 13:10:27.770103 25387 raft_consensus.cc:1240] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
W20260709 13:10:27.839391 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 740486us
I20260709 13:10:27.801910 25394 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:27.786239 31478 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:27.787256 25388 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:27.685029 31151 consensus_queue.cc:260] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.753252 31789 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:27.785481 25358 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.678s	user 0.009s	sys 0.017s
W20260709 13:10:27.907655 25358 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.800s	user 0.010s	sys 0.017s
I20260709 13:10:27.907716 31151 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:10:27.796011 25244 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.872s	user 0.011s	sys 0.022s
W20260709 13:10:27.765518 31753 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.008s	sys 0.017s
I20260709 13:10:27.908991 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2655) took 2157 ms. Trace:
I20260709 13:10:27.783819 31774 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:27.802126 25209 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:27.910809 31769 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:10:27.911260 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 109230us
I20260709 13:10:27.911695 31769 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:27.912056 25209 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:27.848018 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:27.912132 31769 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:27.913568 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:27.914149 25239 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:10:27.914428 31768 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:27.914979 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:27.915107 31768 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:27.915594 25239 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:27.916415 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:27.744776 31883 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:27.851444 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:27.850148 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:27.852169 31390 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.917626 25246 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 16: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:27.801463 31660 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:27.918772 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:27.918661 31660 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.919210 31390 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:27.919358 25246 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:27.854599 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:27.921533 25285 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:27.922385 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:27.921000 31769 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:27.860107 31459 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:27.869961 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:44966 (request call id 3) took 1637 ms. Trace:
W20260709 13:10:27.873740 25392 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41830 (request call id 1339) took 1903 ms (client timeout 1534 ms). Trace:
I20260709 13:10:27.922986 25238 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:27.923045 31883 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:27.875021 25237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2659) took 2080 ms. Trace:
I20260709 13:10:27.923599 31883 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:27.856745 31822 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:27.924154 31883 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:27.917609 25241 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:27.879842 31853 consensus_queue.cc:1059] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:10:27.923755 25237 rpcz_store.cc:276] 0709 13:10:25.794627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:25.983911 (+189284us) service_pool.cc:224] Handling call
0709 13:10:25.984203 (+   292us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:26.009369 (+ 25166us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:26.009401 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:26.013474 (+  4073us) log.cc:844] Serialized 26 byte log entry
0709 13:10:27.299446 (+1285972us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:27.320316 (+ 20870us) log.cc:844] Serialized 14 byte log entry
0709 13:10:27.320539 (+   223us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:27.320630 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:27.874723 (+554093us) raft_consensus.cc:1685] finished
0709 13:10:27.874754 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:27.874946 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18071,"prepare.queue_time_us":21143,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":93,"raft.queue_time_us":23501,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":11041,"thread_start_us":1285543,"threads_started":1,"wal-append.queue_time_us":1471012}
I20260709 13:10:27.934414 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:27.934986 25237 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:27.935832 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:27.789043 31730 raft_consensus.cc:697] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:27.802645 31223 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:27.885098 31643 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:27.888698 31862 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:27.893509 31825 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:27.920909 31660 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:27.900023 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:27.903411 31742 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:27.909364 25244 rpcz_store.cc:276] 0709 13:10:25.751400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:25.786144 (+ 34744us) service_pool.cc:224] Handling call
0709 13:10:25.786547 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:25.786808 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:10:25.811184 (+ 24376us) log.cc:844] Serialized 14 byte log entry
0709 13:10:27.908621 (+2097437us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:27.908673 (+    52us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:27.908725 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:27.908799 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:27.908909 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":8509,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":104124,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":71,"thread_start_us":2097796,"threads_started":3,"wal-append.queue_time_us":1309674}
W20260709 13:10:27.908146 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 829915us
I20260709 13:10:27.922521 25285 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:27.876608 31846 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:27.883621 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 1351) took 1193 ms. Trace:
W20260709 13:10:27.923295 25392 rpcz_store.cc:269] 0709 13:10:25.970267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:25.972705 (+  2438us) service_pool.cc:224] Handling call
0709 13:10:27.873458 (+1900753us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400323cd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:27.873643 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3022861568}
I20260709 13:10:27.925271 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:27.966974 25388 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:27.927672 25209 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:27.923975 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:27.969209 25238 raft_consensus.cc:2410] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 13:10:27.917037 25239 raft_consensus.cc:2368] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 19. Current term is 20.
I20260709 13:10:27.856046 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:27.924062 25391 rpcz_store.cc:276] 0709 13:10:26.232911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:26.267777 (+ 34866us) service_pool.cc:224] Handling call
0709 13:10:27.869604 (+1601827us) spinlock_profiling.cc:243] Waited 905 ms on lock 0x7b5400234954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb8735 00007f297bc78d29 00007f29751f71d4 00007f29751f7162 00007f29751f7091 00007f29751f700c 00007f29751f62c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:27.869844 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2080466304}
I20260709 13:10:27.754106 31808 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.755301 31889 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:27.966018 25389 rpcz_store.cc:276] 0709 13:10:26.689848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:26.690056 (+   208us) service_pool.cc:224] Handling call
0709 13:10:26.690461 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:26.691614 (+  1153us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:26.691656 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:26.692313 (+   657us) log.cc:844] Serialized 26 byte log entry
0709 13:10:27.355696 (+663383us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:27.355754 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:27.355852 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:27.883344 (+527492us) raft_consensus.cc:1685] finished
0709 13:10:27.883377 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:27.883540 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":522,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":6212,"thread_start_us":587730,"threads_started":1,"wal-append.queue_time_us":814814}
W20260709 13:10:27.979031 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 51445.9us
W20260709 13:10:27.977572 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 55121.9us
W20260709 13:10:27.938941 25284 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.023s	sys 0.009s
I20260709 13:10:27.794416 31773 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:27.801262 31823 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:27.937371 31390 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:27.942196 31711 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:27.950771 31853 raft_consensus.cc:3055] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:27.941905 31774 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:27.899062 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:27.955974 31850 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:28.002092 25394 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:28.003481 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:27.892305 31861 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.008630 31861 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:28.011442 31774 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:27.977646 31846 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:28.014281 31774 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:27.977535 31154 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:28.015033 31154 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:28.018155 25237 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:27.956252 25315 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.013s	sys 0.015s
I20260709 13:10:27.958061 25246 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:27.905215 31813 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:27.962568 25314 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.009s	sys 0.029s
W20260709 13:10:28.019485 25314 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.009s	sys 0.029s
W20260709 13:10:27.926105 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:27.972968 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:27.981352 25209 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:27.999617 31732 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.020s	sys 0.067s
W20260709 13:10:28.023152 31732 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.020s	sys 0.067s
I20260709 13:10:27.968278 31825 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:28.025914 25314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2841) took 1811 ms. Trace:
I20260709 13:10:28.026686 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:27.981808 25284 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.023s	sys 0.009s
W20260709 13:10:28.029052 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 708431us
W20260709 13:10:28.032833 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:27.987023 25388 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 2330) took 1862 ms (client timeout 1777 ms). Trace:
W20260709 13:10:27.962890 31803 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.011s	sys 0.027s
W20260709 13:10:27.993203 25240 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.016s	sys 0.025s
W20260709 13:10:28.048892 25240 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.017s	sys 0.025s
I20260709 13:10:27.992986 25285 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:10:27.958612 31588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.003s	sys 0.025s
I20260709 13:10:27.997695 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:27.998543 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:27.956274 31862 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:28.048676 25388 rpcz_store.cc:269] 0709 13:10:26.124833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:26.160266 (+ 35433us) service_pool.cc:224] Handling call
0709 13:10:27.986776 (+1826510us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54003d4ad4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:27.986935 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3739224320}
I20260709 13:10:27.997551 31823 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:28.005613 25911 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:28.078830 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 73329.9us
I20260709 13:10:28.079152 25387 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:27.990530 31660 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:27.986330 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:27.985364 31742 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:28.034442 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:28.025465 25359 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:28.095180 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 69806.7us
I20260709 13:10:28.096067 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:28.096596 25359 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:28.095188 25394 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 20: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:28.017081 31154 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:27.953953 31386 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.044749 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:28.112234 25941 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:27.956563 31643 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.045547 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:27.967648 31810 consensus_queue.cc:1059] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.059s
I20260709 13:10:28.075774 25388 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:28.124233 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:28.050957 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 58029.7us
I20260709 13:10:28.051405 31768 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:28.125823 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:28.126603 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:28.126914 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:28.125954 31768 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.128103 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:28.048659 31825 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:28.131989 31825 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:28.081630 25192 auto_rebalancer.cc:629] Failed to schedule move for tablet 1b8652aecc6d4869bb4cad9e0749f3ba: Illegal state: Replica a2a77ecf2c0941618b5bc79d4c04db61 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 16 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:28.082271 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:28.083009 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:28.086107 31580 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:28.150753 31580 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.151733 31643 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:28.086361 31800 consensus_queue.cc:1059] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 13:10:28.051265 31803 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.011s	sys 0.028s
I20260709 13:10:28.092622 25239 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260709 13:10:28.038012 25315 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.013s	sys 0.015s
I20260709 13:10:28.153270 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:28.154446 25320 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:28.129591 31768 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:28.101658 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:28.026494 25314 rpcz_store.cc:276] 0709 13:10:26.214312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:26.282335 (+ 68023us) service_pool.cc:224] Handling call
0709 13:10:26.282667 (+   332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:26.283954 (+  1287us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:26.283996 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:26.284530 (+   534us) log.cc:844] Serialized 26 byte log entry
0709 13:10:27.327514 (+1042984us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:28.019925 (+692411us) log.cc:844] Serialized 14 byte log entry
0709 13:10:28.025450 (+  5525us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:28.025577 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:28.025632 (+    55us) raft_consensus.cc:1685] finished
0709 13:10:28.025654 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:28.025847 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":903,"raft.queue_time_us":59006,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":6475,"thread_start_us":1632193,"threads_started":2,"wal-append.queue_time_us":1170329}
I20260709 13:10:28.102967 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:28.105490 31813 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:27.994880 31773 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:28.107424 25359 leader_election.cc:400] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 19. Current term is 20.
I20260709 13:10:28.002383 31829 consensus_queue.cc:1059] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.079s
I20260709 13:10:28.117499 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:28.074663 31588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.004s	sys 0.030s
I20260709 13:10:28.120182 25940 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:28.033795 31850 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.134402 31386 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:28.134694 31889 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:28.107831 31861 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:28.135814 31459 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:28.136868 31862 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:28.138253 31808 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:28.076303 31886 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:28.139024 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:28.090173 31730 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.141525 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:28.143460 31883 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:28.144802 31893 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:28.079152 25392 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:28.091245 31660 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:28.146835 25317 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:28.147728 25243 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:10:28.047591 31774 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:28.122279 31887 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 13:10:28.153651 31674 raft_consensus.cc:3055] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:28.155840 25285 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:10:28.273046 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 117317us
W20260709 13:10:28.000066 25390 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.015s	sys 0.037s
W20260709 13:10:28.093868 25321 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.000s	sys 0.026s
W20260709 13:10:28.274176 25390 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.015s	sys 0.037s
W20260709 13:10:28.274277 25321 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.800s	user 0.000s	sys 0.027s
I20260709 13:10:28.003896 31853 raft_consensus.cc:740] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:28.161686 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:28.275890 25244 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:28.163812 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:28.103796 25237 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:28.165222 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:28.152902 31580 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:28.123126 31789 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:10:28.189584 25209 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:28.284679 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 95260.8us
I20260709 13:10:28.285385 25209 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:28.122861 31769 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:28.201709 25318 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:28.199225 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:28.183211 31813 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:28.206398 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:28.186913 31894 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:28.266932 31660 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:28.200846 31861 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:28.206235 25391 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 20.
W20260709 13:10:28.264303 25287 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.698s	user 0.007s	sys 0.035s
W20260709 13:10:28.344206 25287 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.778s	user 0.007s	sys 0.035s
W20260709 13:10:28.344527 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 778925us
I20260709 13:10:28.345980 25244 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
W20260709 13:10:28.266942 25909 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.710s	user 0.000s	sys 0.042s
W20260709 13:10:28.347138 25909 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.791s	user 0.000s	sys 0.042s
I20260709 13:10:28.346946 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:28.347410 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 791260us
W20260709 13:10:28.259522 31577 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.015s	sys 0.003s
I20260709 13:10:28.275310 25390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2951) took 1934 ms. Trace:
I20260709 13:10:28.204172 31848 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:28.205693 31886 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:28.278342 25246 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:28.183727 25359 leader_election.cc:403] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
W20260709 13:10:28.359658 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 252358us
I20260709 13:10:28.277495 25239 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:28.188769 31810 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:28.268643 31889 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:28.371948 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:28.291532 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:28.295873 31768 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:28.327163 25243 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: []
W20260709 13:10:28.327940 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:28.290274 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 363602us
I20260709 13:10:28.296324 31813 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:28.277606 31674 raft_consensus.cc:740] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:28.390442 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:28.389930 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:28.389465 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:28.338768 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2662) took 2202 ms. Trace:
I20260709 13:10:28.282421 31386 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:28.205564 31459 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.351977 25390 rpcz_store.cc:276] 0709 13:10:26.341183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:26.440634 (+ 99451us) service_pool.cc:224] Handling call
0709 13:10:26.440902 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:26.441100 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:10:27.446917 (+1005817us) log.cc:844] Serialized 14 byte log entry
0709 13:10:28.274862 (+827945us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:28.274931 (+    69us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:28.275003 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:28.275087 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:28.275234 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":678210,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":1012,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":6449,"thread_start_us":1832905,"threads_started":2,"wal-append.queue_time_us":874165}
I20260709 13:10:28.266636 31893 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:28.393440 25240 rpcz_store.cc:276] 0709 13:10:26.136707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:26.323261 (+186554us) service_pool.cc:224] Handling call
0709 13:10:26.323606 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:27.387682 (+1064076us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:27.387728 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:27.388336 (+   608us) log.cc:844] Serialized 26 byte log entry
0709 13:10:28.049419 (+661083us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:28.049467 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:28.049551 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:28.338553 (+289002us) raft_consensus.cc:1685] finished
0709 13:10:28.338574 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:28.338710 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":45451,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"thread_start_us":1705578,"threads_started":2,"wal-append.queue_time_us":880157}
W20260709 13:10:28.394549 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:28.395166 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:28.373335 31580 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 19. Current term is 20.
I20260709 13:10:28.407272 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:28.408660 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:28.409667 25319 raft_consensus.cc:2410] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 13:10:28.410600 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:28.411201 25319 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:28.356274 31772 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.007s	sys 0.014s
I20260709 13:10:28.395623 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 1124) took 1104 ms. Trace:
I20260709 13:10:28.352443 31861 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:28.413478 25318 rpcz_store.cc:276] 0709 13:10:27.291356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:27.291492 (+   136us) service_pool.cc:224] Handling call
0709 13:10:27.302902 (+ 11410us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:28.290045 (+987143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:28.395416 (+105371us) spinlock_profiling.cc:243] Waited 895 ms on lock 0x7b540005ee54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:28.395560 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2058381568}
I20260709 13:10:28.329753 31894 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.332787 31789 raft_consensus.cc:697] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:28.424324 25212 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:28.417719 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:28.419025 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:28.426471 25943 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 18: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:28.351814 31643 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:10:28.437434 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 145802us
I20260709 13:10:28.424571 25287 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:28.427708 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:28.421947 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:28.428661 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:28.419229 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:28.434595 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:28.367695 31886 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:28.434969 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:28.439626 25943 raft_consensus.cc:2393] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 18: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:10:28.413409 31772 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.007s	sys 0.015s
I20260709 13:10:28.440450 25939 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:28.439016 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:28.446601 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:28.452267 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:28.453418 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:28.454007 25944 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:28.455267 25945 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:28.455691 25941 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:28.447616 25942 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:28.403803 31386 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:28.457577 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:28.462634 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:28.464239 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:28.463546 25209 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:28.461369 25242 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.022s	sys 0.026s
W20260709 13:10:28.469936 25242 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.022s	sys 0.026s
I20260709 13:10:28.444710 31769 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:28.451985 31768 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:28.483450 31768 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:28.483556 31769 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:28.483990 31768 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:28.458940 31711 raft_consensus.cc:3055] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:28.485517 31711 raft_consensus.cc:740] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:28.486613 31711 consensus_queue.cc:260] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.487880 31711 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:28.391716 31810 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.491189 31810 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:28.372279 31848 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.395166 31889 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:28.470388 31386 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:28.512697 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:28.513633 25394 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 13:10:28.461798 31742 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:28.454602 31823 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:28.513113 25911 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:28.519495 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 437173us
I20260709 13:10:28.521227 25390 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:10:28.521858 25358 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:28.528193 25358 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:28.547770 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:28.556704 25238 raft_consensus.cc:1275] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:10:28.568251 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:28.521476 31883 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:28.524740 31903 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:10:28.573468 25358 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:28.445397 31853 consensus_queue.cc:260] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.525955 31742 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:28.479868 31846 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:28.555485 31895 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.354s	user 0.001s	sys 0.001s
W20260709 13:10:28.576481 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 63438.6us
I20260709 13:10:28.555015 31154 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:28.583056 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:28.585520 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:28.593436 25941 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:28.557444 31580 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:28.514679 31674 consensus_queue.cc:260] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:28.588049 31390 consensus_meta.cc:287] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.583s	user 0.012s	sys 0.000s
I20260709 13:10:28.633035 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:28.633611 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:28.636745 25390 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:28.647535 25388 raft_consensus.cc:2393] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 15: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:28.648082 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:28.648641 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:28.581758 31808 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:10:28.642059 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 51479us
I20260709 13:10:28.649538 25388 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:28.582772 31773 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:28.586822 31742 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:28.613437 31732 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:28.649042 25390 raft_consensus.cc:2393] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 20: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:28.637447 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:28.665616 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:28.631989 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:28.578111 31577 consensus_queue.cc:1059] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.066s
I20260709 13:10:28.656629 25358 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:28.587052 31898 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:28.662878 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:28.624218 31154 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:28.672501 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:28.645605 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:28.558610 31789 consensus_queue.cc:237] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:28.659273 31753 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:28.657444 31768 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:28.677093 25388 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 17: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:28.664196 31390 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:28.665536 31732 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:28.695653 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 236179us
I20260709 13:10:28.695626 25212 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:28.691223 31906 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.313s	user 0.000s	sys 0.001s
I20260709 13:10:28.712441 25319 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:28.720495 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2823) took 1389 ms. Trace:
I20260709 13:10:28.683648 31674 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:28.729993 25315 rpcz_store.cc:276] 0709 13:10:27.331027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:27.331154 (+   127us) service_pool.cc:224] Handling call
0709 13:10:27.331447 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:27.331657 (+   210us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:10:28.153899 (+822242us) log.cc:844] Serialized 14 byte log entry
0709 13:10:28.714470 (+560571us) log.cc:844] Serialized 14 byte log entry
0709 13:10:28.719995 (+  5525us) log.cc:844] Serialized 14 byte log entry
0709 13:10:28.720164 (+   169us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:28.720205 (+    41us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:28.720246 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:28.720317 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:28.720428 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":98321,"prepare.queue_time_us":958770,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":12720,"raft.queue_time_us":125849,"spinlock_wait_cycles":16704768,"thread_start_us":1272984,"threads_started":3,"wal-append.queue_time_us":504856}
I20260709 13:10:28.742049 31825 raft_consensus.cc:3055] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:28.742533 31825 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:28.743369 31825 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.744740 31825 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:28.697180 31459 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:28.705842 31732 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:28.704133 31829 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:28.690897 31929 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.060s
I20260709 13:10:28.722280 31892 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:28.810866 31782 raft_consensus.cc:3055] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:28.810580 31154 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:28.811244 31782 raft_consensus.cc:740] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:28.812104 31782 consensus_queue.cc:260] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:28.812736 31922 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
W20260709 13:10:28.819031 31774 consensus_meta.cc:287] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.541s	user 0.010s	sys 0.000s
I20260709 13:10:28.819517 31774 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:10:28.822804 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:28.825620 31918 log.cc:927] Time spent T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.251s	user 0.000s	sys 0.001s
I20260709 13:10:28.835856 31798 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:28.836644 31798 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.837934 31782 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:28.760576 31478 consensus_meta.cc:287] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.826s	user 0.000s	sys 0.012s
I20260709 13:10:28.848868 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:28.851455 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:28.805043 31895 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 13:10:28.860476 31711 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:28.861555 31920 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.268s	user 0.000s	sys 0.000s
I20260709 13:10:28.861370 31711 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:28.862419 31917 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.189s	user 0.000s	sys 0.001s
I20260709 13:10:28.862754 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2694) took 2175 ms. Trace:
I20260709 13:10:28.863659 31711 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:28.857636 25941 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8f209a6539f54de28906492f142efb6b"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:45428
I20260709 13:10:28.864401 25941 raft_consensus.cc:606] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:28.864300 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:28.864869 25941 raft_consensus.cc:614] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:10:28.865178 25246 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:28.860205 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:28.863597 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 1356) took 1105 ms. Trace:
W20260709 13:10:28.863952 31924 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.237s	user 0.000s	sys 0.001s
W20260709 13:10:28.807524 31915 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.336s	user 0.000s	sys 0.001s
W20260709 13:10:28.805501 31151 consensus_meta.cc:287] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.897s	user 0.011s	sys 0.002s
I20260709 13:10:28.866636 25393 rpcz_store.cc:276] 0709 13:10:27.758498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:27.759198 (+   700us) service_pool.cc:224] Handling call
0709 13:10:27.759529 (+   331us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:27.765987 (+  6458us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:27.766033 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:27.785767 (+ 19734us) log.cc:844] Serialized 26 byte log entry
0709 13:10:27.863801 (+ 78034us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b18004a2180. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f29768520d1 00007f2976c6da0f
0709 13:10:28.328452 (+464651us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:28.328511 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:28.328616 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:28.863336 (+534720us) raft_consensus.cc:1685] finished
0709 13:10:28.863357 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:28.863542 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6023,"raft.queue_time_us":22565,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":8943,"spinlock_wait_cycles":7299840,"thread_start_us":464407,"threads_started":1,"wal-append.queue_time_us":804816}
I20260709 13:10:28.863230 25245 rpcz_store.cc:276] 0709 13:10:26.686976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:26.833439 (+146463us) service_pool.cc:224] Handling call
0709 13:10:26.833827 (+   388us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:10:26.870743 (+ 36916us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:26.870788 (+    45us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:10:26.871607 (+   819us) log.cc:844] Serialized 78 byte log entry
0709 13:10:27.691094 (+819487us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:27.691466 (+   372us) log.cc:844] Serialized 14 byte log entry
0709 13:10:27.691822 (+   356us) log.cc:844] Serialized 14 byte log entry
0709 13:10:27.692003 (+   181us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:27.692098 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:28.862452 (+1170354us) raft_consensus.cc:1685] finished
0709 13:10:28.862482 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:28.862674 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":2251,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":409662,"raft.run_cpu_time_us":367,"raft.run_wall_time_us":45277,"thread_start_us":733771,"threads_started":1,"wal-append.queue_time_us":1220649}
W20260709 13:10:28.899247 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 435780us
I20260709 13:10:28.912701 25237 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:28.917594 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:28.918362 25243 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:28.873965 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:28.874980 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:28.758222 31829 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:28.866755 25240 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:28.744915 31850 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:10:28.804368 31810 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:28.925005 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:28.932806 25246 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 13.
I20260709 13:10:28.845624 31459 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:28.928426 25209 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:28.909547 25359 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:28.844173 31892 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:28.930161 25241 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
W20260709 13:10:28.930585 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 506050us
I20260709 13:10:28.923640 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:28.953289 25287 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:28.950904 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:28.890467 31894 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:28.898734 31931 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:28.888764 31901 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:28.889961 31889 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:28.930171 31920 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 13:10:28.923655 31577 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:28.922274 31478 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:28.928576 31711 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:28.975456 25242 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.016s	sys 0.012s
I20260709 13:10:28.935953 31769 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:28.976161 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:28.928025 31829 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:28.967402 31643 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:28.958055 31151 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:28.979748 25242 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.016s	sys 0.012s
I20260709 13:10:28.956864 31753 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:28.943711 31892 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:28.981559 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:28.981837 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:28.982657 25909 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:28.986523 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:28.987178 25209 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:28.987490 25243 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:28.952049 31903 consensus_queue.cc:1059] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 19.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.043s
W20260709 13:10:28.990370 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 57088.9us
I20260709 13:10:28.989310 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2677) took 1200 ms. Trace:
I20260709 13:10:28.953922 31810 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.993621 25285 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:28.991743 25909 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:28.987805 31643 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:28.999161 31643 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:28.993527 25242 rpcz_store.cc:276] 0709 13:10:27.788663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:27.788948 (+   285us) service_pool.cc:224] Handling call
0709 13:10:27.796961 (+  8013us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:27.797236 (+   275us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:10:27.849008 (+ 51772us) log.cc:844] Serialized 14 byte log entry
0709 13:10:28.988634 (+1139626us) log.cc:844] Serialized 14 byte log entry
0709 13:10:28.988937 (+   303us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:28.988998 (+    61us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:10:28.989044 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:28.989089 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:28.989264 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":57638,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":43,"raft.queue_time_us":978,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":42143,"spinlock_wait_cycles":7481600,"thread_start_us":1138852,"threads_started":2,"wal-append.queue_time_us":744472}
I20260709 13:10:28.973388 31459 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.970638 31901 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:29.001183 31928 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.303s	user 0.001s	sys 0.000s
W20260709 13:10:29.011066 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:28.978407 31577 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.998171 31929 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:28.968653 31931 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.990525 31940 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:28.980444 31478 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.986444 31789 consensus_queue.cc:1059] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.019s
W20260709 13:10:29.019623 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 66710.8us
I20260709 13:10:28.984406 31711 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:28.988744 31850 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.018s
I20260709 13:10:29.041818 25392 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260709 13:10:28.997690 31753 raft_consensus.cc:697] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:29.043443 31753 consensus_queue.cc:237] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:29.046232 31933 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:29.046841 31933 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.021195 31934 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:29.051273 31934 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:29.051652 31934 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:29.054625 31901 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:29.056078 31861 consensus_meta.cc:287] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.642s	user 0.004s	sys 0.004s
I20260709 13:10:29.056332 31810 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:29.057180 31927 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.306s	user 0.000s	sys 0.001s
I20260709 13:10:29.058470 31459 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:29.062570 31931 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:29.063090 31742 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:29.065279 31223 consensus_meta.cc:287] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.109s	user 0.006s	sys 0.006s
I20260709 13:10:29.065704 31223 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.021296 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:29.003105 31889 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:28.989607 31769 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:29.015345 31916 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.319s	user 0.000s	sys 0.001s
W20260709 13:10:29.090802 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 69608us
I20260709 13:10:29.090984 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:29.027469 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 74275.5us
I20260709 13:10:29.003315 31730 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:29.030287 31478 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:29.031358 31660 consensus_meta.cc:287] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.663s	user 0.009s	sys 0.000s
I20260709 13:10:29.031620 31799 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:29.038240 25941 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:29.037384 31932 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:29.056555 31861 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:29.026640 31930 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
W20260709 13:10:29.089458 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 50302.7us
I20260709 13:10:29.090015 25243 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 13:10:29.093277 25287 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:29.096529 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2826) took 1737 ms. Trace:
I20260709 13:10:29.101769 25209 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:29.048987 31933 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:29.106688 25246 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:29.099035 31730 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:29.095774 31742 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:29.128455 25359 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:29.130990 25209 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:29.133095 25359 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:29.134171 25942 raft_consensus.cc:2368] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16.
I20260709 13:10:29.134626 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:29.100824 31950 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:29.104991 31771 consensus_queue.cc:1059] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 19.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20260709 13:10:29.128113 25321 rpcz_store.cc:276] 0709 13:10:27.359634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:27.360055 (+   421us) service_pool.cc:224] Handling call
0709 13:10:27.360394 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:27.464390 (+103996us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 13:10:28.274937 (+810547us) log.cc:844] Serialized 14 byte log entry
0709 13:10:28.551358 (+276421us) log.cc:844] Serialized 14 byte log entry
0709 13:10:28.551535 (+   177us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:28.551578 (+    43us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:10:28.551638 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:29.096257 (+544619us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54000631d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:29.096314 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:29.096457 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":314,"prepare.queue_time_us":859324,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":67,"raft.queue_time_us":8053,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":18698,"spinlock_wait_cycles":251168896,"thread_start_us":1076064,"threads_started":2,"wal-append.queue_time_us":439474}
I20260709 13:10:29.140506 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:29.140750 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:29.140677 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:29.143971 25315 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 17: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:29.141407 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:29.132238 31730 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:29.141623 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:29.128245 31929 raft_consensus.cc:697] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:29.129690 31933 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:29.106397 31660 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.135818 31825 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:29.150566 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:29.133965 31478 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:10:29.136005 31940 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:29.152824 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:29.155536 31861 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:29.137466 31223 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:29.160477 25284 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:29.161753 25284 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:29.168437 31937 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:10:29.124346 31932 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.169087 31937 raft_consensus.cc:697] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 20 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:29.170260 31660 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:29.170892 31932 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:29.170423 31937 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.171298 25312 raft_consensus.cc:2368] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 18. Current term is 19.
I20260709 13:10:29.172204 31929 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.138780 31154 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:29.173584 31154 raft_consensus.cc:697] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:29.174464 31154 consensus_queue.cc:237] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.137898 31588 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:10:29.177595 31588 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:10:29.178167 31886 consensus_meta.cc:287] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.714s	user 0.008s	sys 0.002s
I20260709 13:10:29.178606 31886 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.180791 31886 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:29.199551 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:29.152453 31813 consensus_meta.cc:287] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.727s	user 0.010s	sys 0.003s
I20260709 13:10:29.162420 31825 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:29.194725 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:29.194311 31948 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:29.211638 31897 log.cc:927] Time spent T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 1.010s	user 0.005s	sys 0.009s
I20260709 13:10:29.138554 31950 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:29.184481 31954 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:29.218788 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:29.197005 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:29.142678 31941 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:29.181939 31577 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:29.185688 31711 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:29.195961 31934 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:29.243121 31934 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:29.181859 31643 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.246227 31643 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:29.247041 25321 raft_consensus.cc:2368] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 20. Current term is 21.
W20260709 13:10:29.202526 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 103308us
I20260709 13:10:29.144632 31889 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:29.247848 31889 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.249984 31889 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:29.210127 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 75577.4us
I20260709 13:10:29.160893 31151 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:29.213336 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:29.205781 31825 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:29.251152 25387 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
W20260709 13:10:29.184494 31386 consensus_meta.cc:287] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.636s	user 0.008s	sys 0.000s
I20260709 13:10:29.252159 31386 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.253460 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:29.254345 31386 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:29.254559 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:29.161706 31933 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.255641 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:29.256704 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:29.184016 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:29.211467 31948 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:29.257045 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:29.257437 25246 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:29.224422 31853 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:29.258885 31948 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:29.215641 25390 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 13:10:29.219645 25386 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:10:29.229707 31921 log.cc:927] Time spent T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.555s	user 0.010s	sys 0.000s
W20260709 13:10:29.232873 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:29.235175 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:29.241117 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:29.254916 31386 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:10:29.255558 31588 consensus_queue.cc:1059] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:10:29.256865 31933 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:29.212858 31813 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.225993 31898 raft_consensus.cc:3055] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:29.260882 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:29.278791 25237 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:29.279726 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:29.176782 31892 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.261827 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:29.262354 25313 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 17: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:29.264676 25245 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:29.260974 25359 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:29.271961 25911 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:29.285027 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:29.270334 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:29.275539 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:29.286691 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:29.288851 25212 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:29.285836 25237 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:29.276584 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:29.290213 25246 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:29.276788 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:29.291268 25243 raft_consensus.cc:2376] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 13. Re-sending same reply.
I20260709 13:10:29.286466 25387 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 20.
I20260709 13:10:29.273593 31862 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:29.307263 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:29.310474 25240 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 13:10:29.311910 25386 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:29.313206 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:29.314503 25245 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 21.
I20260709 13:10:29.314867 25909 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:29.313350 25911 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:29.275681 31937 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:29.317636 25940 raft_consensus.cc:2393] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 21: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:29.317942 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:29.278151 31898 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:29.318859 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:29.321768 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:10:29.319782 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 226618us
I20260709 13:10:29.319990 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:29.320104 25909 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:29.322633 25243 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 16
I20260709 13:10:29.323225 25287 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:29.297806 31948 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:29.324401 25909 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:29.325738 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:29.306265 31894 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:29.305819 31151 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:29.318378 31643 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:29.345201 31643 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:29.347378 31643 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:29.329717 31894 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:29.348377 31894 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:29.324379 31937 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:29.349038 25287 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:29.351773 31771 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:29.328141 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:29.325985 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:29.318472 31949 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:29.339072 31861 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:10:29.326505 31948 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:29.354744 31948 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:29.355968 25287 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:29.237888 31954 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.273322 31941 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.360244 31954 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:29.309377 31772 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.018s
I20260709 13:10:29.342959 31933 consensus_queue.cc:1059] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:29.326874 31588 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:29.362196 31933 raft_consensus.cc:3055] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:29.324115 31931 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:29.362576 31933 raft_consensus.cc:740] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:29.326167 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:29.358258 31771 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:29.366070 31813 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:29.353361 31799 consensus_queue.cc:1059] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:29.331226 31711 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:29.367763 31711 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:29.363152 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:29.368162 31711 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:29.368634 31861 raft_consensus.cc:697] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:29.369709 31861 consensus_queue.cc:237] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.374415 31892 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:29.376116 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:29.376796 25386 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:29.363222 31931 raft_consensus.cc:697] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:29.377971 31931 consensus_queue.cc:237] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.368332 31955 raft_consensus.cc:3055] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:29.351806 31937 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:29.325613 31732 consensus_meta.cc:287] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.516s	user 0.005s	sys 0.005s
I20260709 13:10:29.393651 25243 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:29.335646 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:29.394382 25909 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:29.447290 25945 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 1256) took 1348 ms. Trace:
I20260709 13:10:29.469780 25945 rpcz_store.cc:276] 0709 13:10:28.044228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:28.045959 (+  1731us) service_pool.cc:224] Handling call
0709 13:10:29.392824 (+1346865us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x7b5400690fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:29.392981 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":702112640}
I20260709 13:10:29.393978 25911 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:29.471844 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 77490.8us
W20260709 13:10:29.479805 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 85886.3us
I20260709 13:10:29.480276 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:29.500928 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:29.396092 31955 raft_consensus.cc:740] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:29.444954 31386 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:10:29.485538 31459 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:29.584626 31898 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.621308 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32994 (request call id 2701) took 1775 ms. Trace:
I20260709 13:10:29.708777 25241 rpcz_store.cc:276] 0709 13:10:27.845752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:27.925026 (+ 79274us) service_pool.cc:224] Handling call
0709 13:10:29.621005 (+1695979us) spinlock_profiling.cc:243] Waited 962 ms on lock 0x7b5400456854. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:29.621220 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2212079360}
I20260709 13:10:29.322094 31862 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.486559 31949 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:10:29.487965 31892 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:29.721549 25940 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:29.347623 31730 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:29.737449 25209 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.545s (SENT) [suppressed 11 similar messages]
W20260709 13:10:29.707468 31829 consensus_meta.cc:287] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.715s	user 0.008s	sys 0.002s
I20260709 13:10:29.697995 31459 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:29.487255 31732 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:29.796543 31459 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:29.363723 31933 consensus_queue.cc:260] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:29.808274 25938 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 17: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:29.732400 31711 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:29.926954 31960 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.229s	user 0.000s	sys 0.001s
I20260709 13:10:29.928109 31789 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:29.885690 31577 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:29.801760 31829 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:29.973801 31853 consensus_meta.cc:287] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.715s	user 0.000s	sys 0.011s
I20260709 13:10:29.956440 31577 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:29.928804 31789 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:29.836056 31955 consensus_queue.cc:260] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:30.059415 25943 raft_consensus.cc:2410] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 17 because replica has last-logged OpId of term: 15 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
W20260709 13:10:30.077726 31861 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.013s	sys 0.006s
W20260709 13:10:30.094509 31861 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.014s	sys 0.006s
I20260709 13:10:30.102859 25941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 1244) took 1121 ms. Trace:
I20260709 13:10:30.103039 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:30.103248 31941 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:30.103374 25941 rpcz_store.cc:276] 0709 13:10:28.981489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:28.981610 (+   121us) service_pool.cc:224] Handling call
0709 13:10:30.102677 (+1121067us) spinlock_profiling.cc:243] Waited 47.1 ms on lock 0x7b54007044d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:30.102800 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":108379904}
I20260709 13:10:30.103888 25314 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 19.
W20260709 13:10:30.104898 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 742678us
W20260709 13:10:30.105237 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 769632us
I20260709 13:10:30.106204 25212 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:30.106985 31892 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:30.107394 31892 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:30.107438 31933 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:30.107890 31892 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:30.108546 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:30.110379 25315 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 13. Current term is 14. Ops: []
I20260709 13:10:30.111689 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:30.112345 25315 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:30.115778 25314 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 20. Current term is 21. Ops: []
W20260709 13:10:30.120175 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:30.137601 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:30.139387 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:30.148312 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:30.149019 25314 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:30.156816 25314 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
W20260709 13:10:30.158090 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 45064.2us
I20260709 13:10:30.159283 25212 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:30.172555 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:30.119067 31949 raft_consensus.cc:697] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 21 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:30.129663 31810 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:30.144420 31948 consensus_meta.cc:287] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.789s	user 0.012s	sys 0.000s
I20260709 13:10:30.298537 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:30.321044 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 149061us
I20260709 13:10:30.325830 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:30.330854 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:30.344594 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:30.354146 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:30.358090 25314 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 21: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:30.382684 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:30.384270 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:30.425297 25242 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.165s	user 0.048s	sys 0.063s
W20260709 13:10:30.427330 25242 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.168s	user 0.049s	sys 0.063s
I20260709 13:10:30.428118 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:30.419468 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:30.421873 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:30.429183 31962 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.589s	user 0.002s	sys 0.000s
I20260709 13:10:30.459038 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:30.461318 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:30.464119 25319 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:30.465595 25315 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:30.373726 31941 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:30.374886 31730 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:30.468014 31941 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:30.475889 31949 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:30.423657 31810 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:30.428696 31889 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:30.481431 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2694) took 2139 ms. Trace:
I20260709 13:10:30.533706 25238 rpcz_store.cc:276] 0709 13:10:28.341951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:28.372733 (+ 30782us) service_pool.cc:224] Handling call
0709 13:10:28.414923 (+ 42190us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:28.557105 (+142182us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:30.481152 (+1924047us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b540013ae54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:30.481362 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":325148032}
I20260709 13:10:30.498203 31973 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:10:30.498471 31894 consensus_meta.cc:287] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 1.150s	user 0.005s	sys 0.006s
I20260709 13:10:30.463244 31970 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:30.501674 31825 consensus_meta.cc:287] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.250s	user 0.009s	sys 0.002s
I20260709 13:10:30.505683 31829 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:30.508786 31941 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:30.511102 31892 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:30.514978 31459 consensus_meta.cc:287] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.710s	user 0.009s	sys 0.002s
I20260709 13:10:30.335891 31948 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:30.458458 31154 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:30.536551 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:30.482632 31711 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:30.590807 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 139114us
W20260709 13:10:30.591010 25209 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.848s	user 0.002s	sys 0.011s
I20260709 13:10:30.554534 31954 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:30.597234 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:30.594539 25209 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.852s	user 0.003s	sys 0.011s
W20260709 13:10:30.599189 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 861708us
I20260709 13:10:30.599601 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:30.559054 31772 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:30.605999 25387 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 17. Current term is 18. Ops: [17.15-17.15]
I20260709 13:10:30.607976 25315 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
W20260709 13:10:30.610693 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 451454us
I20260709 13:10:30.611609 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:30.593108 31829 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:30.573069 31894 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:30.618624 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:30.588294 31825 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:30.604403 31886 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:30.586551 31970 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:30.626657 31977 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:30.627252 31977 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:30.627645 31825 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:30.628227 31825 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:30.629508 31886 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:30.631783 31886 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:30.628705 31825 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:30.633455 31973 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:30.634155 31973 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:30.634724 31825 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:30.596226 31154 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:30.594602 31459 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:30.639230 31459 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:30.640583 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:30.641260 25386 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:30.642498 31893 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:30.643183 31893 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:30.643712 31810 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:30.645391 31948 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:30.647593 31938 raft_consensus.cc:3055] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:30.648054 31938 raft_consensus.cc:740] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:30.648252 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:30.648646 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:30.648948 25391 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:30.649047 31154 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:30.645403 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 1160) took 1392 ms. Trace:
I20260709 13:10:30.649473 25945 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 1265) took 1170 ms. Trace:
I20260709 13:10:30.649937 25945 rpcz_store.cc:276] 0709 13:10:29.479168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:29.499875 (+ 20707us) service_pool.cc:224] Handling call
0709 13:10:30.648497 (+1148622us) spinlock_profiling.cc:243] Waited 8.23 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:10:30.649415 (+   918us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18929920}
I20260709 13:10:30.649799 25242 rpcz_store.cc:276] 0709 13:10:29.252993 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:29.258047 (+  5054us) service_pool.cc:224] Handling call
0709 13:10:29.258449 (+   402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:29.258892 (+   443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:29.258932 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:29.259494 (+   562us) log.cc:844] Serialized 26 byte log entry
0709 13:10:30.427862 (+1168368us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:30.427904 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:30.427978 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:30.645191 (+217213us) raft_consensus.cc:1685] finished
0709 13:10:30.645213 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:30.645345 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8115,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":7668,"thread_start_us":1168001,"threads_started":1,"wal-append.queue_time_us":1327691}
I20260709 13:10:30.616503 31966 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:30.653543 31966 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:30.655488 25386 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:30.655779 31966 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:30.658396 25315 raft_consensus.cc:606] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:30.658947 25315 raft_consensus.cc:614] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:10:30.660805 31848 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:30.659108 25911 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:30.661595 31894 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:30.661999 25911 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:30.663339 31974 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:30.663821 31974 raft_consensus.cc:697] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:30.603969 31954 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:30.665367 31970 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:30.665557 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:30.665951 25936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 1259) took 1344 ms. Trace:
I20260709 13:10:30.666157 25941 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:30.649526 31892 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:30.672365 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:30.661149 31938 consensus_queue.cc:260] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.9, Last appended by leader: 9, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:30.666319 25936 rpcz_store.cc:276] 0709 13:10:29.320996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:29.321104 (+   108us) service_pool.cc:224] Handling call
0709 13:10:30.665755 (+1344651us) spinlock_profiling.cc:243] Waited 742 ms on lock 0x7b54006608d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:30.665894 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1707015424}
W20260709 13:10:30.670022 25942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 1239) took 2158 ms (client timeout 2099 ms). Trace:
I20260709 13:10:30.670717 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:30.668177 31954 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:30.674669 25386 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:10:30.674383 25942 rpcz_store.cc:269] 0709 13:10:28.511651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:28.512438 (+   787us) service_pool.cc:224] Handling call
0709 13:10:30.669772 (+2157334us) spinlock_profiling.cc:243] Waited 613 ms on lock 0x7b5400381b54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:30.669929 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1409195392}
I20260709 13:10:30.610533 31772 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:30.680569 25941 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:30.681741 25386 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:30.684309 25911 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:30.662249 31894 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:30.687143 31985 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:30.689072 31929 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:30.691632 31769 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:30.673904 31973 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:30.694164 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59304 (request call id 1247) took 1381 ms. Trace:
I20260709 13:10:30.695250 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:30.697255 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 1161) took 1444 ms. Trace:
I20260709 13:10:30.698453 31808 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:30.707020 25240 rpcz_store.cc:276] 0709 13:10:29.263654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:29.288656 (+ 25002us) service_pool.cc:224] Handling call
0709 13:10:29.289051 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:29.311205 (+ 22154us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:30.696946 (+1385741us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b54003c2dd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:30.697188 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":48507648}
I20260709 13:10:30.700496 31772 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:30.618405 31829 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:30.664671 31974 consensus_queue.cc:237] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:30.681317 31892 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:30.705840 25938 rpcz_store.cc:276] 0709 13:10:29.312823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:29.312943 (+   120us) service_pool.cc:224] Handling call
0709 13:10:30.693941 (+1380998us) spinlock_profiling.cc:243] Waited 494 ms on lock 0x7b54004ad8d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:30.694096 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1136538368}
W20260709 13:10:30.709649 25908 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.990s	user 0.004s	sys 0.010s
W20260709 13:10:30.710067 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:30.710148 25358 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.821s	user 0.005s	sys 0.009s
I20260709 13:10:30.713469 25316 raft_consensus.cc:1275] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
W20260709 13:10:30.721657 25908 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.002s	user 0.004s	sys 0.010s
W20260709 13:10:30.722105 25358 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.833s	user 0.005s	sys 0.009s
W20260709 13:10:30.723868 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 834712us
W20260709 13:10:30.705452 31925 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 2.144s	user 0.016s	sys 0.029s
I20260709 13:10:30.702769 31894 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:30.643738 31893 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:30.702270 31938 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:30.714028 31769 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:30.707118 31806 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:30.726249 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2863) took 1398 ms. Trace:
I20260709 13:10:30.734465 25317 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 13. Current term is 14. Ops: []
I20260709 13:10:30.713917 31929 raft_consensus.cc:697] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 19 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:30.735041 25320 rpcz_store.cc:276] 0709 13:10:29.311076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:29.311215 (+   139us) service_pool.cc:224] Handling call
0709 13:10:30.695075 (+1383860us) spinlock_profiling.cc:243] Waited 1.64 ms on lock 0x7b58000e3498. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc780a0 00007f297bc77984 00007f29751f26d4 00007f29751f2662
0709 13:10:30.709738 (+ 14663us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3768704}
I20260709 13:10:30.730183 31894 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:30.738345 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:30.736744 25359 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:30.742274 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:30.742827 25240 raft_consensus.cc:3060] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:30.735548 31990 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:30.745982 31990 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:30.742036 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:30.726054 31808 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:30.751952 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:30.756173 25242 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 19 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 13:10:30.756565 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:30.752652 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:30.757313 25317 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:30.754036 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:30.728627 31892 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:30.756134 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:30.753309 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:30.707607 31985 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:30.754773 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:30.759163 25394 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:10:30.759867 25390 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:30.760038 25314 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
W20260709 13:10:30.764184 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:30.771569 25240 raft_consensus.cc:2468] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:30.774266 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:30.774859 25394 raft_consensus.cc:2376] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20. Re-sending same reply.
I20260709 13:10:30.775096 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:30.775460 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:30.775707 25317 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 16
I20260709 13:10:30.777245 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:30.778463 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:30.780139 25386 raft_consensus.cc:2368] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16.
I20260709 13:10:30.780236 25358 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:30.778644 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:30.781162 25389 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:30.781213 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:30.782732 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:30.784035 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:30.784718 25386 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:30.785881 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:30.786033 25390 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:30.786063 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:30.760543 31825 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:30.756565 31154 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:30.736040 31769 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:30.790326 25386 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:30.795287 31987 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:30.796046 31987 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:30.799158 31987 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:30.813741 31987 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:30.814333 25939 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:30.808168 25390 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:30.791528 25389 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 21.
I20260709 13:10:30.737206 31806 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:30.801797 31980 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:10:30.792472 31954 consensus_queue.cc:1059] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.013s
I20260709 13:10:30.790890 25908 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:30.809551 31892 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:30.810534 31985 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:30.810708 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 2969) took 2177 ms. Trace:
I20260709 13:10:30.812021 31990 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:30.816991 25389 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:10:30.820892 25939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 1263) took 1409 ms. Trace:
I20260709 13:10:30.788563 31989 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:30.829572 25939 rpcz_store.cc:276] 0709 13:10:29.411494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:29.411626 (+   132us) service_pool.cc:224] Handling call
0709 13:10:30.813927 (+1402301us) spinlock_profiling.cc:243] Waited 4.7 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:10:30.814137 (+   210us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:30.820527 (+  6390us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:30.820818 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10803456}
I20260709 13:10:30.826627 25392 rpcz_store.cc:276] 0709 13:10:28.632923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:28.633642 (+   719us) service_pool.cc:224] Handling call
0709 13:10:28.634055 (+   413us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:29.042328 (+408273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:30.810418 (+1768090us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x7b54000dded4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:30.810622 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":937074944}
I20260709 13:10:30.827066 25908 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:30.794639 31154 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:30.793661 31825 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:30.828043 31853 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:30.816313 31929 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:30.839147 31806 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:30.840262 31893 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:30.792315 31966 consensus_queue.cc:1059] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 13:10:30.842615 25908 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:30.814302 31987 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:30.826035 31948 raft_consensus.cc:2804] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 13:10:30.895903 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 53399.9us
I20260709 13:10:30.849810 31154 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:30.848752 31848 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:30.829084 31986 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:30.849715 31825 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:30.914059 25908 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:30.977618 32004 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:31.025405 32004 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:30.982121 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:31.029680 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:31.030941 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:30.984143 25318 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 19 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 13:10:31.032579 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.035079 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:31.028800 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.036458 25318 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 21: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:30.987468 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.038077 25386 raft_consensus.cc:2410] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 13:10:31.041762 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.042116 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 1379) took 1666 ms. Trace:
I20260709 13:10:31.042750 25387 rpcz_store.cc:276] 0709 13:10:29.375190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:29.375338 (+   148us) service_pool.cc:224] Handling call
0709 13:10:29.375713 (+   375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:30.607834 (+1232121us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:31.041348 (+433514us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b5400147b54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:31.041870 (+   522us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2827764608}
I20260709 13:10:30.992976 31769 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:31.044750 31769 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:31.045749 31769 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:30.897296 31772 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:30.994454 31892 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:30.848759 31829 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:30.935215 31987 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:31.050725 31892 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:30.853662 31989 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:31.055230 25389 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 13. Current term is 14. Ops: []
I20260709 13:10:30.841473 31996 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.097s
I20260709 13:10:30.937683 31894 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.055583 31989 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:31.056329 31769 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:31.058315 25943 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 1260) took 1736 ms (client timeout 1524 ms). Trace:
I20260709 13:10:30.918753 31986 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:31.008383 31938 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:31.011250 31990 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:31.061622 31990 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.062330 31986 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:31.058939 25943 rpcz_store.cc:269] 0709 13:10:29.325340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:29.325479 (+   139us) service_pool.cc:224] Handling call
0709 13:10:31.057744 (+1732265us) spinlock_profiling.cc:243] Waited 732 ms on lock 0x7b5400504e54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:31.058006 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1683708288}
I20260709 13:10:31.064407 31990 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:31.012380 31893 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:31.067395 25318 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 13:10:31.067201 31893 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.067857 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.068811 25314 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 21: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:31.067430 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.013427 31985 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:31.019290 25312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2847) took 2916 ms (client timeout 1545 ms). Trace:
W20260709 13:10:31.022122 25313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2854) took 2894 ms (client timeout 2237 ms). Trace:
I20260709 13:10:31.055372 32004 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:31.071434 31985 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:31.071589 25313 rpcz_store.cc:269] 0709 13:10:28.127274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:28.198895 (+ 71621us) service_pool.cc:224] Handling call
0709 13:10:31.021568 (+2822673us) spinlock_profiling.cc:243] Waited 972 ms on lock 0x7b5400276354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:31.021887 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2234626432}
W20260709 13:10:31.071219 25312 rpcz_store.cc:269] 0709 13:10:28.102896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:28.103034 (+   138us) service_pool.cc:224] Handling call
0709 13:10:31.018742 (+2915708us) spinlock_profiling.cc:243] Waited 967 ms on lock 0x7b5400274a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:31.019024 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2223929728}
I20260709 13:10:31.072923 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:31.074122 31985 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:31.074483 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.076197 25238 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 10. (term mismatch)
I20260709 13:10:31.076664 31732 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:31.079031 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:31.079440 25317 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:31.079974 25241 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:31.059401 31894 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:31.076552 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.080766 31893 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:31.082427 25312 raft_consensus.cc:2393] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:31.077559 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:30.914746 31948 raft_consensus.cc:697] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:31.083956 31798 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:31.084194 31892 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:31.084971 31948 consensus_queue.cc:237] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.060355 31938 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:31.074059 25242 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:31.076706 25238 pending_rounds.cc:85] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 9
I20260709 13:10:31.088313 25238 pending_rounds.cc:107] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 18.10
I20260709 13:10:31.088702 25238 raft_consensus.cc:2889] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:31.089862 31862 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:31.090233 31938 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:31.097965 31853 raft_consensus.cc:3055] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:10:31.098614 31853 raft_consensus.cc:740] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:31.114140 31848 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:31.115896 31953 raft_consensus.cc:3055] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:31.116434 31953 raft_consensus.cc:740] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:31.134383 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:31.118040 31953 consensus_queue.cc:260] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.141767 31892 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:31.142964 31892 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.107285 25315 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
W20260709 13:10:31.159665 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:31.159500 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:31.161006 31893 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:31.161170 31571 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:31.161612 31893 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:31.162233 31893 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:31.162087 31571 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.168673 25392 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 13. Current term is 14. Ops: []
I20260709 13:10:31.141776 31154 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:31.173007 25388 raft_consensus.cc:2368] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17.
W20260709 13:10:31.175367 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:31.175855 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:10:31.208247 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:31.274549 31825 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.292119 31825 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:31.294137 25239 raft_consensus.cc:1240] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:31.294269 31892 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:31.177963 25209 leader_election.cc:400] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16.
I20260709 13:10:31.294971 31768 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:31.203295 25321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2856) took 2063 ms (client timeout 1510 ms). Trace:
W20260709 13:10:31.295648 25321 rpcz_store.cc:269] 0709 13:10:29.139778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:29.139937 (+   159us) service_pool.cc:224] Handling call
0709 13:10:31.202713 (+2062776us) spinlock_profiling.cc:243] Waited 93.5 ms on lock 0x7b54000b22d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:31.203059 (+   346us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":215146240}
I20260709 13:10:31.295207 25209 leader_election.cc:403] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:10:31.297166 31768 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16.
W20260709 13:10:31.297736 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 120113us
I20260709 13:10:31.271204 31977 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:31.300062 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:31.301162 25242 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 17: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:31.175040 31154 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:31.301894 31846 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:31.302456 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:31.181826 25939 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:45428
I20260709 13:10:31.304311 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:31.305047 25939 raft_consensus.cc:606] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:31.305286 25241 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 21
I20260709 13:10:31.305567 25939 raft_consensus.cc:614] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:10:31.308598 31893 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:31.284718 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 375564us
I20260709 13:10:31.310717 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:31.310781 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:31.311153 32005 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:31.314375 25242 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:31.313511 31853 consensus_queue.cc:260] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.314914 25244 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:10:31.314589 32005 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:31.315580 31853 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:31.316244 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:31.317667 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:31.317727 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:31.318568 25246 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:31.319362 31810 raft_consensus.cc:3055] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:10:31.319832 31810 raft_consensus.cc:740] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:31.322391 31805 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:31.322733 31732 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:31.324106 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:31.324340 25386 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:31.324916 25237 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:31.294179 25394 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:31.313768 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:31.330574 25240 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:31.300851 31973 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:31.340456 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:31.342919 25237 raft_consensus.cc:1240] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:31.321089 31810 consensus_queue.cc:260] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.344084 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:31.344614 31973 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:31.345636 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:31.346035 31810 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:31.346279 25237 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:31.349515 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:31.350251 25244 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
I20260709 13:10:31.351179 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:31.352303 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:31.352564 31893 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:31.342597 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:31.353008 25244 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 17: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:31.371163 31931 consensus_queue.cc:1059] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.004s
I20260709 13:10:31.360803 25243 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:10:31.376453 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:31.367904 31571 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:31.358470 25940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 1261) took 2032 ms. Trace:
I20260709 13:10:31.375393 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:31.372283 31850 consensus_queue.cc:1059] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.007s
I20260709 13:10:31.378930 25244 raft_consensus.cc:3060] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:31.379115 31850 consensus_queue.cc:1243] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer a2a77ecf2c0941618b5bc79d4c04db61 log is divergent from this leader: its last log entry 12.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:10:31.379738 31850 raft_consensus.cc:3055] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:10:31.380087 31850 raft_consensus.cc:740] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:31.348904 31808 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:31.388247 25389 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:31.375293 31931 raft_consensus.cc:3055] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:10:31.389093 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 78939.1us
I20260709 13:10:31.377413 25243 raft_consensus.cc:2393] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 18: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:31.393816 25314 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:31.362195 25238 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:31.377980 25940 rpcz_store.cc:276] 0709 13:10:29.329015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:29.329158 (+   143us) service_pool.cc:224] Handling call
0709 13:10:29.329427 (+   269us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:29.736191 (+406764us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:31.358035 (+1621844us) spinlock_profiling.cc:243] Waited 392 ms on lock 0x7b54006310d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:31.358301 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":901114240}
I20260709 13:10:31.400470 25240 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 14.
I20260709 13:10:31.404048 25241 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 21.
I20260709 13:10:31.393260 31931 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:10:31.406780 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 94196.3us
I20260709 13:10:31.410598 25242 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:31.401799 31977 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:31.413056 25391 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 13:10:31.415253 31977 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.413117 25937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 1262) took 2018 ms. Trace:
I20260709 13:10:31.417296 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.417744 25937 rpcz_store.cc:276] 0709 13:10:29.394782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:29.411192 (+ 16410us) service_pool.cc:224] Handling call
0709 13:10:31.397996 (+1986804us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b5800003c98. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973b8db71 00007f2973b8da22 00007f297caa6c09 00007f297bc00ee0 00007f297bcca0ab 00007f297bcca131 00007f297bc8bfc0 00007f297bc76929 00007f297bc763d2 00007f29751edbd4
0709 13:10:31.398338 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:31.405923 (+  7585us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:31.405957 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:31.408561 (+  2604us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:31.412721 (+  4160us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:31.412801 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:31.412931 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23638272}
I20260709 13:10:31.419235 25395 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:10:31.419673 31973 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:31.421537 31789 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:31.423034 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:31.423461 25241 raft_consensus.cc:2376] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17. Re-sending same reply.
I20260709 13:10:31.425573 31850 consensus_queue.cc:260] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.430922 31850 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:31.421955 31577 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:31.433840 25908 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:31.436815 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:31.438334 31931 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.439946 31931 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:31.441251 32006 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:31.441856 32006 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:31.442646 25245 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 13. Current term is 14. Ops: []
I20260709 13:10:31.444089 32006 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:31.450095 31953 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:31.452011 25386 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:31.454703 25246 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:31.455421 32005 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:31.455822 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:31.455977 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:31.456540 25246 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:31.458947 25242 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:31.474777 25244 raft_consensus.cc:2468] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 21.
I20260709 13:10:31.476032 25246 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:31.478958 31965 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:10:31.481067 25389 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 21.
I20260709 13:10:31.481673 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:31.482682 25244 raft_consensus.cc:2368] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17.
W20260709 13:10:31.486546 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 76399.5us
I20260709 13:10:31.486594 31810 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:31.488534 25908 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:31.495693 25237 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:31.497212 25911 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:31.498421 31977 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:31.504842 25911 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:31.505774 31853 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:31.506304 31853 raft_consensus.cc:697] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:31.506649 31861 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:10:31.507267 31861 raft_consensus.cc:697] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 21 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:31.507356 31853 consensus_queue.cc:237] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.509333 31976 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:10:31.510908 31949 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:31.508520 31861 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.512802 31955 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:31.511338 31949 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:31.514516 31949 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:31.509343 25245 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:10:31.518435 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:31.519717 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:31.520943 31976 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:31.521857 31976 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.500905 25243 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:31.520509 25245 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 21: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:10:31.530052 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 60752.1us
W20260709 13:10:31.533690 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 45321.9us
I20260709 13:10:31.534822 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 13:10:31.533535 31386 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.004s
I20260709 13:10:31.546384 31898 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:31.549499 25244 raft_consensus.cc:1275] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:10:31.555351 25241 raft_consensus.cc:1275] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:10:31.557232 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:31.493542 31810 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.048s
I20260709 13:10:31.559283 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:31.562723 25245 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 17: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:31.563953 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:31.565027 25244 raft_consensus.cc:2368] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17.
I20260709 13:10:31.561482 25243 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:31.573639 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:31.575412 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:31.577406 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:31.578262 25237 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
W20260709 13:10:31.579285 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:31.580118 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:31.581830 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2712) took 2726 ms. Trace:
I20260709 13:10:31.582155 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:31.583910 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:31.584815 25244 raft_consensus.cc:2376] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 21. Re-sending same reply.
I20260709 13:10:31.582295 25239 rpcz_store.cc:276] 0709 13:10:28.849029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:28.851996 (+  2967us) service_pool.cc:224] Handling call
0709 13:10:28.852252 (+   256us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:31.294833 (+2442581us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:31.561740 (+266907us) spinlock_profiling.cc:243] Waited 2.44 s on lock 0x7b5400456354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:31.575331 (+ 13591us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5614624640}
I20260709 13:10:31.590545 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:10:31.594249 25237 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:31.590898 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:31.593266 25243 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260709 13:10:31.607642 31904 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:31.608587 25287 leader_election.cc:400] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16.
I20260709 13:10:31.609045 25287 leader_election.cc:403] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:10:31.609315 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:31.610693 25287 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:31.611776 25241 raft_consensus.cc:2410] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 13:10:31.611857 25287 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:31.617095 25359 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:31.617997 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 53397.5us
I20260709 13:10:31.620568 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.621490 31886 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16.
I20260709 13:10:31.622088 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:31.622846 31937 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:10:31.629613 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:31.629827 25359 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:31.630561 31789 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:31.630937 25240 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 20. Current term is 21. Ops: []
I20260709 13:10:31.631000 31789 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:31.631615 31789 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:31.622394 31886 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:31.632197 31886 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:31.632249 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:31.632629 31886 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:31.633029 25937 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:31.633370 31972 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:10:31.635437 31972 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:31.635807 31972 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:31.636164 31972 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:31.627460 31949 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.636442 31976 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:31.637697 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:31.638340 31949 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:31.639314 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:31.640112 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:31.595664 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.646803 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:31.648262 25394 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:31.649581 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:31.650002 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.650388 25936 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:31.650784 25316 raft_consensus.cc:2368] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 18. Current term is 19.
I20260709 13:10:31.651891 25393 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:10:31.652520 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:31.652691 25238 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 19
I20260709 13:10:31.656637 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:31.657478 25941 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 21: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:31.657398 25938 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:31.646795 25939 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 21 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 20. Current term is 21. Ops: []
I20260709 13:10:31.649405 31976 consensus_queue.cc:1059] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.012s
I20260709 13:10:31.661840 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.662242 31976 raft_consensus.cc:3055] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:31.662767 31976 raft_consensus.cc:740] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:31.663784 31976 consensus_queue.cc:260] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:31.670071 25937 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:31.662786 31899 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:31.680047 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:31.669842 31976 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:31.686825 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.687402 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:31.687646 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.691227 25936 raft_consensus.cc:2410] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 13:10:31.692744 25238 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:31.608395 31904 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:31.695176 25940 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:31.695372 25316 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:31.698160 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.698365 25937 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
I20260709 13:10:31.699718 25937 pending_rounds.cc:85] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 13
I20260709 13:10:31.699921 25937 pending_rounds.cc:107] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20260709 13:10:31.700070 25937 raft_consensus.cc:2889] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:31.704691 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.707948 25390 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:10:31.709712 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:31.709864 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.711045 25316 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:31.711581 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.712661 25314 raft_consensus.cc:2410] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 19 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 13:10:31.709877 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.714900 25395 raft_consensus.cc:2393] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 17: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:31.721858 25909 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:31.722678 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:31.725867 31904 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:31.725982 25941 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:31.687909 31899 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.709967 31577 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
I20260709 13:10:31.731886 31899 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:31.732854 25394 raft_consensus.cc:2393] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 18: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:31.679109 31459 consensus_queue.cc:1059] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.060s
I20260709 13:10:31.734059 25909 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:31.734519 31459 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:31.734870 31931 raft_consensus.cc:3055] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:31.735229 31931 raft_consensus.cc:740] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:10:31.736227 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:31.736315 31931 consensus_queue.cc:260] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.741786 31931 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:31.735268 31459 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:31.753966 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.756531 25943 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
W20260709 13:10:31.758980 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:31.761967 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 55635.9us
I20260709 13:10:31.764896 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:31.765834 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 66808.2us
W20260709 13:10:31.768388 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 156693us
I20260709 13:10:31.766330 25359 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:31.764868 25909 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:31.776402 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:31.776613 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.775439 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:31.779771 31899 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:31.780200 31899 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:31.780558 31899 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:31.785444 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.785842 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:31.791100 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 52789.7us
I20260709 13:10:31.792055 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:31.777805 25359 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:31.792932 25941 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
I20260709 13:10:31.794883 25943 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:31.797725 25321 raft_consensus.cc:2393] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 14: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:31.797899 25909 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:31.799132 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.799782 25909 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:31.800318 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.805745 25390 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 21.
I20260709 13:10:31.805873 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.809573 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:31.811866 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.812237 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.813020 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.829715 25313 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:31.801079 25909 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:31.814188 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:31.806478 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:31.809543 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.814477 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.832227 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.818987 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.818243 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.833487 25395 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:31.810570 25359 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:31.812341 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.813498 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.801533 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.811038 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:31.818135 25284 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:31.814421 25285 leader_election.cc:400] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17.
I20260709 13:10:31.828751 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.817808 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.818045 25287 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:31.838454 25285 leader_election.cc:403] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:10:31.839671 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:31.838606 25940 raft_consensus.cc:2410] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 18 because replica has last-logged OpId of term: 17 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260709 13:10:31.813839 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:31.810483 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:31.833875 25319 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:31.809903 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.819916 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:31.805408 31850 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:31.848039 31850 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:31.848451 31850 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:31.848738 31981 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:10:31.851699 31660 raft_consensus.cc:2804] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:10:31.852313 31660 raft_consensus.cc:697] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:31.852408 25287 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:31.819223 31853 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 13:10:31.853329 31660 consensus_queue.cc:237] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:31.859589 31904 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:10:31.861418 31981 raft_consensus.cc:697] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 21 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:31.862316 31952 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:31.862911 31952 raft_consensus.cc:697] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:31.862540 31981 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.863824 31952 consensus_queue.cc:237] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.865038 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:31.865845 25938 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:31.818356 31789 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:31.870071 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:31.871011 31789 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:31.871191 25939 raft_consensus.cc:2410] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 12.
I20260709 13:10:31.873389 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 1383) took 1279 ms. Trace:
I20260709 13:10:31.819398 31949 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:31.873858 25393 rpcz_store.cc:276] 0709 13:10:30.593725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:30.593856 (+   131us) service_pool.cc:224] Handling call
0709 13:10:30.594212 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:31.652255 (+1058043us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:31.873096 (+220841us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54003e4754. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:31.873319 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2431027456}
I20260709 13:10:31.875377 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:31.876502 25942 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 17: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:31.810870 31782 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:31.877861 31782 raft_consensus.cc:697] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:31.878887 31782 consensus_queue.cc:237] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.818643 31459 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:31.881458 31459 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:31.873314 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 72355.3us
W20260709 13:10:31.881873 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 62097.9us
I20260709 13:10:31.881897 31459 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:10:31.895473 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 81191.7us
I20260709 13:10:31.898100 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.898989 25390 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
W20260709 13:10:31.901355 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 90883.5us
I20260709 13:10:31.905439 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.906896 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:31.906263 25392 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:31.912290 25317 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:31.820762 31972 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:31.911859 31808 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:31.959246 31808 raft_consensus.cc:697] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:31.960553 31808 consensus_queue.cc:237] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:31.910240 31789 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:31.926466 25358 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:31.965634 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 53953.8us
I20260709 13:10:31.966923 31967 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:10:31.973474 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 57542.2us
I20260709 13:10:31.974645 25287 leader_election.cc:400] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17.
I20260709 13:10:31.920420 31955 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 17
W20260709 13:10:31.929194 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 111292us
I20260709 13:10:31.930292 31976 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:31.937825 25285 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:31.943037 31972 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:31.944227 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 49314.8us
I20260709 13:10:31.903766 31853 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
I20260709 13:10:32.000164 25938 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
W20260709 13:10:31.957176 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 55004.8us
I20260709 13:10:32.009246 25359 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:32.010756 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 258785us
W20260709 13:10:32.036846 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 45640.3us
W20260709 13:10:32.039791 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:32.048391 31793 consensus_queue.cc:1059] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.035s
I20260709 13:10:32.048966 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:32.049716 25317 raft_consensus.cc:2376] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19. Re-sending same reply.
I20260709 13:10:32.049834 25209 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:32.052256 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:32.057670 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:32.053956 25359 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:32.061226 31990 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:32.061501 31953 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:10:32.070730 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 133066us
I20260709 13:10:32.071633 25285 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:32.041842 31967 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:32.064971 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:32.070186 31953 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:32.049852 31810 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.017s
I20260709 13:10:32.078991 31577 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:32.080426 25284 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:32.092586 25313 raft_consensus.cc:3055] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:10:32.093713 25359 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:32.094321 25319 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 20. Current term is 21. Ops: []
I20260709 13:10:32.093151 25313 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:10:32.096560 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:32.055832 31904 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:32.097620 25358 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:32.082407 31953 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:32.053484 31886 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:32.103474 31899 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:32.090132 31967 raft_consensus.cc:697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:10:32.105324 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 55637.3us
I20260709 13:10:32.107771 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:32.092370 31577 raft_consensus.cc:697] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:32.109891 25312 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:10:32.112528 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:32.111956 31904 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:32.114312 31976 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:32.113705 25313 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:32.115178 25908 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:32.138366 25315 raft_consensus.cc:2368] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 18. Current term is 19.
I20260709 13:10:32.120263 31955 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17.
I20260709 13:10:32.150385 25313 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:32.151613 31459 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:32.153337 32006 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:32.136919 31898 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:32.146533 31948 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.010s
I20260709 13:10:32.154016 31898 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:32.161396 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 63896.7us
I20260709 13:10:32.138152 31976 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:32.131997 31850 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:32.173893 31899 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:32.175256 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 60227.4us
I20260709 13:10:32.176959 25209 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:32.146232 31789 consensus_queue.cc:1048] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.020s
W20260709 13:10:32.185747 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:32.186282 31990 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:32.164521 31949 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:32.166946 31822 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:32.164592 31898 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:32.188789 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:32.183905 31732 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:32.166262 31976 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:32.186515 31886 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:32.194458 25314 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 13. Current term is 14. Ops: []
I20260709 13:10:32.168120 31577 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:32.241091 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 83071.6us
I20260709 13:10:32.201740 31850 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:32.249244 25387 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 20. Current term is 21. Ops: []
I20260709 13:10:32.250168 25389 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 13:10:32.250927 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:32.252036 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:32.218046 31949 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:32.255369 25239 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:10:32.217799 31732 raft_consensus.cc:697] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 19 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:32.265414 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:32.266347 25241 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:32.262180 25392 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:32.270646 25209 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:32.274336 25388 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 20. Current term is 21. Ops: [19.14-20.15]
I20260709 13:10:32.275719 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:32.280268 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:32.242594 31459 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:32.291051 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 105081us
I20260709 13:10:32.290166 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:32.294173 25393 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:32.296830 25392 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:32.297941 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:32.299136 25908 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:32.195566 31967 consensus_queue.cc:237] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:32.303100 25287 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:32.254235 31972 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.040s
W20260709 13:10:32.306277 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:32.316577 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:32.282718 31904 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:32.325104 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:32.325690 31961 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:32.334004 31961 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:32.308610 31886 consensus_queue.cc:1059] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 13:10:32.315771 31459 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:32.336351 31961 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:32.336583 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 44810.2us
I20260709 13:10:32.330801 31937 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.007s
I20260709 13:10:32.325446 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:32.338603 25940 raft_consensus.cc:2410] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 15 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260709 13:10:32.338536 25908 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:32.344213 31732 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:32.320355 31990 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:32.350250 31571 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:32.350574 31937 raft_consensus.cc:3055] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:32.351097 31937 raft_consensus.cc:740] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:32.351636 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:32.352092 31937 consensus_queue.cc:260] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:32.308509 31952 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:32.354629 25313 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:10:32.358356 25389 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:32.361935 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:32.363229 31937 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:32.364037 31899 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:10:32.366189 25359 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:32.366880 31899 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:32.355796 31850 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.011s
I20260709 13:10:32.367326 31899 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:32.367702 31899 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:32.369877 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:32.314689 31955 consensus_queue.cc:1059] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 13:10:32.304303 31660 consensus_queue.cc:1059] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 13:10:32.374563 25393 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:32.357016 31904 raft_consensus.cc:3055] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:32.356439 31952 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:32.377427 25238 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
W20260709 13:10:32.378520 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 306970us
I20260709 13:10:32.396273 25393 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:32.400219 25287 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:32.368875 31953 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:32.373914 31571 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.045s
W20260709 13:10:32.423249 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 44059.5us
I20260709 13:10:32.356889 31782 consensus_queue.cc:1059] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.063s
I20260709 13:10:32.379393 31952 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:32.379580 31904 raft_consensus.cc:740] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:32.362742 31976 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:32.433260 31953 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:32.432876 31789 raft_consensus.cc:3055] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 13:10:32.442463 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 182995us
I20260709 13:10:32.433003 31799 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 13:10:32.458726 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 396539us
I20260709 13:10:32.426527 31972 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 13:10:32.465868 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:32.446785 31789 raft_consensus.cc:740] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:10:32.471861 25143 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53264 (request call id 209) took 18336 ms (client timeout 15000 ms). Trace:
I20260709 13:10:32.471431 31789 consensus_queue.cc:260] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:32.473332 31793 raft_consensus.cc:3055] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:32.473903 31793 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:32.475080 31976 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:32.474846 31793 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:32.476649 31961 raft_consensus.cc:3055] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:10:32.477092 31961 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:32.477411 25285 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:32.478138 31961 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:32.479441 31961 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:32.476838 31793 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:32.481431 31955 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:32.482409 31932 raft_consensus.cc:3055] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:32.482839 31932 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:32.486056 32032 consensus_queue.cc:1059] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.018s
I20260709 13:10:32.487282 32032 raft_consensus.cc:3055] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:32.487717 32032 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:10:32.469990 32024 log.cc:927] Time spent T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
W20260709 13:10:32.470364 32025 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 13:10:32.455302 32031 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 13:10:32.492360 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 197696us
I20260709 13:10:32.505002 25284 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:32.507412 32020 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.346s	user 0.005s	sys 0.000s
I20260709 13:10:32.508080 32037 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:32.502844 32031 raft_consensus.cc:697] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:32.510016 32031 consensus_queue.cc:237] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:32.463330 31953 consensus_queue.cc:1059] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
W20260709 13:10:32.515538 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 53788.1us
I20260709 13:10:32.516510 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:32.517452 32037 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:32.519021 25209 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:32.521551 25393 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
W20260709 13:10:32.504976 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 74868.9us
I20260709 13:10:32.530637 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:32.531332 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:32.532034 25386 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:10:32.533746 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:32.550441 25391 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 13:10:32.550740 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:32.504618 31904 consensus_queue.cc:260] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:32.561334 25386 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
W20260709 13:10:32.554873 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 74474.9us
I20260709 13:10:32.494293 31976 consensus_queue.cc:1059] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 13:10:32.519739 32037 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:32.488744 32032 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:32.474789 25144 catalog_manager.cc:5697] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 10 to 11, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 11 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:32.588642 25287 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:32.590191 31952 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:32.618113 31955 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:32.613207 32033 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:32.614054 31799 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:32.616300 31976 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:32.630390 31952 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:32.630082 32033 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:32.626683 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:32.631109 31976 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:32.627621 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:32.611970 31789 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:10:32.608354 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:32.636665 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:10:32.642417 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:32.643352 25386 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:32.646522 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:32.647285 25240 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:32.648240 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:32.630223 31799 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:32.656965 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:32.673444 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:32.693804 25246 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:32.581321 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:32.711380 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 211209us
I20260709 13:10:32.558947 31793 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:32.734794 25386 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:32.738600 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:32.740012 25240 raft_consensus.cc:2376] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19. Re-sending same reply.
I20260709 13:10:32.582357 32037 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:32.639206 31955 raft_consensus.cc:697] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:32.658322 31976 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:32.707281 31953 raft_consensus.cc:3055] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Stepping down as leader of term 14
W20260709 13:10:32.755990 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 196827us
I20260709 13:10:32.581907 31949 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.166s
I20260709 13:10:32.717146 31459 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:32.741737 31990 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 13:10:32.757297 31898 consensus_meta.cc:287] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.557s	user 0.011s	sys 0.000s
I20260709 13:10:32.751643 32032 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:32.794464 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:32.794792 31898 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:32.796574 32033 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:32.797155 31898 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:32.798189 25358 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:32.799530 31972 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:10:32.800099 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:32.800647 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:32.800912 31972 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:32.801318 31808 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20260709 13:10:32.804411 32040 log.cc:927] Time spent T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 13:10:32.752663 32037 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:32.755406 31571 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:32.705832 31732 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.053s
I20260709 13:10:32.753623 31904 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:32.713259 31899 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:32.824577 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:32.825793 25237 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:32.770761 25285 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:32.755215 31953 raft_consensus.cc:740] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:10:32.776831 32050 log.cc:927] Time spent T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.182s	user 0.001s	sys 0.001s
I20260709 13:10:32.777963 25144 catalog_manager.cc:5697] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 7 to 12. New cstate: current_term: 12 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:10:32.782867 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 223166us
I20260709 13:10:32.783839 25378 tablet_service.cc:1558] Processing DeleteTablet for tablet eda233ace2a24b379fef320a5f610c7a with delete_type TABLET_DATA_TOMBSTONED (TS a2a77ecf2c0941618b5bc79d4c04db61 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44978
I20260709 13:10:32.789023 31932 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:32.810138 25389 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:32.833709 31782 raft_consensus.cc:2764] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election decision vote started in defunct term 18: won
I20260709 13:10:32.841614 25144 catalog_manager.cc:5697] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:10:32.835820 31571 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:32.850560 31932 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:32.852866 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:32.853348 31955 consensus_queue.cc:237] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:32.821096 25240 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:10:32.854745 25209 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:32.472256 25143 rpcz_store.cc:269] 0709 13:10:14.135637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:14.135836 (+   199us) service_pool.cc:224] Handling call
0709 13:10:32.471613 (+18335777us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:10:28.009263 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:10:28.009915 (+   652us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:10:28.009977 (+    62us) write_op.cc:435] Acquired schema lock
0709 13:10:28.009999 (+    22us) tablet.cc:662] Decoding operations
0709 13:10:28.011305 (+  1306us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:10:28.011407 (+   102us) write_op.cc:641] Partition lock acquired for write op
0709 13:10:28.011438 (+    31us) tablet.cc:685] Acquiring locks for 14 operations
0709 13:10:28.012935 (+  1497us) tablet.cc:701] Row locks acquired
0709 13:10:28.012987 (+    52us) write_op.cc:260] PREPARE: finished
0709 13:10:28.013202 (+   215us) write_op.cc:270] Start()
0709 13:10:28.013354 (+   152us) write_op.cc:276] Timestamp: P: 1783602628013137 usec, L: 0
0709 13:10:28.013390 (+    36us) op_driver.cc:348] REPLICATION: starting
0709 13:10:28.014100 (+   710us) log.cc:844] Serialized 5541 byte log entry
0709 13:10:29.131506 (+1117406us) op_driver.cc:464] REPLICATION: finished
0709 13:10:29.557116 (+425610us) write_op.cc:301] APPLY: starting
0709 13:10:29.704517 (+147401us) tablet.cc:1367] starting BulkCheckPresence
0709 13:10:30.024624 (+320107us) tablet.cc:1370] finished BulkCheckPresence
0709 13:10:30.024802 (+   178us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:10:30.838629 (+813827us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:10:30.839151 (+   522us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=28,mrs_lookups=0
0709 13:10:30.839240 (+    89us) write_op.cc:312] APPLY: finished
0709 13:10:30.839798 (+   558us) log.cc:844] Serialized 129 byte log entry
0709 13:10:30.920681 (+ 80883us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:10:30.926566 (+  5885us) write_op.cc:454] Released schema lock
0709 13:10:30.930241 (+  3675us) write_op.cc:341] FINISH: Updating metrics
0709 13:10:30.937704 (+  7463us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:10:30.937736 (+    32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":9934424,"reported_tablets":35,"child_traces":[["op",{"apply.queue_time_us":310475,"apply.run_cpu_time_us":69207,"apply.run_wall_time_us":1438630,"cfile_cache_hit":58,"cfile_cache_hit_bytes":62591,"num_ops":14,"prepare.queue_time_us":842702,"prepare.run_cpu_time_us":25286,"prepare.run_wall_time_us":410936,"raft.queue_time_us":210044,"raft.run_cpu_time_us":45430,"raft.run_wall_time_us":281619,"replication_time_us":1085205,"thread_start_us":1338638,"threads_started":5,"wal-append.queue_time_us":622615,"wal-append.run_cpu_time_us":25235,"wal-append.run_wall_time_us":2444238}]]}
I20260709 13:10:32.857782 25387 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
W20260709 13:10:32.830509 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 59828.9us
I20260709 13:10:32.853899 31990 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:32.880717 25285 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:32.879747 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 79212.2us
I20260709 13:10:32.882993 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:32.776124 25246 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:32.883857 25285 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:32.837598 32033 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:32.884589 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:32.887348 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:32.888058 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:32.871276 32042 raft_consensus.cc:2764] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 13:10:32.889566 25388 raft_consensus.cc:1275] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:10:32.893282 25241 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 13:10:32.894426 25237 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:32.896538 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:32.877980 31990 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:32.899514 31799 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:32.882524 31949 consensus_queue.cc:1059] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.022s
I20260709 13:10:32.901497 25144 catalog_manager.cc:5697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 10 to 11, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 11 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:10:32.828049 31459 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:32.907706 31660 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:32.910218 32053 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:32.914453 25287 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:32.897104 31953 consensus_queue.cc:260] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:32.939038 31899 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:32.928511 31990 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:32.942461 25244 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:10:32.943159 25312 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 13:10:32.943813 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:32.894416 31808 raft_consensus.cc:2764] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
W20260709 13:10:32.975003 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:32.945968 25144 catalog_manager.cc:5697] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 9 to 11, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 11 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:10:32.946928 31861 raft_consensus.cc:3055] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 13:10:33.000087 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 85783.1us
I20260709 13:10:33.012951 25144 catalog_manager.cc:5697] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 4 to 9, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 9 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:33.017211 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:32.990779 31861 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:33.020020 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:10:33.057363 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 127449us
I20260709 13:10:33.059245 25285 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:33.003981 31660 raft_consensus.cc:697] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:33.060539 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:33.070019 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:33.057394 32058 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 13:10:33.006253 32068 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:33.077764 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:33.148530 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:33.148449 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:33.156615 25937 raft_consensus.cc:2393] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 14: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:33.060891 31949 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:33.061194 31732 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:33.177951 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:33.188032 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:33.188166 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:33.166615 32031 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.025s	sys 0.020s
I20260709 13:10:33.136281 31850 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:33.203366 25144 catalog_manager.cc:5697] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 9 to 13, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 13 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:10:33.206400 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:33.207393 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:33.057214 31822 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.111s
I20260709 13:10:33.210482 25387 raft_consensus.cc:1240] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260709 13:10:33.207300 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:33.246188 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:33.248646 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:33.246986 25312 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:33.252239 25316 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:33.251590 25314 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
W20260709 13:10:33.245641 25209 leader_election.cc:336] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.545s (SENT)
I20260709 13:10:33.188514 32068 raft_consensus.cc:697] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:33.257707 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:33.215282 32031 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.025s	sys 0.020s
I20260709 13:10:33.258263 25245 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 13:10:33.258365 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:33.269233 25319 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:33.270201 25212 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:33.270331 32066 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:33.270835 32066 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:33.270951 32035 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 13:10:33.259056 25320 raft_consensus.cc:1275] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
W20260709 13:10:33.261372 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 197694us
I20260709 13:10:33.268103 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:33.179157 31793 consensus_queue.cc:1059] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.082s
I20260709 13:10:33.245800 32030 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:33.233289 31577 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:33.299835 25315 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:33.254369 32071 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:33.275790 25144 catalog_manager.cc:5697] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 3 to 10. New cstate: current_term: 10 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:10:33.271214 32066 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:33.279021 25909 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:33.279724 32052 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
W20260709 13:10:33.281715 32051 log.cc:927] Time spent T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.210s	user 0.001s	sys 0.000s
W20260709 13:10:33.281821 32047 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 13:10:33.286253 31886 raft_consensus.cc:3055] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:10:33.272240 31660 consensus_queue.cc:237] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.295646 25131 catalog_manager.cc:5028] TS a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): tablet eda233ace2a24b379fef320a5f610c7a (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:10:33.307240 25318 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
W20260709 13:10:33.271864 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 120420us
W20260709 13:10:33.307442 32070 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.142s	user 0.000s	sys 0.001s
I20260709 13:10:33.315945 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:33.316756 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:33.247578 31732 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:33.321411 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 124442us
W20260709 13:10:33.322160 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:33.307469 32037 consensus_meta.cc:287] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.500s	user 0.012s	sys 0.000s
I20260709 13:10:33.191202 31810 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.119s
I20260709 13:10:33.307534 32030 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.255648 31850 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.345733 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:33.271546 31799 consensus_queue.cc:1048] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260709 13:10:33.308303 31571 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:33.338166 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 92540.5us
I20260709 13:10:33.340577 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:33.310388 32042 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:33.344164 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:33.192541 31949 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.333968 25144 catalog_manager.cc:5697] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 9 to 14, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 14 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:10:33.327203 31886 raft_consensus.cc:740] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:10:33.351604 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 72670us
I20260709 13:10:33.264863 31861 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.390329 32082 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:33.390872 32082 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.320211 32071 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:33.393426 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2761) took 1042 ms. Trace:
I20260709 13:10:33.393667 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:33.394093 32071 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:33.394717 32071 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:33.393843 25239 rpcz_store.cc:276] 0709 13:10:32.351414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:32.357732 (+  6318us) service_pool.cc:224] Handling call
0709 13:10:32.363638 (+  5906us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:32.371727 (+  8089us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:32.371761 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:32.385501 (+ 13740us) log.cc:844] Serialized 26 byte log entry
0709 13:10:32.626349 (+240848us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:32.626396 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:32.626476 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:33.393203 (+766727us) raft_consensus.cc:1685] finished
0709 13:10:33.393225 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:33.393368 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7675,"raft.queue_time_us":10366,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":3798,"thread_start_us":196140,"threads_started":1,"wal-append.queue_time_us":586453}
I20260709 13:10:33.395367 32071 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:33.395723 32071 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:33.396064 32071 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:33.397918 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:33.387663 32039 consensus_queue.cc:1059] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.015s
I20260709 13:10:33.307869 31577 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.400759 25314 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260709 13:10:33.351821 25389 raft_consensus.cc:3055] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:33.405375 25389 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:33.423136 31850 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:33.426960 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:33.429909 32062 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 13:10:33.431363 32088 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:33.432893 31571 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:33.433416 31571 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.434367 25909 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:33.436995 31949 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:33.438182 31949 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:10:33.437613 31886 consensus_queue.cc:260] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:33.439379 31886 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:33.440485 31577 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:33.346240 32037 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:33.441749 32079 log.cc:927] Time spent T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 13:10:33.442361 25243 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:33.442870 32037 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:33.443135 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:33.392974 32082 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:33.385933 32068 consensus_queue.cc:237] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:33.444330 25314 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:10:33.435530 25909 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:33.425467 25209 leader_election.cc:336] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.694s (SENT)
W20260709 13:10:33.450250 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 200514us
W20260709 13:10:33.457619 32049 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.577s	user 0.004s	sys 0.000s
I20260709 13:10:33.458240 32031 raft_consensus.cc:3055] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:33.458648 32031 raft_consensus.cc:740] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:33.458976 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2752) took 1213 ms. Trace:
I20260709 13:10:33.460047 31732 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:33.463762 25239 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 19
I20260709 13:10:33.464727 25209 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:33.466972 31459 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:33.466812 31990 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.469448 25245 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:33.473615 31990 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:33.478644 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:33.481277 31808 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:33.481732 31808 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:33.482098 31808 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:33.482578 31789 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:10:33.459020 32030 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:33.486107 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:33.486863 32090 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:33.487552 32090 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:33.459374 25242 rpcz_store.cc:276] 0709 13:10:32.245479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:32.245601 (+   122us) service_pool.cc:224] Handling call
0709 13:10:32.265310 (+ 19709us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:32.420323 (+155013us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:32.420352 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:32.430223 (+  9871us) log.cc:844] Serialized 26 byte log entry
0709 13:10:32.586515 (+156292us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:32.595531 (+  9016us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:32.595624 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:33.456815 (+861191us) spinlock_profiling.cc:243] Waited 3.24 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:33.458679 (+  1864us) raft_consensus.cc:1685] finished
0709 13:10:33.458700 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:33.458892 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":23978,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":3615,"spinlock_wait_cycles":7460736,"thread_start_us":286434,"threads_started":2,"wal-append.queue_time_us":413167}
I20260709 13:10:33.493755 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:33.496243 25319 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:33.500439 25245 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:10:33.502061 25244 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 17: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:33.435504 31571 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:33.504612 25284 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.767s	user 0.016s	sys 0.004s
W20260709 13:10:33.506489 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:33.506737 25284 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.016s	sys 0.004s
W20260709 13:10:33.507107 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 794794us
W20260709 13:10:33.513140 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 77667.4us
W20260709 13:10:33.517762 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:33.518244 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:33.518977 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:33.519939 25287 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:33.521642 25284 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:33.528859 25318 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:33.506155 25314 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 15. (term mismatch)
I20260709 13:10:33.531147 25314 pending_rounds.cc:85] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 14
I20260709 13:10:33.531452 25314 pending_rounds.cc:107] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 13:10:33.531723 25314 raft_consensus.cc:2889] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:33.503686 25212 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:33.512806 25389 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.547209 25312 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 19 index: 17. (index mismatch)
I20260709 13:10:33.502604 31850 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:33.561105 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:33.561838 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 58241.8us
I20260709 13:10:33.514358 31967 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:10:33.516141 32090 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:33.562592 25212 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:33.517338 31459 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:33.512880 31789 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:33.573319 25388 raft_consensus.cc:2393] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 19: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:33.573112 25389 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:33.555107 31972 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:33.576737 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:33.589620 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:33.594599 25394 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:33.592262 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:33.593442 25212 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:33.595927 25390 raft_consensus.cc:2393] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 15: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:33.576795 31386 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.015s
I20260709 13:10:33.419052 32042 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:33.597801 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 76256.1us
I20260709 13:10:33.603055 25393 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:10:33.609813 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:33.610082 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:33.610755 25390 raft_consensus.cc:2368] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 18. Current term is 19.
I20260709 13:10:33.610684 25395 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:33.581319 32053 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:33.570484 32087 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:33.593318 32068 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:33.591419 32082 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:33.616365 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 63666.3us
I20260709 13:10:33.613334 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:33.620914 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:33.618330 25284 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:33.626838 25389 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 19 index: 17. (index mismatch)
W20260709 13:10:33.628206 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 208037us
I20260709 13:10:33.644011 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:33.607730 32037 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:33.607486 31990 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:33.607971 32090 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:33.652441 32042 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:33.603432 31972 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:33.628566 31898 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 13:10:33.574513 31850 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.597848 31789 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:33.616417 32068 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:33.679070 32093 raft_consensus.cc:2764] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 13:10:33.679330 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:33.679656 25936 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 18 index: 15. (term mismatch)
I20260709 13:10:33.680487 31972 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:33.675431 32088 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:33.615263 32056 consensus_queue.cc:1059] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.171s
I20260709 13:10:33.721626 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:33.694794 32029 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:33.682397 25392 raft_consensus.cc:2376] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 21. Re-sending same reply.
W20260709 13:10:33.663395 32039 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 19 current term: 19 is higher.
I20260709 13:10:33.705580 32053 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:33.709923 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 64245us
I20260709 13:10:33.712179 25936 pending_rounds.cc:85] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 14
I20260709 13:10:33.718231 25394 raft_consensus.cc:2410] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 13:10:33.726718 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:33.732748 25936 pending_rounds.cc:107] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 17.15
I20260709 13:10:33.704659 32042 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:33.715013 32075 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.263s	user 0.000s	sys 0.001s
I20260709 13:10:33.729164 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:33.728503 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:33.707396 25940 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:10:33.632750 32082 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.733079 25936 raft_consensus.cc:2889] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:33.718994 32072 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:10:33.743683 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:33.744015 32107 raft_consensus.cc:3055] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:33.738571 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:33.709923 31459 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.031s
I20260709 13:10:33.745160 31459 consensus_queue.cc:1243] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer 2329e428ebbf484fa3ea9452133df567 log is divergent from this leader: its last log entry 16.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 13:10:33.745055 32042 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:33.747777 32042 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:33.744436 32107 raft_consensus.cc:740] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:33.749500 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:33.749262 32107 consensus_queue.cc:260] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.750087 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:33.750662 32107 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:33.751134 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:33.752087 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:33.752151 32090 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:33.752482 25240 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:33.752686 32090 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:33.744525 32082 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:33.675653 31990 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.744475 25312 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:33.710790 32068 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:33.768270 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:33.794934 25237 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
I20260709 13:10:33.753093 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:33.767398 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:33.789642 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:33.766978 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:33.797008 25315 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 20: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:10:33.795051 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 55156.3us
W20260709 13:10:33.760005 32099 log.cc:927] Time spent T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 13:10:33.833287 32093 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:33.833793 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:33.833886 32093 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.835041 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:33.835830 25936 raft_consensus.cc:2393] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 15: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:10:33.759382 32074 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.359s	user 0.001s	sys 0.001s
W20260709 13:10:33.755614 32080 log.cc:927] Time spent T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.369s	user 0.000s	sys 0.001s
W20260709 13:10:33.766360 32081 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.366s	user 0.000s	sys 0.002s
I20260709 13:10:33.800064 25240 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 19: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:10:33.822108 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 43753.2us
I20260709 13:10:33.855193 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:33.858687 25909 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:33.818320 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 155408us
I20260709 13:10:33.825253 31949 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:10:33.761595 31861 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:33.825688 32071 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:33.740833 32053 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:33.756345 32048 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:33.826864 31459 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:10:33.836509 25242 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:33.716571 31808 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.875010 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:33.729511 32088 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.884680 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:33.884936 25389 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:33.885545 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:33.882138 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:33.888386 25387 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 15: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:33.818712 31853 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.061s
I20260709 13:10:33.894474 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:33.733887 32029 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.892270 25284 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:33.890254 25936 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:33.898516 25388 raft_consensus.cc:2393] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 17: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:33.901542 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:33.902976 32087 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:33.903482 32087 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:33.904002 32087 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:33.907433 25212 leader_election.cc:336] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.530s (SENT)
I20260709 13:10:33.909293 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:33.914147 32114 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:10:33.897507 31981 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:33.915197 31981 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:33.915601 31981 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:33.892663 32048 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.916189 32111 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:33.916728 32111 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:33.917707 32095 log.cc:927] Time spent T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.244s	user 0.000s	sys 0.001s
I20260709 13:10:33.917904 32048 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:33.918893 32111 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:33.919946 31990 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:33.921818 31577 raft_consensus.cc:3055] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:33.922223 31577 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:10:33.914625 32096 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:10:33.923036 31577 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:33.908521 32030 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:33.925156 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:33.928398 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 1182) took 1030 ms. Trace:
I20260709 13:10:33.928805 25246 rpcz_store.cc:276] 0709 13:10:32.897741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:32.897884 (+   143us) service_pool.cc:224] Handling call
0709 13:10:32.898154 (+   270us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:32.898396 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:32.898420 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:32.971300 (+ 72880us) log.cc:844] Serialized 26 byte log entry
0709 13:10:33.180621 (+209321us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:33.206971 (+ 26350us) log.cc:844] Serialized 14 byte log entry
0709 13:10:33.311200 (+104229us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:33.311304 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:33.754853 (+443549us) spinlock_profiling.cc:243] Waited 33.2 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:33.928076 (+173223us) raft_consensus.cc:1685] finished
0709 13:10:33.928198 (+   122us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:33.928331 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29938,"prepare.queue_time_us":35109,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":2236,"raft.queue_time_us":52429,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":4399,"spinlock_wait_cycles":76295424,"thread_start_us":278218,"threads_started":2,"wal-append.queue_time_us":329196}
I20260709 13:10:33.932943 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 1421) took 1014 ms. Trace:
I20260709 13:10:33.935485 25212 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:33.933362 25391 rpcz_store.cc:276] 0709 13:10:32.918454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:32.918578 (+   124us) service_pool.cc:224] Handling call
0709 13:10:32.918940 (+   362us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:32.919375 (+   435us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:32.919407 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:32.920039 (+   632us) log.cc:844] Serialized 52 byte log entry
0709 13:10:32.987742 (+ 67703us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:32.987792 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:32.987860 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:33.932739 (+944879us) raft_consensus.cc:1685] finished
0709 13:10:33.932760 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:33.932891 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1882,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"thread_start_us":67277,"threads_started":1,"wal-append.queue_time_us":427908}
I20260709 13:10:33.945564 31990 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:33.947279 31981 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.949265 32083 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:10:33.912269 32066 consensus_meta.cc:287] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.578s	user 0.011s	sys 0.000s
W20260709 13:10:33.965464 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 73315.7us
I20260709 13:10:33.888043 32091 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:33.949683 31981 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:33.990463 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:33.991529 25394 raft_consensus.cc:2368] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 18. Current term is 19.
I20260709 13:10:33.911671 32093 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:33.989639 25940 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 21 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 20. Current term is 21. Ops: []
I20260709 13:10:33.949738 32083 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:33.991348 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:33.995580 25239 consensus_meta.cc:287] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.531s	user 0.012s	sys 0.000s
I20260709 13:10:33.983166 25212 leader_election.cc:400] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 18. Current term is 19.
I20260709 13:10:33.996119 25238 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:33.997792 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:33.996695 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:33.997210 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:33.996587 25212 leader_election.cc:403] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:10:34.000213 25312 pending_rounds.cc:85] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 13
I20260709 13:10:34.000869 25312 pending_rounds.cc:107] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 16.14
I20260709 13:10:34.001143 25312 raft_consensus.cc:2889] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:34.004302 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:33.993273 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:34.008631 25316 raft_consensus.cc:2410] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 22 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 13:10:34.013517 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:34.014348 25939 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:34.014430 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:34.015559 25945 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 15: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:34.018203 25936 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:34.018530 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:34.021113 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:34.031440 31952 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:34.032075 31952 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:34.033339 31990 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:34.033834 31990 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:34.034201 31990 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:34.034515 32094 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:34.035017 32094 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.036824 25944 raft_consensus.cc:2410] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 13:10:33.982815 32066 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:33.998165 32083 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:34.006920 25212 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:34.040238 25238 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:34.057757 25938 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 21.
W20260709 13:10:34.056402 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 49588.6us
W20260709 13:10:34.114565 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:34.112185 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:34.099676 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:34.124034 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:34.001173 32111 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:34.003762 31789 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:34.130570 25940 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:10:34.131942 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:34.132740 25317 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 19.
I20260709 13:10:34.046674 32029 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:34.133984 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:34.037092 32094 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:34.051553 32071 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:34.041414 31952 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:34.137053 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 49252.3us
I20260709 13:10:34.144199 25212 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:34.046253 31990 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.045534 32091 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:34.066649 32093 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 18. Current term is 19.
I20260709 13:10:34.174206 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:34.176910 25319 raft_consensus.cc:2393] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 17: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:34.178442 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:34.184998 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:34.166797 32071 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:34.146579 25358 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:34.147349 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 261162us
I20260709 13:10:34.147426 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:34.039604 32066 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:34.188241 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 44134.4us
I20260709 13:10:34.188333 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:34.129031 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:34.167225 25316 raft_consensus.cc:2393] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 19: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:34.167927 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:34.026253 32042 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:34.188472 25359 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:34.201924 25314 raft_consensus.cc:2393] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 14: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:10:34.202466 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 55999.6us
I20260709 13:10:34.203259 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:34.213851 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:34.214859 25238 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:34.219230 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:34.220016 25317 raft_consensus.cc:2393] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 20: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:34.222090 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:34.232898 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:34.270002 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:34.272152 25318 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 19.
I20260709 13:10:34.207386 31808 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:34.218500 31898 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:34.220932 32006 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:10:34.173075 32091 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.287962 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:34.290230 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:34.231316 32053 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:34.279965 25238 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:34.298982 25393 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:10:34.299527 31972 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:34.299721 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:34.299935 32042 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:34.300091 31972 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:34.299945 25285 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:34.300586 32042 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:34.303067 25394 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:10:34.303746 32042 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:34.304082 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:34.306237 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:34.312098 25387 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:34.315693 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:34.315730 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:34.325825 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 107834us
I20260709 13:10:34.290357 32105 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:10:34.288870 31789 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:34.291616 32006 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:34.328871 25239 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 13.
I20260709 13:10:34.291587 31898 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:34.293048 31967 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:34.320955 32107 consensus_meta.cc:287] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.570s	user 0.009s	sys 0.004s
I20260709 13:10:34.268703 32108 consensus_queue.cc:1048] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 13:10:34.340982 32088 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:34.342759 32091 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:34.360067 32108 consensus_queue.cc:1243] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer 2329e428ebbf484fa3ea9452133df567 log is divergent from this leader: its last log entry 14.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:10:34.347707 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:34.348206 32083 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.351459 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:34.360888 32124 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:34.352244 31990 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:34.361402 32124 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:34.361788 32124 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:34.352684 31789 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.363085 32091 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:34.363592 32091 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.365638 32091 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:34.355885 32090 consensus_meta.cc:287] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.565s	user 0.004s	sys 0.005s
I20260709 13:10:34.366514 32090 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.336253 32006 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:34.346486 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:34.361246 25243 raft_consensus.cc:2393] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 20: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:10:34.377115 32067 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 1.113s	user 0.001s	sys 0.001s
I20260709 13:10:34.333683 31808 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:34.378662 31808 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.379050 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 1417) took 1592 ms. Trace:
I20260709 13:10:34.380196 25386 rpcz_store.cc:276] 0709 13:10:32.786486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:32.786628 (+   142us) service_pool.cc:224] Handling call
0709 13:10:32.787000 (+   372us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:32.787404 (+   404us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:32.787437 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:32.788022 (+   585us) log.cc:844] Serialized 52 byte log entry
0709 13:10:32.987753 (+199731us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:10:32.987799 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:32.987872 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:34.337680 (+1349808us) spinlock_profiling.cc:243] Waited 2.04 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:34.378732 (+ 41052us) raft_consensus.cc:1685] finished
0709 13:10:34.378755 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:34.378990 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1836,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":110,"spinlock_wait_cycles":4701056,"thread_start_us":199334,"threads_started":1,"wal-append.queue_time_us":457020}
I20260709 13:10:34.382635 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 1181) took 1485 ms. Trace:
I20260709 13:10:34.382782 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:34.383471 25315 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:34.383195 25241 rpcz_store.cc:276] 0709 13:10:32.897368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:32.897497 (+   129us) service_pool.cc:224] Handling call
0709 13:10:32.897792 (+   295us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:32.993646 (+ 95854us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:32.993925 (+   279us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:33.003196 (+  9271us) log.cc:844] Serialized 26 byte log entry
0709 13:10:33.603268 (+600072us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:33.603306 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:33.603377 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:34.381676 (+778299us) spinlock_profiling.cc:243] Waited 4.3 ms on lock 0x7b24001206c8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973a4b9c4 00007f2973a49acb 00007f2973c04dc9 00007f2973c04a55 00007f2973c03fb0 00007f297687a488 00007f297687abe8 00007f2976852398 00007f297684d76a 00007f297684cbb6
0709 13:10:34.382198 (+   522us) raft_consensus.cc:1685] finished
0709 13:10:34.382229 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:34.382561 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4945,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"spinlock_wait_cycles":9895808,"thread_start_us":544587,"threads_started":2,"wal-append.queue_time_us":696557}
W20260709 13:10:34.385301 32123 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 13:10:34.386773 31972 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:34.383772 32090 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:34.387307 31967 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:34.388095 25943 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:34.389389 31967 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:34.335116 32105 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:34.390460 32105 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:10:34.394021 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:34.395468 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:34.396590 25319 raft_consensus.cc:2410] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 21 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
W20260709 13:10:34.405232 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:34.406481 25209 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:34.335932 31898 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:34.331949 32127 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:34.422498 32127 raft_consensus.cc:697] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:34.423463 32127 consensus_queue.cc:237] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.425799 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:34.412843 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 83772.6us
I20260709 13:10:34.419822 32083 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:34.428052 25358 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:34.407601 32068 consensus_meta.cc:287] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.555s	user 0.006s	sys 0.004s
I20260709 13:10:34.435899 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:34.437369 25192 auto_rebalancer.cc:629] Failed to schedule move for tablet 72001570ee2c41cb88f0e9eac74992c7: Illegal state: Replica 2329e428ebbf484fa3ea9452133df567 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 19 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:34.438079 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:10:34.409646 32087 consensus_meta.cc:287] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.505s	user 0.011s	sys 0.000s
I20260709 13:10:34.442862 32133 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:34.451411 32068 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.440737 32006 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:34.442116 25358 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:34.462162 25212 leader_election.cc:336] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.812s (SENT)
I20260709 13:10:34.443230 32124 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:34.445200 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:34.436409 25314 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:34.450292 25940 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:10:34.451653 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:34.465809 25209 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:34.467604 25209 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:34.466271 25237 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:10:34.463521 31990 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:34.461699 31850 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:10:34.473795 25209 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:34.466813 32090 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:34.468396 32071 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:34.472844 32101 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:10:34.484689 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:34.476415 32037 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:34.463423 32087 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.477963 32068 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:34.480669 32042 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:34.511456 25940 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:34.478534 31967 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.020s
I20260709 13:10:34.515710 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:34.497743 32109 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:34.510967 32101 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:10:34.511422 32104 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:34.508190 32105 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:34.544795 31850 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:34.554448 25240 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:10:34.555958 32087 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:34.563066 25940 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:10:34.567438 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:34.568526 25936 raft_consensus.cc:2368] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18.
I20260709 13:10:34.571277 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:34.573734 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:34.557544 32124 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:34.557898 32101 raft_consensus.cc:697] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:34.574033 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:34.578876 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:34.579546 25240 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 19: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:34.580523 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:34.583156 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:34.582691 25240 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:34.583735 25940 raft_consensus.cc:1240] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 13:10:34.552654 32104 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:34.664394 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 190643us
I20260709 13:10:34.665306 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:34.572845 32127 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.005s
I20260709 13:10:34.566051 31850 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.551582 32109 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.578372 32006 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:34.666721 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 182121us
I20260709 13:10:34.675519 25936 raft_consensus.cc:1275] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:10:34.677280 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:34.681082 25240 raft_consensus.cc:2468] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 22.
I20260709 13:10:34.663187 32091 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:34.683858 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:34.690114 32031 consensus_queue.cc:260] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:34.678001 31898 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:34.690893 25944 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 1277) took 1539 ms (client timeout 1638 ms). Trace:
I20260709 13:10:34.696933 25245 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:34.670531 32124 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:34.688020 31789 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:34.697713 25240 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 19
I20260709 13:10:34.700187 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:10:34.712272 31898 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:34.712570 32109 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:34.703106 32091 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:34.701469 32136 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
I20260709 13:10:34.717721 32068 consensus_queue.cc:1059] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:10:34.718885 31386 raft_consensus.cc:3055] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:34.719319 31386 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:34.720331 31386 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.720899 32091 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:34.721714 31386 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:34.695376 25392 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
W20260709 13:10:34.703243 25944 rpcz_store.cc:269] 0709 13:10:33.151164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:33.151530 (+   366us) service_pool.cc:224] Handling call
0709 13:10:34.690689 (+1539159us) spinlock_profiling.cc:243] Waited 848 ms on lock 0x7b54006310d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:34.690836 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1949908736}
I20260709 13:10:34.710444 31850 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:34.716298 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:34.727504 32031 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:34.731837 25358 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:34.732699 32006 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:34.732987 31577 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:34.738427 32106 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:10:34.739959 25358 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:34.710527 32124 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.743675 25284 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:34.743875 32124 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:34.745075 25284 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:34.746030 25358 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:34.748781 25240 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:10:34.758183 32006 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:34.758819 32127 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:34.761262 32006 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:34.766479 32006 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:34.769053 31953 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:34.760390 32056 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:34.776684 25358 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:34.778158 32056 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:34.781301 31822 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:34.781430 32056 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.783833 32056 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:34.789626 25358 leader_election.cc:400] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18.
I20260709 13:10:34.790056 25358 leader_election.cc:403] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:10:34.790537 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:34.792546 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:34.776568 32109 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:34.773947 31850 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:34.793684 32124 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:10:34.777232 32072 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.027s
I20260709 13:10:34.814059 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:34.800751 32145 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:34.794786 32114 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:34.794418 31789 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.011s
I20260709 13:10:34.801275 31899 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:34.822424 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:34.820698 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:34.821769 25940 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:34.816641 32039 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:34.822695 25945 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:10:34.816650 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:34.823215 25394 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:34.822964 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:34.822468 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:34.826623 25389 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 19: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:34.827782 25393 raft_consensus.cc:2393] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 19: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:34.830065 25284 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:34.834754 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:34.835673 32056 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:34.814224 32144 consensus_queue.cc:1048] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.022s
W20260709 13:10:34.838542 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 173301us
I20260709 13:10:34.838802 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:34.840454 32087 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:34.841116 32087 raft_consensus.cc:697] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 19 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:34.797518 31898 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.023s
I20260709 13:10:34.810446 32109 raft_consensus.cc:697] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:34.840665 25209 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:34.842108 32087 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.845189 25212 leader_election.cc:400] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 18. Current term is 19.
I20260709 13:10:34.847483 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:34.810196 32033 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.021s
I20260709 13:10:34.867794 25212 leader_election.cc:403] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:10:34.813230 32116 consensus_queue.cc:1059] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 13:10:34.845386 32090 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:34.869592 32144 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.004s
I20260709 13:10:34.873713 32144 consensus_queue.cc:1243] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer f646dfd532974dceadfe00005d2bbf59 log is divergent from this leader: its last log entry 13.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:10:34.873363 32090 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.876021 32090 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:34.875705 32106 consensus_queue.cc:1059] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:10:34.882462 25284 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:34.780571 32101 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.810096 31850 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.887094 25938 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:10:34.889110 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:34.891426 32071 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:34.892652 32144 raft_consensus.cc:3055] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:34.892843 32068 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:10:34.899298 25359 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:34.903982 25937 pending_rounds.cc:85] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 11
I20260709 13:10:34.912540 25937 pending_rounds.cc:107] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 13.12
I20260709 13:10:34.912851 25937 raft_consensus.cc:2889] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:34.830883 32039 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:34.921034 32072 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.005s
I20260709 13:10:34.881160 25212 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:34.939713 32114 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18.
I20260709 13:10:34.923759 32119 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:34.921661 31577 raft_consensus.cc:2804] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:10:34.906016 32144 raft_consensus.cc:740] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:34.911051 32068 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:34.950120 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:34.819813 32145 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:34.948829 32104 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:34.959765 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 78724.5us
I20260709 13:10:34.964164 31990 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:10:34.984555 25358 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.66:42315 timed out after 1.781s (SENT) [suppressed 48 similar messages]
I20260709 13:10:34.972147 32068 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:34.971172 31577 raft_consensus.cc:697] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:34.984230 31990 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:34.960583 32109 consensus_queue.cc:237] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:34.968128 32119 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:35.011474 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 170851us
I20260709 13:10:35.013585 31990 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:35.034405 25209 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:35.036913 32137 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.485s	user 0.004s	sys 0.000s
I20260709 13:10:35.042953 32155 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:35.043489 32155 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:35.043874 32155 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:35.044377 32133 raft_consensus.cc:3055] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:35.044802 32133 raft_consensus.cc:740] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:35.020356 32037 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 18. Current term is 19.
I20260709 13:10:35.056255 25237 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:10:35.054359 32033 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:35.020903 32087 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:35.045696 32133 consensus_queue.cc:260] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:35.063731 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 79203.5us
I20260709 13:10:35.017768 32144 consensus_queue.cc:260] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.065347 32107 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:35.073203 32118 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.003s
I20260709 13:10:35.074534 32144 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:35.075826 32145 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:35.069519 32006 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:35.085093 32006 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:35.069792 32091 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:35.086042 32091 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.087357 32006 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:35.088107 32091 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:35.090057 32133 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:35.102434 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:35.103263 25387 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 14.
W20260709 13:10:35.115633 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:35.121707 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:35.123857 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:35.124508 25389 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 21.
I20260709 13:10:35.064271 32033 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:35.127804 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 77709.2us
I20260709 13:10:35.130430 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:35.132304 25389 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:35.132226 25209 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:35.097961 32039 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:35.117758 31976 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:35.066669 32087 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.139227 25209 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:35.139768 31861 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:10:35.128098 32071 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:35.101173 31386 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:35.073832 32107 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:35.146818 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:35.140971 32042 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:10:35.159658 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:35.149884 25394 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:35.101810 31799 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:35.166618 31799 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:35.167232 32105 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:35.142756 31808 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:35.169771 32033 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:35.171101 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2786) took 1403 ms. Trace:
I20260709 13:10:35.101702 32006 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:35.158211 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:35.158910 32090 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:35.148496 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:35.178094 25387 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:35.171509 25244 rpcz_store.cc:276] 0709 13:10:33.768044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:33.768624 (+   580us) service_pool.cc:224] Handling call
0709 13:10:33.769022 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:33.769476 (+   454us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:33.769517 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:33.818685 (+ 49168us) log.cc:844] Serialized 26 byte log entry
0709 13:10:34.982891 (+1164206us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b1800454c00. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f29768520d1 00007f2976c6da0f
0709 13:10:35.019621 (+ 36730us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:35.019676 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:35.019757 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:35.142887 (+123130us) raft_consensus.cc:1685] finished
0709 13:10:35.142907 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:35.171036 (+ 28129us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":105,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":32077,"spinlock_wait_cycles":27195648,"thread_start_us":36522,"threads_started":1,"wal-append.queue_time_us":1279768}
I20260709 13:10:35.073566 31577 consensus_queue.cc:237] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:35.169860 31808 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:35.193413 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.194890 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:10:35.198259 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 49839.9us
I20260709 13:10:35.205358 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.209426 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:35.216159 25244 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 15: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:35.216408 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:10:35.217993 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 74014.5us
I20260709 13:10:35.220783 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:35.184212 32006 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.221421 25938 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:35.222096 25911 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:35.222853 31799 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:35.212435 32037 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:35.224391 32037 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.226682 32037 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:35.223836 32105 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:35.227461 32105 raft_consensus.cc:697] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:35.228432 32105 consensus_queue.cc:237] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.187258 31850 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:35.235100 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:35.217011 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.237243 25389 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:35.244712 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:35.181003 31386 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:35.269877 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.149024 31976 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:35.266042 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.186820 31808 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:35.268288 25209 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:35.277304 32112 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:35.278467 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:35.267484 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:35.281679 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:35.286316 25242 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 1188) took 1554 ms (client timeout 2061 ms). Trace:
I20260709 13:10:35.283421 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.231619 32127 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:35.282416 25238 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:10:35.286988 25244 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
W20260709 13:10:35.286769 25242 rpcz_store.cc:269] 0709 13:10:33.723831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:33.723995 (+   164us) service_pool.cc:224] Handling call
0709 13:10:35.278410 (+1554415us) spinlock_profiling.cc:243] Waited 91.5 ms on lock 0x7b54001f6dd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:35.278533 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":210485120}
I20260709 13:10:35.288019 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.285823 25945 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:35.233928 31799 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:35.294467 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:35.296252 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.281805 32037 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:35.319638 25938 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 19.
I20260709 13:10:35.307143 32127 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:35.340472 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 72256.2us
I20260709 13:10:35.290906 32068 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.347342 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:35.298983 31799 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.358714 31386 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:35.295869 32112 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.372026 25320 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:35.342694 31990 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:35.339210 32037 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:10:35.365715 32166 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 13:10:35.366541 32164 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 13:10:35.360117 31808 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:35.345116 31850 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:35.405699 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:35.413997 25239 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 2783) took 1197 ms (client timeout 1539 ms). Trace:
I20260709 13:10:35.414664 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:35.414486 25239 rpcz_store.cc:269] 0709 13:10:34.216912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:34.275414 (+ 58502us) service_pool.cc:224] Handling call
0709 13:10:35.413846 (+1138432us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b54002d05d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:35.413944 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24278912}
I20260709 13:10:35.416669 25389 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:35.371810 32119 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:35.340803 32127 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:35.371321 31932 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:35.419350 25240 raft_consensus.cc:1240] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: []
W20260709 13:10:35.411278 32031 consensus_meta.cc:287] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.683s	user 0.013s	sys 0.000s
I20260709 13:10:35.404464 32037 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:35.430608 25387 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:10:35.431382 25321 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
W20260709 13:10:35.423322 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 201298us
I20260709 13:10:35.420840 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:35.404861 32101 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:35.433882 25320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2876) took 2043 ms (client timeout 1781 ms). Trace:
W20260709 13:10:35.435799 32170 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 13:10:35.435091 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.445206 25245 raft_consensus.cc:2410] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 13:10:35.447036 32042 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:35.432049 32118 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:35.434928 25320 rpcz_store.cc:269] 0709 13:10:33.390139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:33.397603 (+  7464us) service_pool.cc:224] Handling call
0709 13:10:35.433716 (+2036113us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b540002c6d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:35.433824 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4135167360}
I20260709 13:10:35.449322 32042 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.449467 32118 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:35.430816 31932 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.452636 32042 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:35.419351 32087 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:35.478896 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:35.433763 32037 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:35.455152 25911 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:35.483263 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 48924.5us
I20260709 13:10:35.482795 32083 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:35.487334 32119 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:35.487413 25320 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "292e92613a634963a5de1f016eb951ea"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:10:35.453092 31932 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:35.488636 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.488765 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:35.434504 32155 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:35.493002 31386 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:35.503871 32155 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:35.503715 31386 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:35.492012 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:35.489996 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:35.505559 25312 raft_consensus.cc:2368] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 21. Current term is 22.
I20260709 13:10:35.505949 31386 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:35.507230 25212 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:35.508085 32155 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:10:35.508790 32155 raft_consensus.cc:697] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:35.509680 32155 consensus_queue.cc:237] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:35.510201 32127 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:35.511654 31990 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:35.512463 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.493664 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.497886 25314 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:35.493117 25320 raft_consensus.cc:606] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:35.499600 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:35.514218 25320 raft_consensus.cc:614] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:10:35.514605 25317 raft_consensus.cc:3060] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:35.500226 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:35.516000 25318 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:35.517920 25316 pending_rounds.cc:85] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 8
I20260709 13:10:35.518244 25316 pending_rounds.cc:107] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260709 13:10:35.518474 25316 raft_consensus.cc:2889] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:35.521026 32068 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:35.522557 32112 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:35.446656 32101 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:35.537472 32101 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:35.501410 31976 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:35.539193 25317 raft_consensus.cc:2410] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 22 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 13:10:35.539912 25393 raft_consensus.cc:2410] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 13:10:35.500680 32119 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:35.541953 32119 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:35.529713 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:35.547477 25389 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:35.548599 25911 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:35.551366 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.553247 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.553778 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.554225 25312 raft_consensus.cc:2410] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 19 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 13:10:35.555249 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.557720 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:35.560107 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 147492us
I20260709 13:10:35.534842 25318 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:35.535506 32090 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:10:35.573380 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 203245us
W20260709 13:10:35.546470 32181 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:10:35.591490 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.591953 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.564246 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:35.592651 25313 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:35.577334 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:35.568032 25285 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:10:35.594393 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 43482.2us
I20260709 13:10:35.596684 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:35.595415 25911 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:35.592561 25314 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 15: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:35.599733 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.600531 25314 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:35.601651 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:35.576610 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.597078 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:35.597244 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:35.603044 25320 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 19: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:35.597703 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:35.604166 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:35.604563 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.605477 25244 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:35.606648 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.606842 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.608073 25244 raft_consensus.cc:2393] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 22: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:35.608335 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.610023 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.611037 25320 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:35.612200 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:35.613072 25320 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:35.617220 32179 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:10:35.574051 32083 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:35.617780 32179 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:35.618306 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.621690 25237 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:35.622706 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.622973 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:35.623330 25318 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 21.
I20260709 13:10:35.623783 25387 raft_consensus.cc:2410] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 13:10:35.624200 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.624810 25318 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:35.618306 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.625916 25315 raft_consensus.cc:2376] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15. Re-sending same reply.
I20260709 13:10:35.626868 31799 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:35.628000 25315 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:35.619971 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.551801 31952 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:35.620090 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:35.618377 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:35.632483 25313 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:35.632978 25321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36864 (request call id 2893) took 1476 ms (client timeout 1619 ms). Trace:
W20260709 13:10:35.636034 25321 rpcz_store.cc:269] 0709 13:10:34.156270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:34.156893 (+   623us) service_pool.cc:224] Handling call
0709 13:10:35.632815 (+1475922us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b5400443254. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:35.632907 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2879823232}
W20260709 13:10:35.641904 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 398111us
I20260709 13:10:35.572927 32133 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:35.587416 25209 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:35.618446 32112 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:35.648933 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.617937 32083 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:35.649616 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 62308.1us
I20260709 13:10:35.650492 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.651252 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:35.651508 25239 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 14.
I20260709 13:10:35.651938 25318 raft_consensus.cc:2376] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15. Re-sending same reply.
I20260709 13:10:35.618939 32042 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:35.657295 25209 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:35.665923 25319 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:35.618374 32087 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:35.637049 32179 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:35.672829 25209 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:35.687764 25321 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:10:35.695001 25320 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:35.696271 25313 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
I20260709 13:10:35.697139 25312 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 13:10:35.699018 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.631778 31952 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:35.705053 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.648662 32112 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:35.666385 31967 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:35.668943 32083 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:35.626912 32037 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.666200 32090 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:35.712301 25209 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:35.712157 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.694314 32093 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:35.710899 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.713891 25246 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:35.732628 25209 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:35.734007 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.667855 32133 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.709815 32112 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:35.750557 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 107194us
I20260709 13:10:35.747432 25313 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:10:35.752866 25909 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:35.760601 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.766178 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.687417 32042 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.698120 32087 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.775328 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:35.778620 25239 raft_consensus.cc:2393] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 15: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:35.741724 31990 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:10:35.782853 25315 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:10:35.731487 32180 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:35.796823 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 64258.7us
I20260709 13:10:35.799235 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.802016 25315 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:10:35.804738 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.805603 25313 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:35.807080 25237 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:35.808792 25242 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:10:35.774507 31861 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:35.752008 32090 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:35.730036 31967 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:35.815419 25246 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
W20260709 13:10:35.831933 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 236612us
I20260709 13:10:35.831789 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.833971 25313 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 15: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:35.813958 31861 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:35.835592 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 54200.6us
I20260709 13:10:35.837255 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:35.861840 32187 log.cc:927] Time spent T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:10:35.832944 25911 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:35.867177 25911 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:35.874388 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:35.879072 32087 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:35.836330 32182 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:35.854852 31861 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:35.883083 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.885332 25320 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 22: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:35.891702 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.893589 32037 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:35.893847 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.894826 25320 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:35.896867 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.898298 32006 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:35.899235 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:35.912073 25318 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 19: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:35.913165 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.914575 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.915464 25318 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 18: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:35.899319 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.916215 25320 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:35.917636 25320 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:35.856346 32042 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:35.902024 32184 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:10:35.903707 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2792) took 1548 ms. Trace:
W20260709 13:10:35.904731 32191 log.cc:927] Time spent T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 13:10:35.904618 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:35.921432 25238 rpcz_store.cc:276] 0709 13:10:34.355743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:34.451869 (+ 96126us) service_pool.cc:224] Handling call
0709 13:10:35.278564 (+826695us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:35.287367 (+  8803us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:35.903535 (+616168us) spinlock_profiling.cc:243] Waited 811 ms on lock 0x7b54001f6dd0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:35.903654 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1864339328}
I20260709 13:10:35.906364 32133 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:35.884017 32183 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 13:10:35.920281 32119 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:35.871986 32199 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:35.923604 25942 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:35.881222 32200 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:35.923925 25911 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:35.907631 32182 raft_consensus.cc:697] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:35.932006 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:35.932328 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:35.936196 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:35.929317 32200 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:35.939344 32200 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:10:35.941568 32192 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260709 13:10:35.931878 31952 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:35.947049 32112 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:35.948511 25314 raft_consensus.cc:2368] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15.
I20260709 13:10:35.930619 32119 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.950824 32119 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:35.938370 31967 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:35.934098 32198 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:35.954762 32033 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:10:35.956164 32033 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:35.958010 31861 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.959084 32119 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:10:35.960516 31967 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:35.963972 31967 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:35.964471 31967 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:35.954916 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:35.970211 25287 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:35.971418 25287 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:35.971818 32182 consensus_queue.cc:237] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.973831 25287 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:35.961781 32112 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:35.980125 32199 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:35.974952 32119 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:35.940580 32179 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:35.998111 31952 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:10:36.009459 25945 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:10:36.013044 32180 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:36.014991 32179 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:36.034533 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:36.035341 25393 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:36.021567 25287 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:36.045898 25287 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:36.049055 25287 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:36.020560 31386 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:36.068732 25358 leader_election.cc:400] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 21. Current term is 22.
I20260709 13:10:36.069147 25358 leader_election.cc:403] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260709 13:10:36.050336 32068 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:36.031319 32200 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:36.073716 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:36.062742 31799 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:36.055301 32072 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:36.063374 32179 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:36.075596 25358 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:36.077468 32179 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:36.079743 32179 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:36.080722 32083 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:36.082654 32107 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:36.082865 32083 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:36.087409 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:36.088044 25394 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:36.097361 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 240077us
I20260709 13:10:36.098133 25909 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:36.099769 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:36.100375 25392 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:36.103463 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:36.104590 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:36.105183 32180 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:36.105901 25312 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:10:36.108183 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:36.109434 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:36.110180 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:36.110551 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:36.111255 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:36.111533 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:36.111833 25395 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:36.114756 25285 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:36.075464 32144 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:36.116499 32179 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:36.116925 32179 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:36.117297 32179 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:36.120015 25243 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 13:10:36.109483 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:36.112762 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:36.121587 25241 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:36.123786 25245 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:10:36.125730 32200 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:36.128764 31861 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:36.129971 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:36.130622 25388 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:36.132505 25392 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 20.
I20260709 13:10:36.133215 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:36.133826 25245 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:36.098994 25909 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:36.135288 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:36.136570 25909 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:36.137539 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:36.138880 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 63368.8us
I20260709 13:10:36.139935 25358 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:36.140908 25359 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:36.133724 25911 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:36.134569 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:36.134766 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:36.135890 25239 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:36.144394 25244 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 21.
I20260709 13:10:36.149670 25394 raft_consensus.cc:2468] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:36.149739 25285 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:36.151932 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:36.150945 25911 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:36.159247 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:36.146104 31577 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:36.144728 31799 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:36.147491 31861 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:36.150786 32177 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:10:36.150650 32133 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:36.167553 25393 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 22.
I20260709 13:10:36.153841 32200 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:10:36.157115 32083 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:36.175127 25285 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:36.164103 32144 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:36.162496 32202 raft_consensus.cc:2804] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:36.177821 25209 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:36.181159 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:36.168784 32107 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:36.183609 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:36.169592 32042 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:36.186281 25942 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:10:36.148908 31967 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:36.194371 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:36.196041 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:36.181983 32109 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 21. Current term is 22.
I20260709 13:10:36.179785 32177 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:36.190793 32107 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:36.200451 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:36.216984 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:36.201400 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:36.190423 31898 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:36.181197 32169 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:36.197439 32042 raft_consensus.cc:697] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:36.173096 32037 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.020s
I20260709 13:10:36.213713 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:36.238821 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:36.240078 25315 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:36.239399 25320 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 15: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:36.191519 32144 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:36.182389 32133 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:36.187436 32083 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:36.260581 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:36.231530 25312 raft_consensus.cc:1275] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 11. (term mismatch)
I20260709 13:10:36.228775 25209 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:36.241003 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:36.261447 25312 pending_rounds.cc:85] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 10
I20260709 13:10:36.261775 25312 pending_rounds.cc:107] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 20.11
I20260709 13:10:36.255553 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:36.190706 32200 raft_consensus.cc:697] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 20 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:36.259881 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:36.200110 32087 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election won for term 18
W20260709 13:10:36.258072 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 68690.8us
I20260709 13:10:36.262440 25941 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:36.286355 25315 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 22: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:36.231369 32177 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:36.203884 32202 raft_consensus.cc:697] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:36.262040 25312 raft_consensus.cc:2889] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:36.294087 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:36.294579 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:36.295439 25315 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:36.296583 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:36.288985 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 60311.4us
I20260709 13:10:36.238878 32107 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:36.238790 32180 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:36.291647 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:36.242954 25908 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:36.302441 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:36.257843 32032 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:36.258538 31861 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:36.206424 32072 consensus_queue.cc:1059] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.081s
I20260709 13:10:36.309772 25285 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:36.284605 31898 raft_consensus.cc:697] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:36.370049 31898 consensus_queue.cc:237] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:36.315250 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:36.315359 25320 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "37bfef8566b143cf94971f26f417678c"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:10:36.376339 25320 raft_consensus.cc:606] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:36.376711 25320 raft_consensus.cc:614] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:10:36.212445 32175 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 13:10:36.316447 32205 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 13:10:36.262529 32169 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:36.317642 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:36.293748 31990 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:36.328735 31967 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:36.329214 32200 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:36.286417 32090 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:36.333825 32118 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:36.284485 32083 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:36.260336 32093 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:36.334852 32208 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
I20260709 13:10:36.262373 32133 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:36.259371 32144 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:36.336971 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:36.423604 31990 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:36.342152 32204 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
I20260709 13:10:36.341477 32179 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:36.343832 32214 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 13:10:36.345479 25388 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:36.346046 32068 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:36.347216 32037 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:10:36.356231 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 195819us
I20260709 13:10:36.357043 25318 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 22: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:10:36.357223 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 114363us
I20260709 13:10:36.231182 32199 consensus_queue.cc:1059] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.062s
I20260709 13:10:36.367170 32180 raft_consensus.cc:697] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:36.389559 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:36.398002 32202 consensus_queue.cc:237] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:36.465215 25939 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 22: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:36.399653 25317 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:36.465289 32180 consensus_queue.cc:237] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:36.408456 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:36.408648 25192 placement_policy_util.cc:248] tablet eda233ace2a24b379fef320a5f610c7a: not considering replicas for movement since the tablet's status is 'UNAVAILABLE'
W20260709 13:10:36.424799 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 125292us
I20260709 13:10:36.427935 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:36.447646 25312 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:36.255235 32178 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:36.449709 31990 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:36.453342 25284 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:36.456194 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:36.457110 25320 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:10:36.480026 25320 raft_consensus.cc:606] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:36.480459 25320 raft_consensus.cc:614] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:10:36.439334 32198 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:10:36.473383 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 321496us
I20260709 13:10:36.473634 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:36.503424 25358 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:36.474917 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:36.504707 25242 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:36.479396 25209 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:36.500478 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 190828us
I20260709 13:10:36.435251 32090 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:36.511634 25939 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20260709 13:10:36.508117 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 48708us
I20260709 13:10:36.445045 32093 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:36.430898 32042 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:36.478176 32219 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:36.502403 32031 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:36.504061 32178 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:36.534466 32219 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:36.541301 25242 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 20.
I20260709 13:10:36.541563 32219 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:36.535903 32087 raft_consensus.cc:697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:36.542850 32087 consensus_queue.cc:237] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:36.532718 31808 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:36.547132 32037 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:36.549341 25320 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:10:36.551707 25320 raft_consensus.cc:606] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:36.531358 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:36.552117 25320 raft_consensus.cc:614] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:10:36.469345 32068 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:36.547672 32037 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:36.540670 32177 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:36.549091 32178 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:36.549293 32222 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:36.535073 32031 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:36.608796 25320 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:10:36.670028 25320 raft_consensus.cc:606] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:36.670749 25320 raft_consensus.cc:614] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:10:36.646834 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 186011us
I20260709 13:10:36.681002 25238 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
W20260709 13:10:36.584652 32218 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
I20260709 13:10:36.685829 25237 raft_consensus.cc:1240] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 21. Current term is 22. Ops: []
W20260709 13:10:36.589756 32215 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.143s	user 0.002s	sys 0.000s
I20260709 13:10:36.686514 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:10:36.686115 25908 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:36.561897 32114 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.120s
I20260709 13:10:36.689680 32179 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:36.690135 32217 log.cc:927] Time spent T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 13:10:36.682431 32222 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:36.682235 32031 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:36.610071 31967 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:36.697504 31967 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:36.698086 31967 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:36.698449 31861 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:36.700623 32177 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:36.701203 32222 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:36.685448 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 175240us
I20260709 13:10:36.642462 32037 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:36.561781 31577 consensus_queue.cc:1059] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.123s
I20260709 13:10:36.663491 32082 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:36.712589 32179 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:36.712294 32082 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:36.713730 31577 raft_consensus.cc:3055] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:36.714154 31577 raft_consensus.cc:740] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:36.715098 31577 consensus_queue.cc:260] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:36.716286 31577 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:36.719107 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 157780us
W20260709 13:10:36.705119 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 225772us
I20260709 13:10:36.570055 32107 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:36.687479 32169 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:36.724653 25284 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:36.731541 25908 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:36.736074 25209 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:36.740871 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 229601us
I20260709 13:10:36.741328 25209 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:36.741804 25285 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:36.743783 25359 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:36.721807 32219 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:36.739456 32198 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:36.748857 25937 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:36.730901 32120 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:36.753690 25359 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:36.750875 25285 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:36.756345 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:36.757259 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:36.757980 25319 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:36.764808 25287 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:36.744184 32177 raft_consensus.cc:2764] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election decision vote started in defunct term 19: won
I20260709 13:10:36.764508 31886 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 13:10:36.784546 32211 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.605s	user 0.002s	sys 0.000s
I20260709 13:10:36.784395 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:36.768802 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:36.770151 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:36.787642 25387 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:36.771045 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:36.789305 25386 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:36.790477 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:36.791229 25386 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:36.794222 25209 leader_election.cc:400] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 14 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15.
I20260709 13:10:36.797469 25395 raft_consensus.cc:1240] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:36.806517 32219 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:36.807018 32242 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:36.807515 32242 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:36.807955 32242 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:36.744793 32222 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.045s
I20260709 13:10:36.747373 32144 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:36.814385 32144 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:36.754165 32087 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.066s
I20260709 13:10:36.772181 25312 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:36.826685 32107 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:36.786072 32072 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:36.799887 31886 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:36.784754 32133 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:36.858381 25391 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:36.774724 25319 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:36.778574 25389 raft_consensus.cc:1240] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 13:10:36.780144 32093 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:36.752916 32090 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:36.749750 32042 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:36.781427 32226 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:36.746575 32006 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 13:10:36.749715 32114 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:36.749593 31808 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 13:10:36.753064 32234 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:36.782892 32032 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:36.921545 32032 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:36.783423 32238 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:10:36.871294 32233 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 13:10:36.843458 32068 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:36.847478 32072 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:36.857450 31886 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:36.922860 32090 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:36.961956 32090 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:36.959599 32072 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:36.939354 32234 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:36.918264 32114 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:36.994421 32114 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:37.001526 25320 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
W20260709 13:10:37.015256 32196 log.cc:927] Time spent T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 1.144s	user 0.001s	sys 0.002s
W20260709 13:10:37.037047 32186 log.cc:927] Time spent T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 1.553s	user 0.007s	sys 0.000s
I20260709 13:10:37.041461 32118 raft_consensus.cc:3055] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:37.042022 32118 raft_consensus.cc:740] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:37.043035 32118 consensus_queue.cc:260] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:37.044428 32118 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:37.054090 31386 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:37.054656 31386 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:37.063306 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:10:37.104151 31850 raft_consensus.cc:3055] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:37.104683 31850 raft_consensus.cc:740] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:37.105705 31850 consensus_queue.cc:260] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.106775 31850 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:37.126847 32042 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:37.127892 31799 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:37.135548 31799 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:37.127591 32042 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.128494 32112 consensus_queue.cc:1059] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.014s
I20260709 13:10:37.142474 32112 consensus_queue.cc:1243] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer f646dfd532974dceadfe00005d2bbf59 log is divergent from this leader: its last log entry 15.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:10:37.160957 32238 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:37.161777 32238 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.162652 32225 raft_consensus.cc:3055] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:37.163331 32225 raft_consensus.cc:740] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:37.164160 32225 consensus_queue.cc:260] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.165480 32225 raft_consensus.cc:3060] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:37.180500 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 649170us
I20260709 13:10:37.181069 25911 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:37.181375 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:37.185179 25240 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 18. Current term is 19. Ops: [18.15-18.15]
W20260709 13:10:37.179471 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 414676us
I20260709 13:10:37.214268 25358 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:37.214179 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:37.217024 25911 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:37.217974 25389 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:10:37.219645 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:37.228473 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:37.231503 25319 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:10:37.234537 25237 raft_consensus.cc:2393] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 20: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:10:37.235250 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 470495us
W20260709 13:10:37.235500 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:37.238255 25358 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:37.240110 25358 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:37.241657 25387 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:37.241881 25391 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:10:37.245041 25908 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:37.247877 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:37.256026 25358 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:37.259363 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:37.260663 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:37.261453 25245 raft_consensus.cc:3060] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:37.265694 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:37.237186 32101 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:37.234388 32199 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:10:37.266294 25243 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
W20260709 13:10:37.269263 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 46353.8us
W20260709 13:10:37.271399 25285 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.012s	sys 0.066s
W20260709 13:10:37.273278 25285 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.012s	sys 0.066s
I20260709 13:10:37.218730 32177 consensus_queue.cc:1059] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 13:10:37.277624 25313 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 13:10:37.241350 32093 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:37.285849 25237 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 13:10:37.278647 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 515464us
I20260709 13:10:37.265406 32091 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:10:37.291349 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:37.256117 32144 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:37.298730 25908 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:37.265571 32222 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:37.303367 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2887) took 1797 ms. Trace:
I20260709 13:10:37.307518 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:37.309211 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:37.272600 32101 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:37.275655 32033 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:37.312711 25243 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:37.280758 32082 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:37.318369 25319 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:37.252921 31898 consensus_queue.cc:1059] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 13:10:37.326264 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:37.325322 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:37.340600 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:37.297698 32133 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:37.342379 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:37.274361 32175 consensus_queue.cc:1059] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.041s
I20260709 13:10:37.306543 32222 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:37.306758 31808 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:37.345155 25287 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:37.334968 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 52047.3us
W20260709 13:10:37.367120 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 111560us
I20260709 13:10:37.329620 32101 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:37.315035 31799 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:37.299144 32093 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:37.337343 32169 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:37.331559 32091 raft_consensus.cc:697] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:37.392969 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:37.408547 32083 consensus_meta.cc:287] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.962s	user 0.012s	sys 0.004s
I20260709 13:10:37.409104 32083 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.392441 32224 consensus_queue.cc:1059] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.144s
I20260709 13:10:37.417320 32221 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.117s
I20260709 13:10:37.418637 32236 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:37.346194 32234 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:37.420320 32227 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 13:10:37.421365 32031 consensus_meta.cc:287] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.728s	user 0.012s	sys 0.005s
I20260709 13:10:37.421837 32031 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.385375 32222 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:10:37.423939 25192 auto_rebalancer.cc:629] Failed to schedule move for tablet 7f37bfaeb12446c1a84af172a6f3dde9: Illegal state: Replica 46de6919f85341c6a9b574dff112e35a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 14 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
W20260709 13:10:37.425602 32179 consensus_meta.cc:287] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.713s	user 0.020s	sys 0.023s
I20260709 13:10:37.426061 32179 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:37.429291 31861 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:37.429884 32032 consensus_meta.cc:287] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.508s	user 0.008s	sys 0.009s
I20260709 13:10:37.429929 31861 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.449039 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:37.449927 32037 consensus_meta.cc:287] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.742s	user 0.009s	sys 0.008s
I20260709 13:10:37.450363 32037 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.431018 32093 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:37.452101 25192 auto_rebalancer.cc:629] Failed to schedule move for tablet 9c23d7a710c04baf86ab9ba1a2ddc9cf: Illegal state: Replica 2329e428ebbf484fa3ea9452133df567 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 18 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:37.452535 32037 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:37.453763 25238 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:37.458528 25941 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 13:10:37.460273 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:37.460546 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:37.460886 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:37.461650 25314 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
W20260709 13:10:37.432098 32242 consensus_meta.cc:287] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.624s	user 0.013s	sys 0.035s
I20260709 13:10:37.399636 32169 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:37.400943 31799 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:10:37.419853 31967 consensus_meta.cc:287] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.721s	user 0.014s	sys 0.004s
I20260709 13:10:37.433213 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:37.433331 32260 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:10:37.419840 32091 consensus_queue.cc:237] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.396806 32230 raft_consensus.cc:2804] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:37.436982 32090 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:37.440522 25315 raft_consensus.cc:1275] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:10:37.392786 32223 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.152s
I20260709 13:10:37.423971 32031 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:37.307579 25316 rpcz_store.cc:276] 0709 13:10:35.490873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:35.491034 (+   161us) service_pool.cc:224] Handling call
0709 13:10:35.517683 (+ 26649us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:35.519126 (+  1443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:35.519156 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:35.519663 (+   507us) log.cc:844] Serialized 52 byte log entry
0709 13:10:35.670901 (+151238us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:36.808161 (+1137260us) spinlock_profiling.cc:243] Waited 5.46 ms on lock 0x7b18004266a0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f297685a4bc 00007f2973ba848b 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc2455 00007f297a6aa3c7 00007f2976cbc9a7 00007f2976ccc0bf
0709 13:10:36.808356 (+   195us) log.cc:844] Serialized 14 byte log entry
0709 13:10:37.268511 (+460155us) log.cc:844] Serialized 14 byte log entry
0709 13:10:37.269961 (+  1450us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:37.270085 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:37.282399 (+ 12314us) raft_consensus.cc:1685] finished
0709 13:10:37.282421 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:37.288049 (+  5628us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14251,"prepare.queue_time_us":342201,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":67,"raft.queue_time_us":3626,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":10474,"spinlock_wait_cycles":12550912,"thread_start_us":733919,"threads_started":3,"wal-append.queue_time_us":300321}
I20260709 13:10:37.447220 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:37.463369 25245 raft_consensus.cc:2468] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:37.456075 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:37.517342 25937 raft_consensus.cc:2393] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 19: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:37.446904 32222 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.472795 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:37.478453 25909 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:37.487370 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:37.493511 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:37.437529 25243 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 21.
I20260709 13:10:37.471974 32200 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:37.493853 32225 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:37.469259 32032 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:37.512261 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:37.469515 32169 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:10:37.505591 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 259510us
I20260709 13:10:37.465082 32107 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:37.504215 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 248280us
I20260709 13:10:37.453089 32037 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:37.555347 25317 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 17. Current term is 18. Ops: []
W20260709 13:10:37.504612 32178 consensus_meta.cc:287] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.820s	user 0.019s	sys 0.027s
I20260709 13:10:37.538084 25936 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20260709 13:10:37.538175 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 59850.6us
I20260709 13:10:37.494104 31967 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:37.546504 32114 consensus_meta.cc:287] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.547s	user 0.012s	sys 0.004s
I20260709 13:10:37.546790 32032 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:37.547345 31386 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.469498 32242 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:37.549988 31990 consensus_meta.cc:287] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.041s	user 0.006s	sys 0.013s
W20260709 13:10:37.553895 32249 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.185s	user 0.000s	sys 0.000s
I20260709 13:10:37.536602 32200 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:37.561297 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 118452us
I20260709 13:10:37.515843 32238 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:37.518435 32175 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:37.554754 32251 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
W20260709 13:10:37.581087 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 206854us
I20260709 13:10:37.538483 32199 raft_consensus.cc:697] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 22 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:37.582015 25395 raft_consensus.cc:1240] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:37.553965 32222 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:37.584404 25312 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:37.586597 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:37.618198 25393 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:37.587083 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:37.588779 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:37.589818 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:37.619138 25391 raft_consensus.cc:3060] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:37.589882 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:37.575160 25390 consensus_queue.cc:237] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:10:37.579603 25209 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:37.537563 32230 raft_consensus.cc:697] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:10:37.631322 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 51788.7us
I20260709 13:10:37.554987 32225 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:37.570156 32107 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.643680 25393 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
W20260709 13:10:37.631167 32208 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
W20260709 13:10:37.642500 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 103207us
I20260709 13:10:37.647341 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:37.592578 31808 raft_consensus.cc:697] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:37.567893 32037 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:37.568528 32114 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:37.578006 31990 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:37.567662 32178 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:37.655061 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:37.644835 32235 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:37.680455 25237 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:37.678875 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 291235us
I20260709 13:10:37.681973 25287 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:37.638909 32101 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.677626 32268 raft_consensus.cc:3055] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:37.711298 32266 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:37.677788 32235 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:37.710721 32178 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:37.712050 32224 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:37.712309 32082 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:37.712679 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 65426.1us
I20260709 13:10:37.712782 31386 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:37.671347 32091 consensus_queue.cc:1059] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 13:10:37.713853 32230 consensus_queue.cc:237] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.715314 32242 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:37.615049 32175 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.698294 32263 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:37.717695 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:37.718288 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 70723.1us
I20260709 13:10:37.690979 32068 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:37.705459 31808 consensus_queue.cc:237] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:37.685004 25287 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:37.735952 31967 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:37.736495 31577 consensus_meta.cc:287] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.020s	user 0.003s	sys 0.053s
I20260709 13:10:37.737275 32278 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 13:10:37.691606 31886 consensus_meta.cc:287] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.731s	user 0.018s	sys 0.000s
I20260709 13:10:37.738432 32268 raft_consensus.cc:740] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:37.740207 32235 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:37.766148 25358 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:37.766527 25318 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:10:37.762917 25909 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:37.769093 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 84181.5us
I20260709 13:10:37.775451 25394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 2444) took 1005 ms. Trace:
I20260709 13:10:37.779066 25393 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:10:37.782181 25287 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:37.772686 32118 consensus_meta.cc:287] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.728s	user 0.009s	sys 0.018s
I20260709 13:10:37.784456 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:37.784471 25237 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:37.776012 25394 rpcz_store.cc:276] 0709 13:10:36.770121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:36.770270 (+   149us) service_pool.cc:224] Handling call
0709 13:10:36.770539 (+   269us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:36.797776 (+ 27237us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:36.797822 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:36.798427 (+   605us) log.cc:844] Serialized 26 byte log entry
0709 13:10:37.227287 (+428860us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:37.341615 (+114328us) log.cc:844] Serialized 14 byte log entry
0709 13:10:37.775036 (+433421us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:37.775151 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:37.775209 (+    58us) raft_consensus.cc:1685] finished
0709 13:10:37.775229 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:37.775388 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":299726,"prepare.queue_time_us":99808,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":3069,"raft.queue_time_us":307779,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":19021,"thread_start_us":686046,"threads_started":3,"wal-append.queue_time_us":453847}
I20260709 13:10:37.788280 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:37.789239 25243 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:37.789304 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:37.771378 31386 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:37.792721 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:37.793365 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:37.794967 25317 raft_consensus.cc:2368] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 17. Current term is 18.
I20260709 13:10:37.682925 32199 consensus_queue.cc:237] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.757445 32082 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.814791 32268 consensus_queue.cc:260] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.745678 32266 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:37.815892 32225 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:37.769949 32224 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.021s
I20260709 13:10:37.816509 25391 raft_consensus.cc:2468] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 21.
I20260709 13:10:37.817788 32175 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:37.781184 31850 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:37.796409 32270 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.025s
I20260709 13:10:37.772571 31886 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.823608 32279 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:10:37.793807 32247 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:37.825500 31990 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:37.776124 31789 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:37.796373 32120 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:37.771457 32222 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
I20260709 13:10:37.828434 32037 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:37.785557 31577 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:37.829593 31898 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:37.830358 32101 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:37.887293 31577 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.889652 31577 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:37.890692 31577 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:37.891369 31577 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.892204 32042 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:37.892426 32266 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:37.894408 32107 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:37.878325 32114 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:37.898345 31577 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:37.832320 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:37.791505 25287 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:37.902558 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 111163us
I20260709 13:10:37.902112 25136 catalog_manager.cc:5697] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 11 to 21, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 21 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:37.903491 25287 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:37.903680 31961 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:37.904273 31961 raft_consensus.cc:697] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:37.812614 32277 raft_consensus.cc:3055] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:10:37.905323 32277 raft_consensus.cc:740] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:37.905200 31961 consensus_queue.cc:237] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:37.905825 32120 raft_consensus.cc:2804] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:10:37.906634 32120 raft_consensus.cc:697] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:37.906713 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:10:37.907665 32072 consensus_meta.cc:287] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.916s	user 0.017s	sys 0.001s
I20260709 13:10:37.907588 32120 consensus_queue.cc:237] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:37.908118 32072 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:37.786291 32068 consensus_queue.cc:1059] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
W20260709 13:10:37.847661 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 52368.7us
W20260709 13:10:37.909778 32256 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.517s	user 0.002s	sys 0.002s
I20260709 13:10:37.910518 32068 raft_consensus.cc:3055] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:37.910889 32068 raft_consensus.cc:740] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:37.911233 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:37.911697 32068 consensus_queue.cc:260] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.914085 25390 raft_consensus.cc:1240] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:37.914160 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:37.868275 31850 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:37.878368 32268 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:37.795853 25389 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:37.787328 32263 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:37.848068 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 81987.4us
I20260709 13:10:37.842037 32200 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:37.874483 25243 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
I20260709 13:10:37.920558 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:37.884326 25321 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 21. Current term is 22. Ops: []
I20260709 13:10:37.832607 32169 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.882298 32247 raft_consensus.cc:697] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:37.906206 32277 consensus_queue.cc:260] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.813189 32258 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.046s
W20260709 13:10:37.916477 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:37.912865 32221 consensus_queue.cc:1059] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.004s
I20260709 13:10:37.917646 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:37.966583 32175 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:37.918648 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:37.967481 25394 raft_consensus.cc:2393] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 20: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:37.967167 32175 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.920588 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:37.852937 32278 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 13:10:37.923009 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:37.904057 25136 catalog_manager.cc:5697] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 10 to 18, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 18 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:37.912922 32068 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:37.920993 25359 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:37.866285 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 82793us
W20260709 13:10:37.972980 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 52063.7us
I20260709 13:10:37.971354 32006 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:37.975153 32287 consensus_queue.cc:1059] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.050s
I20260709 13:10:37.976643 31886 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:37.977433 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:37.978581 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:37.978801 31577 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:37.979885 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:37.979560 31577 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.980822 32278 raft_consensus.cc:3055] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:37.981221 32278 raft_consensus.cc:740] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:37.981665 32037 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:10:37.982038 32278 consensus_queue.cc:260] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.983189 32037 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:37.983361 32278 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:37.983863 32037 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:37.923171 32263 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.984570 32037 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:10:37.926714 32224 consensus_queue.cc:1048] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.059s
I20260709 13:10:37.972054 32200 consensus_queue.cc:1059] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.017s
I20260709 13:10:37.986841 32263 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:37.987232 32200 raft_consensus.cc:3055] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:37.987625 32200 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
W20260709 13:10:37.989289 32282 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.100s	user 0.000s	sys 0.002s
I20260709 13:10:37.991446 32178 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:37.991914 32178 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:37.993203 32202 consensus_queue.cc:1059] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:10:37.993854 32082 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:37.994243 32287 raft_consensus.cc:3055] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:37.988484 32200 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:37.994794 32287 raft_consensus.cc:740] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:37.994987 32247 consensus_queue.cc:237] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:37.997408 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:37.997715 25389 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:10:37.916075 31850 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:10:37.961017 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:37.961037 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:37.971074 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 50552.5us
I20260709 13:10:37.981745 32114 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:10:38.006857 32263 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 13:10:38.008162 32250 log.cc:927] Time spent T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.611s	user 0.004s	sys 0.000s
I20260709 13:10:38.013502 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 2445) took 1243 ms. Trace:
I20260709 13:10:38.013928 32198 raft_consensus.cc:3055] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:38.014348 32198 raft_consensus.cc:740] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:38.013881 25392 rpcz_store.cc:276] 0709 13:10:36.770426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:36.770821 (+   395us) service_pool.cc:224] Handling call
0709 13:10:36.771149 (+   328us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:36.786949 (+ 15800us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:36.786995 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:36.829633 (+ 42638us) log.cc:844] Serialized 26 byte log entry
0709 13:10:37.255259 (+425626us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:10:37.373922 (+118663us) log.cc:844] Serialized 14 byte log entry
0709 13:10:37.581022 (+207100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:37.581099 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:38.007598 (+426499us) spinlock_profiling.cc:243] Waited 2.68 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:38.013187 (+  5589us) raft_consensus.cc:1685] finished
0709 13:10:38.013293 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:38.013440 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":291942,"prepare.queue_time_us":242227,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":288957,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":5137,"spinlock_wait_cycles":6162048,"thread_start_us":489035,"threads_started":2,"wal-append.queue_time_us":484833}
I20260709 13:10:38.015349 32178 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:37.960654 25939 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 13:10:38.002414 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:37.995669 32287 consensus_queue.cc:260] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.018455 32169 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:38.019196 32287 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:38.020990 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:38.021498 32290 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:38.022039 32290 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.022847 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:38.023500 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:38.025771 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.027240 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:38.027714 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:38.027838 25944 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 19.
I20260709 13:10:38.028831 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:38.028347 25136 catalog_manager.cc:5697] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 reported cstate change: leader changed from <none> to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 17 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:38.029386 25936 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:38.029803 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.030158 32270 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:38.030587 25390 raft_consensus.cc:2410] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 13:10:38.034252 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:38.035025 25243 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 19: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:38.024104 25943 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:38.004588 25317 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:38.015305 32198 consensus_queue.cc:260] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.029956 25136 catalog_manager.cc:5697] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:38.038269 25936 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:37.995590 32200 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:38.005376 32091 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:38.054802 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:38.064301 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:38.065521 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:38.068464 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:10:38.086681 25239 raft_consensus.cc:2393] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 22: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:38.096025 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:38.107105 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:38.112934 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:38.024160 32290 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:37.999269 32175 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:38.079418 25241 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:10:38.061709 32277 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:38.118428 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:38.122941 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:38.094553 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:38.061585 32114 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:38.130249 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:38.131819 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:38.136060 25246 raft_consensus.cc:2393] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 21: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:38.137508 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:38.138155 25238 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:38.140882 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:38.078555 25943 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 21.
I20260709 13:10:38.147406 25909 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:38.149576 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:38.151266 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:38.152729 25239 raft_consensus.cc:2393] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 18: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:38.160751 25238 raft_consensus.cc:2468] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
W20260709 13:10:38.162424 25144 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39582 (request call id 309) took 58829 ms (client timeout 15000 ms). Trace:
I20260709 13:10:38.165575 25285 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:38.164915 25209 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:38.171505 25909 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:38.174742 32037 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:38.179785 32037 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:38.180253 32037 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:10:38.163192 25144 rpcz_store.cc:269] 0709 13:09:39.333033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:39.356517 (+ 23484us) service_pool.cc:224] Handling call
0709 13:10:38.145354 (+58788837us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:10:34.341221 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:10:34.342015 (+   794us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:10:34.342064 (+    49us) write_op.cc:435] Acquired schema lock
0709 13:10:34.342078 (+    14us) tablet.cc:662] Decoding operations
0709 13:10:34.343348 (+  1270us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:10:34.343465 (+   117us) write_op.cc:641] Partition lock acquired for write op
0709 13:10:34.343513 (+    48us) tablet.cc:685] Acquiring locks for 9 operations
0709 13:10:34.344550 (+  1037us) tablet.cc:701] Row locks acquired
0709 13:10:34.344624 (+    74us) write_op.cc:260] PREPARE: finished
0709 13:10:34.344876 (+   252us) write_op.cc:270] Start()
0709 13:10:34.345060 (+   184us) write_op.cc:276] Timestamp: P: 1783602634344790 usec, L: 0
0709 13:10:34.345110 (+    50us) op_driver.cc:348] REPLICATION: starting
0709 13:10:34.364452 (+ 19342us) log.cc:844] Serialized 3636 byte log entry
0709 13:10:34.835594 (+471142us) op_driver.cc:464] REPLICATION: finished
0709 13:10:34.957277 (+121683us) write_op.cc:301] APPLY: starting
0709 13:10:34.957411 (+   134us) tablet.cc:1367] starting BulkCheckPresence
0709 13:10:35.450291 (+492880us) tablet.cc:1370] finished BulkCheckPresence
0709 13:10:35.450414 (+   123us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:10:35.516239 (+ 65825us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:10:35.516598 (+   359us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=18,key_file_lookups=18,delta_file_lookups=18,mrs_lookups=0
0709 13:10:35.516669 (+    71us) write_op.cc:312] APPLY: finished
0709 13:10:35.517166 (+   497us) log.cc:844] Serialized 89 byte log entry
0709 13:10:35.517376 (+   210us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:10:35.517496 (+   120us) write_op.cc:454] Released schema lock
0709 13:10:35.517557 (+    61us) write_op.cc:341] FINISH: Updating metrics
0709 13:10:35.520439 (+  2882us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:10:35.520473 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":89842,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":121314,"apply.run_cpu_time_us":18417,"apply.run_wall_time_us":563207,"cfile_cache_hit":38,"cfile_cache_hit_bytes":41711,"num_ops":9,"prepare.queue_time_us":396504,"prepare.run_cpu_time_us":39245,"prepare.run_wall_time_us":233826,"raft.queue_time_us":122176,"raft.run_cpu_time_us":82563,"raft.run_wall_time_us":122701,"replication_time_us":472266,"thread_start_us":642539,"threads_started":4,"wal-append.queue_time_us":338449,"wal-append.run_cpu_time_us":37137,"wal-append.run_wall_time_us":1807391}]]}
I20260709 13:10:38.092905 32091 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.185966 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:38.179898 32292 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:10:38.178675 32169 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:10:38.172595 31961 raft_consensus.cc:2804] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:38.177670 31990 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:38.170159 31967 consensus_queue.cc:1059] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.065s
I20260709 13:10:38.239195 31577 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:38.185719 32072 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:38.188685 32260 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:38.208662 32290 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:38.259034 32244 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:38.259609 32244 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.260758 32235 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:38.261806 32244 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:38.263602 32235 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:38.263751 25390 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:38.265300 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.266091 25390 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 16: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:38.233433 32263 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:38.267823 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:38.269024 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.269330 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.269774 25394 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 20: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:38.270702 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:38.291795 25393 raft_consensus.cc:2393] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 17: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:38.295033 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.295629 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.296377 25388 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 19.
I20260709 13:10:38.296729 25909 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:38.297271 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:38.297472 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.299620 25393 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:38.302459 25394 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:10:38.303594 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.305006 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:38.305689 25388 raft_consensus.cc:2368] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 17. Current term is 18.
I20260709 13:10:38.306500 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.306425 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:38.118096 25136 catalog_manager.cc:5697] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:38.251935 31961 raft_consensus.cc:697] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:38.250319 31990 raft_consensus.cc:697] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:38.269905 32260 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:38.322620 32260 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:38.321935 25136 catalog_manager.cc:5697] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 6 to 21, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 21 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:38.323833 25136 catalog_manager.cc:5697] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 13 to 19. New cstate: current_term: 19 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:38.325621 25136 catalog_manager.cc:5697] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 10 to 17. New cstate: current_term: 17 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:38.252569 31850 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.329818 32091 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:38.330511 25938 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:38.332221 25387 raft_consensus.cc:2410] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 19 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260709 13:10:38.329955 25136 catalog_manager.cc:5697] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 12 to 15, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 15 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:38.304744 32263 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:38.333521 32290 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:38.333410 32263 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.267344 31782 raft_consensus.cc:3055] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:38.335657 31782 raft_consensus.cc:740] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:38.336442 32274 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:38.336627 31782 consensus_queue.cc:260] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.292125 32223 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:38.339440 32223 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.344528 31990 consensus_queue.cc:237] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.351154 32093 raft_consensus.cc:3055] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:38.351768 32093 raft_consensus.cc:740] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:10:38.360764 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 224993us
I20260709 13:10:38.243870 32114 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:38.337153 32274 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:38.363814 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:38.364452 25239 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:38.331107 31850 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:38.333691 25136 catalog_manager.cc:5697] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 12 to 18. New cstate: current_term: 18 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:38.368171 25284 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:38.366533 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.369860 25241 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:10:38.373510 25393 raft_consensus.cc:2393] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 19: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:38.418828 25390 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:38.416064 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.369207 25136 catalog_manager.cc:5697] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 11 to 21, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 21 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:38.360826 32169 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.045s
W20260709 13:10:38.436132 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 259801us
I20260709 13:10:38.363458 31808 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.050s
I20260709 13:10:38.406756 31386 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:38.406441 32119 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:38.408215 32244 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:38.352912 32093 consensus_queue.cc:260] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:38.442281 25284 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:38.445454 32178 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:38.428881 31952 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:38.444144 25359 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:38.424307 32263 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:38.461403 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:38.472117 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:38.475682 25315 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:38.472954 25312 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 13:10:38.483136 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:38.483765 25317 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:38.489187 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.423725 31961 consensus_queue.cc:237] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:38.477474 32107 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 13:10:38.498006 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 201292us
I20260709 13:10:38.495787 25212 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:38.479821 32093 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:38.481684 32224 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:38.504148 25212 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:38.466338 31386 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:38.468696 32119 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.515663 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:38.470443 32244 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:38.496544 32133 raft_consensus.cc:3055] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:38.521850 25940 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:38.522765 32298 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:38.523315 32298 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:38.511147 31990 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:38.526305 25136 catalog_manager.cc:5697] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 12 to 17, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 17 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:38.528183 25136 catalog_manager.cc:5697] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:38.529737 32083 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:38.530261 25136 catalog_manager.cc:5697] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 10 to 14, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 14 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:38.532668 31782 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:38.532573 25136 catalog_manager.cc:5697] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 12 to 18, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 18 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:38.478650 32178 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:38.516832 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:38.514283 32199 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:38.517256 25944 raft_consensus.cc:1240] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:38.518167 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:38.578330 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:38.505656 32224 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:38.507695 25212 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:38.569561 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:38.506502 32235 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:38.575217 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:38.499348 32260 consensus_queue.cc:1059] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.021s
I20260709 13:10:38.506603 32107 raft_consensus.cc:697] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:10:38.578529 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:38.519927 32133 raft_consensus.cc:740] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:10:38.534725 32278 consensus_meta.cc:287] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.551s	user 0.013s	sys 0.000s
I20260709 13:10:38.500777 25909 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:38.590531 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:38.510110 32216 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.073s
W20260709 13:10:38.584570 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 77002.6us
W20260709 13:10:38.594501 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 93836.2us
I20260709 13:10:38.593779 25212 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:38.595240 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:38.597517 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:38.597829 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:38.598866 32296 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
W20260709 13:10:38.600538 32288 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.609s	user 0.000s	sys 0.007s
I20260709 13:10:38.601384 32178 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:38.602530 25393 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:38.603902 25212 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:38.604260 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:38.601966 32169 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:38.605194 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:38.605867 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:38.606185 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:38.586437 32199 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.583747 32037 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.605810 25239 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:38.602394 32270 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:38.616612 25136 catalog_manager.cc:5697] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 12 to 18, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 18 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:38.607331 32179 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:38.602766 32223 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:38.603013 32274 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:38.630571 32037 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:38.631266 25240 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:38.622874 32042 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.009s
I20260709 13:10:38.613602 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:38.632853 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:38.633769 32199 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:38.634016 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:38.629097 25136 catalog_manager.cc:5697] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 12 to 15, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 15 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:38.613883 25358 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:38.611609 32107 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.621384 32298 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:38.613947 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:38.636554 32298 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:38.618590 32177 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:38.637405 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:38.637952 25940 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:38.640580 25937 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:38.641748 25944 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 22.
I20260709 13:10:38.641278 25136 catalog_manager.cc:5697] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 17 to 19, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 19 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:38.586128 32224 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:38.587826 31990 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:38.635020 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:38.649415 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:38.655587 25936 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:38.647387 25937 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 13:10:38.624626 32244 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:38.650961 32068 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:38.655850 25239 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:38.663928 25284 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:38.635039 32287 consensus_meta.cc:287] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.615s	user 0.010s	sys 0.000s
I20260709 13:10:38.665663 25945 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:38.653838 25136 catalog_manager.cc:5697] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 11 to 19. New cstate: current_term: 19 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:38.672101 25284 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:38.678074 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 74239.4us
I20260709 13:10:38.679709 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.626188 32119 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:38.624277 31386 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:38.683146 25212 leader_election.cc:400] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 17. Current term is 18.
I20260709 13:10:38.655375 31990 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:38.723452 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.707463 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 1222) took 1388 ms. Trace:
I20260709 13:10:38.727571 25321 rpcz_store.cc:276] 0709 13:10:37.319132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:37.319240 (+   108us) service_pool.cc:224] Handling call
0709 13:10:37.319487 (+   247us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:37.923502 (+604015us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:38.707168 (+783666us) spinlock_profiling.cc:243] Waited 565 ms on lock 0x7b5400044554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:38.707386 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1298592000}
I20260709 13:10:38.735704 25287 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:38.679342 32236 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 13:10:38.696352 32200 consensus_meta.cc:287] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.616s	user 0.009s	sys 0.005s
I20260709 13:10:38.683204 32274 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:38.740399 25392 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:10:38.741494 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:38.742875 25941 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:38.689244 25212 leader_election.cc:403] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
W20260709 13:10:38.753583 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 71159.4us
I20260709 13:10:38.753692 25287 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:38.760836 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:38.746991 25358 consensus_peers.cc:597] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b62f4877af9742298b1e149789ca3090. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:38.767661 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 83241.9us
I20260709 13:10:38.676592 32068 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.702461 31990 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:38.768138 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.768862 25358 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:38.769070 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.713423 32298 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:38.772796 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:38.775063 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:38.779259 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.778690 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.780112 25392 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 16: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:38.774397 25287 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:38.785899 25287 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:38.787473 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.706732 32287 consensus_queue.cc:1059] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 19.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.067s
I20260709 13:10:38.757634 32274 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:38.707401 32290 consensus_queue.cc:1048] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.066s
I20260709 13:10:38.791118 25943 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:38.694677 32243 consensus_queue.cc:1059] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.088s
I20260709 13:10:38.793066 25287 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:38.773381 32144 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:38.769699 32247 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:10:38.781991 31782 raft_consensus.cc:2804] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:10:38.768289 32318 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 17. Current term is 18.
I20260709 13:10:38.788252 32107 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:38.782298 32246 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:38.788725 32236 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:38.804312 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:38.802076 31961 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:38.823552 32068 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:38.804037 32144 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:38.792131 32274 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:38.787590 32298 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:38.718430 25136 catalog_manager.cc:5697] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 12 to 16, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 16 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:38.825588 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:38.781868 32244 consensus_queue.cc:1059] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.021s
I20260709 13:10:38.834725 31782 raft_consensus.cc:697] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:38.872109 32244 raft_consensus.cc:3055] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:10:38.872639 32244 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:38.872426 31782 consensus_queue.cc:237] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.873835 32244 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.841883 25939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59304 (request call id 1281) took 1386 ms. Trace:
I20260709 13:10:38.846292 32235 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:38.847270 32270 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:38.848330 32119 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:38.876281 32119 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:38.850800 25246 raft_consensus.cc:1275] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 11. (term mismatch)
I20260709 13:10:38.877336 25246 pending_rounds.cc:85] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 10
I20260709 13:10:38.877581 25246 pending_rounds.cc:107] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260709 13:10:38.877794 25246 raft_consensus.cc:2889] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:10:38.856238 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 52012us
I20260709 13:10:38.858886 32033 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:38.861007 31861 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:38.866174 25136 catalog_manager.cc:5697] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 15 to 18, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 18 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:38.870853 32198 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:38.875118 25939 rpcz_store.cc:276] 0709 13:10:37.455228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:37.469371 (+ 14143us) service_pool.cc:224] Handling call
0709 13:10:37.503297 (+ 33926us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:38.024443 (+521146us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:38.841693 (+817250us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b5400671e54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:38.841835 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1022381824}
I20260709 13:10:38.875517 32244 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:38.889797 25387 raft_consensus.cc:1275] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:10:38.898994 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:38.863397 32182 raft_consensus.cc:3055] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:38.896231 32068 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.005s
I20260709 13:10:38.907981 32298 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:38.909830 25392 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 13:10:38.923089 25393 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16. Ops: []
W20260709 13:10:38.935015 32286 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.810s	user 0.000s	sys 0.004s
W20260709 13:10:38.937314 32313 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.414s	user 0.002s	sys 0.000s
I20260709 13:10:38.939205 25395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 1469) took 1022 ms. Trace:
I20260709 13:10:38.940322 32144 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:38.939600 25395 rpcz_store.cc:276] 0709 13:10:37.917029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:37.917354 (+   325us) service_pool.cc:224] Handling call
0709 13:10:37.923009 (+  5655us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:37.923395 (+   386us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:37.923429 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:37.924770 (+  1341us) log.cc:844] Serialized 52 byte log entry
0709 13:10:38.355053 (+430283us) spinlock_profiling.cc:243] Waited 4.29 ms on lock 0x7b1800489400. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f29768520d1 00007f2976c6da0f
0709 13:10:38.516406 (+161353us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:38.516447 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:38.516507 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:38.936697 (+420190us) spinlock_profiling.cc:243] Waited 2.37 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:38.938955 (+  2258us) raft_consensus.cc:1685] finished
0709 13:10:38.938980 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:38.939143 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":117005,"raft.queue_time_us":2304,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":34627,"spinlock_wait_cycles":15331200,"thread_start_us":161156,"threads_started":1,"wal-append.queue_time_us":581429}
I20260709 13:10:38.945374 32175 consensus_queue.cc:1059] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:38.946955 32083 raft_consensus.cc:3055] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:38.947422 32083 raft_consensus.cc:740] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:38.948414 32083 consensus_queue.cc:260] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.949875 32083 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:10:38.914685 32272 log.cc:927] Time spent T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 1.163s	user 0.004s	sys 0.003s
I20260709 13:10:38.961228 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2836) took 1767 ms. Trace:
I20260709 13:10:38.896428 32287 consensus_queue.cc:1059] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 13:10:38.961634 25242 rpcz_store.cc:276] 0709 13:10:37.193900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:37.194741 (+   841us) service_pool.cc:224] Handling call
0709 13:10:37.195009 (+   268us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:37.195349 (+   340us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:10:37.282211 (+ 86862us) log.cc:844] Serialized 14 byte log entry
0709 13:10:37.580217 (+298006us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:37.615473 (+ 35256us) log.cc:844] Serialized 26 byte log entry
0709 13:10:37.615615 (+   142us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:37.615644 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:37.615707 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:38.904632 (+1288925us) spinlock_profiling.cc:243] Waited 2.73 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:38.960852 (+ 56220us) raft_consensus.cc:1685] finished
0709 13:10:38.960965 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:38.961120 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13246,"prepare.queue_time_us":181947,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":1645,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":32783,"spinlock_wait_cycles":6274304,"thread_start_us":321559,"threads_started":2,"wal-append.queue_time_us":370909}
I20260709 13:10:38.964040 32287 raft_consensus.cc:3055] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:10:38.964562 32287 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:38.965689 32287 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:38.967281 32287 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:38.905988 32182 raft_consensus.cc:740] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:38.973917 25212 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:38.975142 25212 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:38.978813 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:38.979923 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:38.983654 25395 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:10:38.984254 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:38.984768 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:38.985354 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:38.983583 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:38.986174 25314 raft_consensus.cc:2410] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:10:38.986970 25320 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 13:10:38.989341 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:38.992216 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:38.992377 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:38.993183 25318 raft_consensus.cc:2393] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 18: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:38.993597 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:38.993950 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:38.994169 25320 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:38.994920 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:38.997203 25313 raft_consensus.cc:2468] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
W20260709 13:10:38.997601 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:38.997788 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:38.997571 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:38.998545 25314 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:38.976164 32090 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:10:39.000284 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:39.000388 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:39.001032 25316 raft_consensus.cc:3060] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:38.972595 32312 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:38.985375 32270 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:38.984608 32219 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:38.977412 31386 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:38.977043 32119 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:39.007884 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:39.010762 25313 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:10:39.012038 25314 raft_consensus.cc:2410] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 13:10:39.027873 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:39.028685 25313 raft_consensus.cc:2393] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 19: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:39.031574 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:39.016127 31386 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.002539 32224 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.008960 32175 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.021s
I20260709 13:10:39.041926 31386 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:39.045336 32274 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.046528 25321 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 19. Ops: []
I20260709 13:10:39.047717 32274 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:39.048626 25386 raft_consensus.cc:2393] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 20: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:39.049691 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:39.050812 25242 raft_consensus.cc:2410] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 20 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 13:10:39.052367 25246 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:10:39.056273 25238 raft_consensus.cc:1275] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:10:39.011969 32312 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.059094 32133 consensus_queue.cc:260] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.060949 32312 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:39.062263 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1317) took 1039 ms. Trace:
I20260709 13:10:39.062752 25313 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:39.062636 25938 rpcz_store.cc:276] 0709 13:10:38.023186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:38.025341 (+  2155us) service_pool.cc:224] Handling call
0709 13:10:38.025647 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:38.330926 (+305279us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:39.062049 (+731123us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x7b5400690354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:39.062205 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":700679424}
I20260709 13:10:39.064038 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:39.065891 25316 raft_consensus.cc:2468] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:39.066408 25317 raft_consensus.cc:3060] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:39.066560 25358 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:39.070986 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:39.072104 25358 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:39.078821 25316 raft_consensus.cc:1240] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:39.076913 25942 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:39.074724 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:39.016103 32219 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.073621 32039 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:39.077284 32259 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:39.082386 25312 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:39.072278 25287 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:39.084496 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:39.085425 25316 raft_consensus.cc:1275] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:10:39.086203 25313 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:10:39.087561 25316 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
W20260709 13:10:39.088397 32324 log.cc:927] Time spent T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.090s	user 0.002s	sys 0.000s
I20260709 13:10:39.088936 32274 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:39.084671 31386 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:39.089861 31386 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:39.091114 32310 log.cc:927] Time spent T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.569s	user 0.002s	sys 0.000s
I20260709 13:10:39.092077 31386 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:39.092209 32219 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:39.093056 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:39.094027 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:39.094420 32235 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:39.094928 32235 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:39.096177 32033 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:39.098409 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:39.099881 32266 raft_consensus.cc:3055] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:39.100268 32266 raft_consensus.cc:740] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:39.100308 25908 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:39.077320 32284 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:39.084651 32283 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:39.102252 25909 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:39.102562 25239 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:10:39.104981 25941 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 13:10:39.105322 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:39.105892 25938 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:39.106053 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:39.106757 25945 raft_consensus.cc:2393] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 17: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:39.107630 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:39.108860 25358 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:39.102200 32083 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:39.107703 25908 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:39.112236 25317 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:39.112350 25312 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:39.096880 32235 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:39.116192 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:39.117146 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:39.117456 25908 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:39.116210 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:39.115854 25315 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
I20260709 13:10:39.117280 25939 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:39.119238 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:39.119464 31961 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:10:39.120574 32306 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:39.122982 32297 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:39.123317 25240 raft_consensus.cc:1275] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 13:10:39.123404 32297 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:39.123943 32297 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:39.125411 32263 raft_consensus.cc:3055] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:39.125851 32263 raft_consensus.cc:740] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:39.108981 32266 consensus_queue.cc:260] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:39.126660 32176 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:39.127072 32266 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:39.126796 32263 consensus_queue.cc:260] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 19.17, Last appended by leader: 17, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.128245 25908 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:39.129001 32176 raft_consensus.cc:3060] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:39.129165 32263 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:10:39.133628 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:39.139354 32235 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:39.140023 32235 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.141458 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:39.142436 32235 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:39.142884 25244 raft_consensus.cc:1240] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:39.143226 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:39.145540 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:39.146006 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:39.146203 25240 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:39.146611 25944 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:39.152165 25285 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:39.153687 25359 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:39.153851 25285 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:39.154459 32032 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:39.155463 32259 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:39.156002 25285 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:39.160946 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:39.161465 25284 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:39.161785 25240 raft_consensus.cc:1240] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:39.015177 32182 consensus_queue.cc:260] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:39.164304 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 92227.5us
I20260709 13:10:39.171425 25320 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:39.183257 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:39.184365 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 75575.2us
I20260709 13:10:39.185096 25358 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:39.183331 25359 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:39.185948 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:39.186070 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:39.203963 25315 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:10:39.207217 25209 leader_election.cc:400] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 17. Current term is 18.
I20260709 13:10:39.157060 32259 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:39.211280 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:39.229622 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 46362.1us
I20260709 13:10:39.231052 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:39.234222 25316 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:39.157104 32236 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:39.157579 32032 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:39.182327 32274 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:10:39.245983 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:39.249359 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:39.249740 25320 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 19.
I20260709 13:10:39.247762 25938 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:39.252949 25284 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:39.229193 32114 raft_consensus.cc:2804] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:39.256358 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:39.209323 31967 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:39.257548 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:39.258361 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:39.259763 25238 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 13:10:39.263423 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:39.231385 32176 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:39.202214 32033 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.069s
I20260709 13:10:39.244501 31898 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:10:39.265035 32323 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.287s	user 0.000s	sys 0.001s
I20260709 13:10:39.171684 31386 consensus_queue.cc:1059] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.089s
I20260709 13:10:39.266633 25358 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:39.247620 32032 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:39.267324 32006 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:39.279203 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:39.182139 32119 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.071s
I20260709 13:10:39.251469 32274 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:39.281800 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:39.228803 32222 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.046s
I20260709 13:10:39.252413 32202 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:39.252162 32182 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:39.249359 32039 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:39.289040 25389 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 14. (term mismatch)
I20260709 13:10:39.327008 25389 pending_rounds.cc:85] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 13
I20260709 13:10:39.327265 25389 pending_rounds.cc:107] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 20.14
I20260709 13:10:39.327562 25389 raft_consensus.cc:2889] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:39.235810 32247 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 13:10:39.294836 25358 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:39.332332 31789 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 13:10:39.295517 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:39.301147 25315 raft_consensus.cc:1240] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:39.210287 32318 consensus_queue.cc:1059] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.066s
I20260709 13:10:39.246156 32236 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:39.337430 32236 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:39.262157 32299 consensus_queue.cc:1059] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.070s
I20260709 13:10:39.341768 32299 raft_consensus.cc:3055] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:39.342248 32299 raft_consensus.cc:740] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:39.343312 32299 consensus_queue.cc:260] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:39.249418 32178 consensus_queue.cc:1048] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20260709 13:10:39.260962 32083 consensus_queue.cc:1048] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.052s
I20260709 13:10:39.251540 32284 consensus_queue.cc:1059] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.061s
I20260709 13:10:39.348794 32284 consensus_queue.cc:1243] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer 2329e428ebbf484fa3ea9452133df567 log is divergent from this leader: its last log entry 16.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:10:39.280231 31898 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:39.350354 31898 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:39.276877 32114 raft_consensus.cc:697] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:39.267338 32198 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.052s
I20260709 13:10:39.326040 25394 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 16: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:39.355232 32114 consensus_queue.cc:237] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.332998 31789 raft_consensus.cc:697] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:39.357324 31789 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.332696 25358 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:39.361232 25394 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:39.362260 32144 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:39.362998 32144 raft_consensus.cc:697] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:39.364444 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:39.366931 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:39.364058 32144 consensus_queue.cc:237] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.344967 32299 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:39.395401 31967 consensus_queue.cc:1059] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:10:39.262751 32306 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 13:10:39.398855 25938 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 19. Ops: []
I20260709 13:10:39.314018 32032 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:39.314553 32243 raft_consensus.cc:2804] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:39.314232 32068 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:39.319844 32274 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:39.293658 31577 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.115s
W20260709 13:10:39.397603 32272 log.cc:927] Time spent T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 13:10:39.320017 32312 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.103s
W20260709 13:10:39.398305 31782 consensus_queue.cc:1175] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "2329e428ebbf484fa3ea9452133df567" responder_term: 17 status { last_received { term: 17 index: 10 } last_committed_idx: 10 last_received_current_leader { term: 17 index: 10 } }
I20260709 13:10:39.319880 32006 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.401525 31808 consensus_queue.cc:1059] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.080s
I20260709 13:10:39.560915 25390 raft_consensus.cc:3055] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:39.627341 25390 raft_consensus.cc:740] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:10:39.446480 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:39.439603 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:39.630110 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:39.581055 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:10:39.575277 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 337863us
W20260709 13:10:39.654062 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 413373us
I20260709 13:10:39.655388 25285 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:39.407850 32246 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:10:39.551640 31990 consensus_meta.cc:287] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.760s	user 0.010s	sys 0.000s
W20260709 13:10:39.477819 32334 log.cc:927] Time spent T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
W20260709 13:10:39.427673 32326 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
W20260709 13:10:39.433580 32325 log.cc:927] Time spent T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.349s	user 0.000s	sys 0.001s
I20260709 13:10:39.478022 32219 raft_consensus.cc:3055] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:39.466197 32243 raft_consensus.cc:697] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:39.556058 32133 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:39.464955 32107 consensus_meta.cc:287] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.660s	user 0.006s	sys 0.001s
I20260709 13:10:39.402333 32039 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:39.408874 32244 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.250s
I20260709 13:10:39.467348 32068 raft_consensus.cc:697] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:39.557363 32224 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:39.666039 25390 consensus_queue.cc:260] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:39.682776 25390 raft_consensus.cc:3060] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:39.414482 32236 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:39.654311 32198 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:39.658334 32247 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:10:39.667372 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 414484us
W20260709 13:10:39.660533 32283 consensus_meta.cc:287] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.558s	user 0.009s	sys 0.001s
I20260709 13:10:39.701611 25390 raft_consensus.cc:1275] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
I20260709 13:10:39.702122 25390 pending_rounds.cc:85] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 8
I20260709 13:10:39.702447 25390 pending_rounds.cc:107] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 13:10:39.702741 25390 raft_consensus.cc:2889] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:10:39.675801 32270 consensus_meta.cc:287] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.662s	user 0.001s	sys 0.011s
I20260709 13:10:39.657836 31961 consensus_queue.cc:1059] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.023s
I20260709 13:10:39.443490 31898 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:39.679767 32266 consensus_meta.cc:287] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.552s	user 0.010s	sys 0.000s
I20260709 13:10:39.676115 32219 raft_consensus.cc:740] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:39.669569 31990 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.724843 32068 consensus_queue.cc:237] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.687683 32198 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.728564 32236 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:39.730170 32284 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 16 current term: 16 is higher.
I20260709 13:10:39.675810 32279 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:39.760355 32198 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:10:39.760459 32279 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.762701 32279 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:39.704249 32283 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:10:39.716895 32332 log.cc:927] Time spent T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:10:39.751379 25246 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:39.684671 31952 consensus_queue.cc:1048] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.070s
I20260709 13:10:39.785722 31386 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:39.786429 31386 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.758049 32006 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:39.666344 32246 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.791447 32246 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:39.686728 32107 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:39.792348 32333 ts_tablet_manager.cc:933] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:39.792673 25318 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:39.793901 31386 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:39.794175 25241 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:10:39.752228 32326 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
W20260709 13:10:39.764683 32331 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.097s	user 0.000s	sys 0.002s
I20260709 13:10:39.747685 32270 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:10:39.816841 25238 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.005s	sys 0.020s
W20260709 13:10:39.817500 25238 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.005s	sys 0.020s
I20260709 13:10:39.794649 32236 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:39.810505 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 491864us
I20260709 13:10:39.825106 25287 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:39.829958 25241 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
W20260709 13:10:39.822751 25209 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.008s	sys 0.011s
W20260709 13:10:39.831600 25209 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.009s	sys 0.011s
W20260709 13:10:39.832284 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 621781us
W20260709 13:10:39.844209 25286 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.008s	sys 0.012s
I20260709 13:10:39.845358 25240 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16. Ops: []
W20260709 13:10:39.846380 25286 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.009s	sys 0.013s
I20260709 13:10:39.834345 25209 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:39.828483 25287 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:39.875141 25209 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:39.876849 25244 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.668s	user 0.000s	sys 0.024s
W20260709 13:10:39.877390 25244 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.673s	user 0.000s	sys 0.024s
I20260709 13:10:39.843410 31961 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:39.896574 25209 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:39.898447 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 70103us
I20260709 13:10:39.876144 32270 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:10:39.911724 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:39.907836 25317 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.031s	sys 0.037s
W20260709 13:10:39.919760 25317 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.031s	sys 0.038s
W20260709 13:10:39.908404 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 250082us
I20260709 13:10:39.892473 32318 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:39.819546 32243 consensus_queue.cc:237] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.862036 32236 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:39.901743 25287 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:39.930240 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:39.964502 25209 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:39.966127 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 64481.1us
W20260709 13:10:39.930979 25386 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.771s	user 0.017s	sys 0.013s
W20260709 13:10:39.968006 25386 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.018s	sys 0.014s
I20260709 13:10:39.968107 25314 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260709 13:10:39.971297 25314 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 13:10:39.987051 25209 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:39.987650 25315 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:10:39.988509 25209 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:39.990861 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:39.990727 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:39.991341 25938 raft_consensus.cc:3060] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:39.898931 32341 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:39.910949 32270 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:40.016074 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2934) took 1028 ms. Trace:
I20260709 13:10:39.995272 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:40.006506 32327 log.cc:927] Time spent T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.726s	user 0.000s	sys 0.004s
I20260709 13:10:40.006888 25938 raft_consensus.cc:2468] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:40.010546 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:40.022310 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:40.023089 25313 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:40.011790 25936 raft_consensus.cc:1275] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:10:39.927232 31990 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:39.930878 32093 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.010s	sys 0.026s
I20260709 13:10:40.017606 25312 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 15.
I20260709 13:10:40.018462 25321 raft_consensus.cc:1240] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:39.862311 32219 consensus_queue.cc:260] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:40.016690 25318 rpcz_store.cc:276] 0709 13:10:38.987484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:38.993705 (+  6221us) service_pool.cc:224] Handling call
0709 13:10:40.015794 (+1022089us) spinlock_profiling.cc:243] Waited 798 ms on lock 0x7b54000f8f54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:40.015968 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1835449728}
I20260709 13:10:39.935056 32318 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:40.021046 25939 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:39.992017 31961 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:39.914963 32247 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.101s
I20260709 13:10:40.001639 32298 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:40.017436 32270 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:10:40.027127 32093 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.010s	sys 0.027s
I20260709 13:10:40.063670 31386 consensus_queue.cc:1048] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:10:40.026324 32339 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:10:40.033319 32318 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:10:40.066855 32114 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.687s	user 0.006s	sys 0.035s
W20260709 13:10:40.064266 32032 consensus_meta.cc:287] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.644s	user 0.010s	sys 0.000s
I20260709 13:10:40.042842 32219 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:40.042688 32039 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:40.037133 31990 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:40.062744 31898 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:40.135463 25319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2926) took 1149 ms. Trace:
I20260709 13:10:40.137643 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:40.338613 25943 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:40.043004 32342 consensus_queue.cc:1059] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 13:10:40.140415 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:40.338258 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:40.071784 32338 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:40.142055 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:40.403285 25942 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 15.
I20260709 13:10:40.037168 25387 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:40.166682 25936 raft_consensus.cc:1240] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:40.063733 32298 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:40.179152 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:40.175870 32114 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.007s	sys 0.036s
W20260709 13:10:40.455567 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 117235us
I20260709 13:10:40.389595 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 1236) took 1201 ms. Trace:
I20260709 13:10:40.400580 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32994 (request call id 2858) took 1298 ms. Trace:
I20260709 13:10:40.144028 32346 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:10:40.314244 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 151784us
I20260709 13:10:40.338258 25319 rpcz_store.cc:276] 0709 13:10:38.985606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:38.985735 (+   129us) service_pool.cc:224] Handling call
0709 13:10:38.986024 (+   289us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:38.995065 (+  9041us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:38.995100 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:38.995523 (+   423us) log.cc:844] Serialized 26 byte log entry
0709 13:10:39.036008 (+ 40485us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:39.036052 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:39.036122 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:40.005896 (+969774us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:40.085211 (+ 79315us) raft_consensus.cc:1685] finished
0709 13:10:40.085232 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:40.134463 (+ 49231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8683,"raft.queue_time_us":9396,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":5915,"spinlock_wait_cycles":31775616,"thread_start_us":22596,"threads_started":1,"wal-append.queue_time_us":232412}
I20260709 13:10:40.465560 25246 rpcz_store.cc:276] 0709 13:10:39.101767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:39.109438 (+  7671us) service_pool.cc:224] Handling call
0709 13:10:40.202169 (+1092731us) spinlock_profiling.cc:243] Waited 631 ms on lock 0x7b5400363b54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:40.400510 (+198341us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1452322176}
I20260709 13:10:40.417910 32120 raft_consensus.cc:3055] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:40.424474 32235 raft_consensus.cc:3055] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:40.494099 25393 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:10:40.431241 32133 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:40.527832 25387 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:40.435372 25241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32994 (request call id 2867) took 1274 ms. Trace:
I20260709 13:10:40.530715 25241 rpcz_store.cc:276] 0709 13:10:39.160597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:39.160720 (+   123us) service_pool.cc:224] Handling call
0709 13:10:40.435174 (+1274454us) spinlock_profiling.cc:243] Waited 632 ms on lock 0x7b54000a0354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:40.435303 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1453870336}
I20260709 13:10:40.401646 32039 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:40.449735 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2946) took 1287 ms. Trace:
I20260709 13:10:40.464521 25240 rpcz_store.cc:276] 0709 13:10:39.187745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:39.197468 (+  9723us) service_pool.cc:224] Handling call
0709 13:10:39.197851 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:39.845949 (+648098us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:40.203599 (+357650us) spinlock_profiling.cc:243] Waited 647 ms on lock 0x7b5400363154. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:40.389500 (+185901us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1488398208}
I20260709 13:10:40.543390 25321 rpcz_store.cc:276] 0709 13:10:39.162627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:39.162744 (+   117us) service_pool.cc:224] Handling call
0709 13:10:40.018213 (+855469us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:40.025249 (+  7036us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:40.449449 (+424200us) spinlock_profiling.cc:243] Waited 855 ms on lock 0x7b54000f8f50. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:40.449637 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1966710528}
I20260709 13:10:40.437603 32268 consensus_queue.cc:1059] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:40.182376 32032 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:40.460474 32202 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:40.565904 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:40.541791 31789 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:40.566835 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:40.491381 31850 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:40.491138 32175 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:40.496325 32120 raft_consensus.cc:740] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:40.530627 32235 raft_consensus.cc:740] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:10:40.577418 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 86064us
I20260709 13:10:40.586601 25318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:10:40.588038 25318 raft_consensus.cc:606] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:40.588656 25321 raft_consensus.cc:2368] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18.
W20260709 13:10:40.610340 25239 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.005s	sys 0.014s
W20260709 13:10:40.610728 25239 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.005s	sys 0.014s
W20260709 13:10:40.612087 25209 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.038s	sys 0.052s
W20260709 13:10:40.612418 25209 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.039s	sys 0.052s
W20260709 13:10:40.574538 32347 log.cc:927] Time spent T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.505s	user 0.000s	sys 0.001s
W20260709 13:10:40.582610 32348 log.cc:927] Time spent T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.444s	user 0.000s	sys 0.001s
I20260709 13:10:40.607074 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:10:40.612982 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 624503us
I20260709 13:10:40.621817 25390 raft_consensus.cc:2368] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 19. Current term is 20.
I20260709 13:10:40.622524 25318 raft_consensus.cc:614] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:10:40.629243 25245 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.634s	user 0.026s	sys 0.030s
W20260709 13:10:40.629726 25245 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.026s	sys 0.030s
W20260709 13:10:40.579232 25238 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.684s	user 0.024s	sys 0.066s
I20260709 13:10:40.578433 32224 raft_consensus.cc:3055] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 13:10:40.631749 25320 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.014s	sys 0.005s
W20260709 13:10:40.634701 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 54259.3us
W20260709 13:10:40.630810 25238 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.024s	sys 0.066s
W20260709 13:10:40.642410 25320 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.014s	sys 0.006s
W20260709 13:10:40.635623 25317 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.029s	sys 0.058s
W20260709 13:10:40.634986 32333 thread.cc:641] tablet-download-pool [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.023s	sys 0.033s
W20260709 13:10:40.632131 25314 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.004s	sys 0.015s
W20260709 13:10:40.632927 32243 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.008s	sys 0.011s
I20260709 13:10:40.565865 32039 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:40.645479 25314 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.005s	sys 0.015s
W20260709 13:10:40.645556 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:40.630641 32337 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.839s	user 0.013s	sys 0.043s
I20260709 13:10:40.580740 31789 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:40.582510 31850 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:40.667140 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:40.643810 25317 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.030s	sys 0.058s
W20260709 13:10:40.644805 32333 thread.cc:608] tablet-download-pool [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.024s	sys 0.033s
I20260709 13:10:40.575959 32202 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:40.583103 32175 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:40.637897 32224 raft_consensus.cc:740] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:10:40.650982 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:40.648296 32243 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.634s	user 0.009s	sys 0.011s
W20260709 13:10:40.704231 32337 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.873s	user 0.014s	sys 0.043s
I20260709 13:10:40.649024 32343 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:10:40.737406 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 308599us
I20260709 13:10:40.653486 32362 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:40.777558 32362 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:40.779831 32357 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
W20260709 13:10:40.781092 32354 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.253s	user 0.000s	sys 0.000s
I20260709 13:10:40.781286 32338 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:40.781814 32338 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:40.782362 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 1492) took 1628 ms. Trace:
I20260709 13:10:40.782740 25386 rpcz_store.cc:276] 0709 13:10:39.153467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:39.154203 (+   736us) service_pool.cc:224] Handling call
0709 13:10:39.154545 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:39.159388 (+  4843us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:39.159422 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:39.159905 (+   483us) log.cc:844] Serialized 26 byte log entry
0709 13:10:39.968481 (+808576us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:39.968541 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:39.968635 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:40.782147 (+813512us) raft_consensus.cc:1685] finished
0709 13:10:40.782170 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:40.782298 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4475,"raft.queue_time_us":5230,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":90,"spinlock_wait_cycles":3328,"thread_start_us":808148,"threads_started":1,"wal-append.queue_time_us":1016342}
I20260709 13:10:40.783970 32338 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:40.784708 25242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 1233) took 1600 ms. Trace:
I20260709 13:10:40.785944 32343 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:40.785094 25242 rpcz_store.cc:276] 0709 13:10:39.184192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:39.184302 (+   110us) service_pool.cc:224] Handling call
0709 13:10:39.184612 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:39.608430 (+423818us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:10:39.608478 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:39.609084 (+   606us) log.cc:844] Serialized 26 byte log entry
0709 13:10:40.028745 (+419661us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:40.039620 (+ 10875us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:40.039710 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:40.784479 (+744769us) raft_consensus.cc:1685] finished
0709 13:10:40.784503 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:40.784649 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22819,"raft.queue_time_us":72213,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":9833,"thread_start_us":842563,"threads_started":2,"wal-append.queue_time_us":969407}
W20260709 13:10:40.786283 32360 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 13:10:40.786407 32343 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:40.790995 32039 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:40.791632 32370 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:40.791942 32361 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.209s	user 0.000s	sys 0.001s
I20260709 13:10:40.792236 32370 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:40.793561 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:40.795181 25237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 1234) took 1609 ms. Trace:
I20260709 13:10:40.796427 32202 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:40.745077 32107 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:40.797250 32107 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:40.798363 32356 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:40.798938 32356 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:40.799434 32107 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:40.800034 32339 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:40.800266 32093 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:40.800573 32339 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:40.800762 32093 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:40.802692 32093 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:40.793562 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:40.795567 25237 rpcz_store.cc:276] 0709 13:10:39.185342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:39.185456 (+   114us) service_pool.cc:224] Handling call
0709 13:10:39.243811 (+ 58355us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:39.609516 (+365705us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:39.609561 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:39.610174 (+   613us) log.cc:844] Serialized 52 byte log entry
0709 13:10:40.082517 (+472343us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:40.143221 (+ 60704us) log.cc:844] Serialized 14 byte log entry
0709 13:10:40.710820 (+567599us) spinlock_profiling.cc:243] Waited 5.17 ms on lock 0x7b1800490780. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b3fc0a 00007f297d27f219 00007f2973bc7a40 00007f2973bc24e4 00007f2973bc2455 00007f2976caa060 00007f2976cbc716 00007f2976ccd287 00007f2976ccd208
0709 13:10:40.711255 (+   435us) log.cc:844] Serialized 14 byte log entry
0709 13:10:40.794787 (+ 83532us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:40.794858 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:40.794968 (+   110us) raft_consensus.cc:1685] finished
0709 13:10:40.794983 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:40.795122 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":107876,"prepare.queue_time_us":83837,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":20390,"raft.queue_time_us":1087616,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":20200,"spinlock_wait_cycles":58470656,"thread_start_us":762801,"threads_started":2,"wal-append.queue_time_us":938320}
I20260709 13:10:40.709054 32236 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:40.742302 32344 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.608s	user 0.000s	sys 0.001s
W20260709 13:10:40.776392 32368 log.cc:927] Time spent T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 13:10:40.776731 32266 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:40.777148 32349 log.cc:927] Time spent T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
I20260709 13:10:40.804490 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:40.804486 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:40.810154 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:40.847077 25240 raft_consensus.cc:1240] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:40.811833 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:40.852164 25358 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:40.774829 32335 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:40.802719 32339 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:40.834120 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 68341.6us
I20260709 13:10:40.850170 32032 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:40.870214 32006 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:40.870687 32006 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:40.871053 32006 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:40.872248 32340 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:40.872798 32340 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:40.873414 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 2477) took 1619 ms. Trace:
I20260709 13:10:40.873775 25391 rpcz_store.cc:276] 0709 13:10:39.254502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:39.254688 (+   186us) service_pool.cc:224] Handling call
0709 13:10:39.255124 (+   436us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:39.260430 (+  5306us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:39.260474 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:39.261074 (+   600us) log.cc:844] Serialized 26 byte log entry
0709 13:10:39.800523 (+539449us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:39.800582 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:39.800847 (+   265us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:40.873191 (+1072344us) raft_consensus.cc:1685] finished
0709 13:10:40.873213 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:40.873358 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6037,"raft.queue_time_us":5787,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":11155,"thread_start_us":488998,"threads_started":1,"wal-append.queue_time_us":756263}
I20260709 13:10:40.774071 32333 tablet_copy_client.cc:323] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Beginning tablet copy session from remote peer at address 127.23.140.67:34661
I20260709 13:10:40.878980 32382 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:40.881286 25358 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:40.883677 25358 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:40.887849 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:40.888815 25212 leader_election.cc:400] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 19. Current term is 20.
I20260709 13:10:40.889576 25212 leader_election.cc:403] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 13:10:40.825289 32216 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:40.892941 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:40.845291 32351 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:40.896247 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:40.898574 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 1240) took 1598 ms. Trace:
I20260709 13:10:40.850253 32247 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:40.899101 25317 rpcz_store.cc:276] 0709 13:10:39.300058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:39.300212 (+   154us) service_pool.cc:224] Handling call
0709 13:10:39.323132 (+ 22920us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:39.920621 (+597489us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:39.920670 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:39.986323 (+ 65653us) log.cc:844] Serialized 26 byte log entry
0709 13:10:40.708000 (+721677us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:40.708050 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:40.708140 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:40.898295 (+190155us) raft_consensus.cc:1685] finished
0709 13:10:40.898325 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:40.898501 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":93525,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":6367,"thread_start_us":1318287,"threads_started":2,"wal-append.queue_time_us":665005}
I20260709 13:10:40.918494 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:40.862705 31990 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:40.824954 32039 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.066s
I20260709 13:10:40.867724 32356 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:40.895220 31898 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:40.895270 32338 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 19. Current term is 20.
I20260709 13:10:40.896232 31967 raft_consensus.cc:2804] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:10:40.850800 32266 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:40.890440 32352 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:40.890297 32339 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:40.884796 32340 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:40.893366 32090 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:40.890633 32298 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:40.884610 32335 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:40.937031 31898 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:40.892123 32178 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:40.867486 32224 consensus_queue.cc:260] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:40.946861 32375 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 13:10:40.883224 32382 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:40.921679 32032 consensus_queue.cc:1059] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 13:10:40.919899 32351 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:40.917060 32216 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:40.973524 32318 consensus_meta.cc:287] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.790s	user 0.009s	sys 0.002s
I20260709 13:10:40.929172 31990 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:40.922084 32247 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:40.953194 32236 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:40.953466 31967 raft_consensus.cc:697] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:10:40.986454 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:40.962040 32219 consensus_meta.cc:287] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.652s	user 0.011s	sys 0.000s
I20260709 13:10:40.953869 32339 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:40.983862 25908 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:40.966245 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 84329.4us
I20260709 13:10:41.028561 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:40.934526 32114 consensus_queue.cc:1048] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
W20260709 13:10:40.962164 32297 consensus_meta.cc:287] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.838s	user 0.011s	sys 0.005s
W20260709 13:10:40.979416 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:40.967291 31898 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:40.998004 32224 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:40.945333 32352 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:41.000366 32274 consensus_meta.cc:287] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 1.458s	user 0.013s	sys 0.000s
I20260709 13:10:40.959568 32090 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:41.057015 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:41.058583 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 74834.4us
I20260709 13:10:41.120052 25908 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:41.077787 32371 log.cc:927] Time spent T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.266s	user 0.009s	sys 0.001s
I20260709 13:10:41.078454 32370 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.078759 32266 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.078890 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:41.098999 32297 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.079829 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.000545 32387 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:41.006727 32175 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:40.988813 32178 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.006723 32318 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.035225 32236 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.210024 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.177785 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 2482) took 1921 ms. Trace:
I20260709 13:10:41.222887 25387 rpcz_store.cc:276] 0709 13:10:39.256018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:39.256138 (+   120us) service_pool.cc:224] Handling call
0709 13:10:41.177581 (+1921443us) spinlock_profiling.cc:243] Waited 772 ms on lock 0x7b5400277ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:41.177713 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1775662464}
W20260709 13:10:41.174019 25395 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 2483) took 1917 ms (client timeout 1657 ms). Trace:
W20260709 13:10:41.224781 25395 rpcz_store.cc:269] 0709 13:10:39.256309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:39.258892 (+  2583us) service_pool.cc:224] Handling call
0709 13:10:41.173706 (+1914814us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b540013d8d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:41.173898 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3357555584}
I20260709 13:10:41.227874 25388 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
W20260709 13:10:41.124212 32373 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:10:41.171959 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 1242) took 1178 ms. Trace:
I20260709 13:10:41.131935 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.232100 25390 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:41.162098 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2886) took 1241 ms. Trace:
I20260709 13:10:41.239997 25388 raft_consensus.cc:1240] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260709 13:10:41.125247 32396 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:41.244481 25245 rpcz_store.cc:276] 0709 13:10:39.920874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:39.922152 (+  1278us) service_pool.cc:224] Handling call
0709 13:10:39.990878 (+ 68726us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:40.630157 (+639279us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:40.630203 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:40.640793 (+ 10590us) log.cc:844] Serialized 26 byte log entry
0709 13:10:40.826775 (+185982us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:40.826850 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:40.826981 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:41.161833 (+334852us) raft_consensus.cc:1685] finished
0709 13:10:41.161863 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:41.162024 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":373495,"raft.queue_time_us":176330,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":17064,"thread_start_us":749435,"threads_started":2,"wal-append.queue_time_us":332726}
I20260709 13:10:41.122326 25242 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
W20260709 13:10:41.123565 32366 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.155s	user 0.000s	sys 0.000s
W20260709 13:10:41.125708 32377 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
W20260709 13:10:41.126173 32383 log.cc:927] Time spent T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
I20260709 13:10:41.111791 32337 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:41.158691 32247 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.211099 32120 consensus_queue.cc:260] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:41.315189 32373 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 13:10:41.316037 32120 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:41.316308 32351 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.316812 25314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 1243) took 1322 ms. Trace:
I20260709 13:10:41.262580 25244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2892) took 1114 ms. Trace:
I20260709 13:10:41.263311 32367 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:41.085544 32235 consensus_queue.cc:260] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.265430 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:41.322791 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:41.323920 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:41.324568 25242 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:41.317266 25314 rpcz_store.cc:276] 0709 13:10:39.994241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:40.003879 (+  9638us) service_pool.cc:224] Handling call
0709 13:10:40.004228 (+   349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:40.022862 (+ 18634us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:10:40.023165 (+   303us) log.cc:844] Serialized 14 byte log entry
0709 13:10:40.421683 (+398518us) spinlock_profiling.cc:243] Waited 4.1 ms on lock 0x7b4800271ed8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297bbdbaa1 00007f297bbd2b37 00007f297bbd29ca 00007f297bcca26f 00007f297bcc9f37 00007f297bcde55c 00007f297bcde508 00007f297bcde488 00007f297bcde443 00007f297bcdd6ec 00007f297ac07a96
0709 13:10:40.707530 (+285847us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:40.723130 (+ 15600us) log.cc:844] Serialized 26 byte log entry
0709 13:10:40.743213 (+ 20083us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:40.743248 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:40.743470 (+   222us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:41.316517 (+573047us) raft_consensus.cc:1685] finished
0709 13:10:41.316620 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:41.316754 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57834,"prepare.queue_time_us":855,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":3607,"raft.queue_time_us":28740,"raft.run_cpu_time_us":2462,"raft.run_wall_time_us":404249,"spinlock_wait_cycles":9475456,"thread_start_us":679413,"threads_started":1,"wal-append.queue_time_us":938189}
I20260709 13:10:41.327443 32351 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:41.328176 32376 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 13:10:41.328007 32351 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.329545 25316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 2946) took 1333 ms. Trace:
I20260709 13:10:41.330344 32178 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.332149 32318 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.332227 32236 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.337942 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 2895) took 1179 ms. Trace:
I20260709 13:10:41.329898 25316 rpcz_store.cc:276] 0709 13:10:39.996500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:40.004791 (+  8291us) service_pool.cc:224] Handling call
0709 13:10:40.005113 (+   322us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:40.034249 (+ 29136us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:40.034295 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:40.089584 (+ 55289us) log.cc:844] Serialized 52 byte log entry
0709 13:10:40.436431 (+346847us) spinlock_profiling.cc:243] Waited 1.04 ms on lock 0x7b4800271ed8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297bbdbaa1 00007f297bbd2b37 00007f297bbd29ca 00007f297bcca26f 00007f297bcc9f37 00007f297bcde55c 00007f297bcde508 00007f297bcde488 00007f297bcde443 00007f297bcdd6ec 00007f297ac07a96
0709 13:10:40.667002 (+230571us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:40.667147 (+   145us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:41.028461 (+361314us) spinlock_profiling.cc:243] Waited 4.49 ms on lock 0x7b5400040bd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:41.028509 (+    48us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:41.329343 (+300834us) raft_consensus.cc:1685] finished
0709 13:10:41.329363 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:41.329489 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37523,"raft.queue_time_us":20964,"raft.run_cpu_time_us":10053,"raft.run_wall_time_us":392805,"spinlock_wait_cycles":12726912,"thread_start_us":336146,"threads_started":1,"wal-append.queue_time_us":943842}
I20260709 13:10:41.342070 25246 rpcz_store.cc:276] 0709 13:10:40.158787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:40.569711 (+410924us) service_pool.cc:224] Handling call
0709 13:10:40.583931 (+ 14220us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:40.789498 (+205567us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:40.789532 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:40.790027 (+   495us) log.cc:844] Serialized 26 byte log entry
0709 13:10:40.940787 (+150760us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:40.940895 (+   108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:40.941039 (+   144us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:41.337702 (+396663us) raft_consensus.cc:1685] finished
0709 13:10:41.337722 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:41.337874 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":205108,"raft.queue_time_us":206313,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":101,"thread_start_us":150243,"threads_started":1,"wal-append.queue_time_us":459265}
I20260709 13:10:41.242612 32387 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.319734 32337 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.345600 32387 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.345664 32337 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.269774 32396 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.348654 32396 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.132797 32274 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.349735 32297 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.350968 32274 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.333058 32235 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:41.355207 31789 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.356388 25392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 2495) took 1175 ms. Trace:
I20260709 13:10:41.356668 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2872) took 2087 ms. Trace:
I20260709 13:10:41.357095 25238 rpcz_store.cc:276] 0709 13:10:39.268790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:39.270831 (+  2041us) service_pool.cc:224] Handling call
0709 13:10:39.271201 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:39.817980 (+546779us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:10:39.818020 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:39.818788 (+   768us) log.cc:844] Serialized 26 byte log entry
0709 13:10:40.642832 (+824044us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:40.642971 (+   139us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:40.643198 (+   227us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:41.356471 (+713273us) raft_consensus.cc:1685] finished
0709 13:10:41.356492 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:41.356610 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27151,"raft.queue_time_us":486264,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":80,"thread_start_us":1266506,"threads_started":2,"wal-append.queue_time_us":1059471}
I20260709 13:10:41.356914 25392 rpcz_store.cc:276] 0709 13:10:40.180982 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:40.181434 (+   452us) service_pool.cc:224] Handling call
0709 13:10:40.404035 (+222601us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:40.404440 (+   405us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:40.404479 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:40.404994 (+   515us) log.cc:844] Serialized 26 byte log entry
0709 13:10:40.709878 (+304884us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:10:40.709919 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:40.709981 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:41.356096 (+646115us) raft_consensus.cc:1685] finished
0709 13:10:41.356135 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:41.356316 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44359,"raft.queue_time_us":6391,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":4976,"thread_start_us":304484,"threads_started":1,"wal-append.queue_time_us":536810}
I20260709 13:10:41.359922 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.360641 25392 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:41.361997 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.265681 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:41.265798 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 145853us
I20260709 13:10:41.266079 32405 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:41.117100 32359 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:41.393206 32396 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:41.394093 32405 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
W20260709 13:10:41.394259 32343 consensus_meta.cc:287] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.607s	user 0.010s	sys 0.000s
I20260709 13:10:41.160166 31990 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:41.336309 32270 consensus_meta.cc:287] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 1.267s	user 0.005s	sys 0.005s
I20260709 13:10:41.394577 32405 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:41.395293 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:41.395172 32270 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.393923 32396 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.398226 32351 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.398429 25240 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:41.398442 32396 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:41.125063 32364 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.192s	user 0.000s	sys 0.001s
I20260709 13:10:41.163540 32382 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:41.179816 32365 log.cc:927] Time spent T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.282s	user 0.000s	sys 0.001s
I20260709 13:10:41.280514 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.131186 32339 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.125396 32244 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:41.197034 32358 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:41.195537 32298 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:41.178807 31967 consensus_queue.cc:237] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.255934 32175 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:10:41.370038 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 489815us
W20260709 13:10:41.404338 25909 connection.cc:256] RPC call timeout handler was delayed by -1.0627734419999797s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1937s for a response
I20260709 13:10:41.370904 25943 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:10:41.374821 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:41.426719 25939 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:41.376044 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:41.262334 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:41.264771 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.333504 32335 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.393869 32359 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.242815 25315 rpcz_store.cc:276] 0709 13:10:39.993923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:39.995503 (+  1580us) service_pool.cc:224] Handling call
0709 13:10:39.995817 (+   314us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:40.033697 (+ 37880us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:40.033728 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:40.043360 (+  9632us) log.cc:844] Serialized 26 byte log entry
0709 13:10:40.637965 (+594605us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:40.638010 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:41.171411 (+533401us) spinlock_profiling.cc:243] Waited 23.9 ms on lock 0x7b5400276354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:41.171479 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:41.171545 (+    66us) raft_consensus.cc:1685] finished
0709 13:10:41.171565 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:41.171876 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20476,"raft.queue_time_us":21726,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":20530,"spinlock_wait_cycles":54880000,"thread_start_us":472483,"threads_started":1,"wal-append.queue_time_us":655621}
I20260709 13:10:41.430205 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:41.423628 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.432440 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:10:41.408007 32406 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
I20260709 13:10:41.428103 25388 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 18
I20260709 13:10:41.430979 25938 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:41.428094 25942 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:41.407256 32405 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.404675 32396 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:41.405329 32382 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:41.439828 32297 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 13:10:41.440220 32407 log.cc:927] Time spent T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.171s	user 0.000s	sys 0.001s
I20260709 13:10:41.403708 32362 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.412294 32216 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:41.412003 32404 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:41.431835 32175 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:41.450459 32270 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.440502 32405 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:10:41.394742 32343 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.410511 32244 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.426806 32298 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.326887 32367 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.459692 32175 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:41.427099 32358 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.450572 32396 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.461205 32083 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:41.462698 32352 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:41.465363 32401 log.cc:927] Time spent T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 13:10:41.453063 32382 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.469316 32083 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:41.470850 32006 consensus_meta.cc:287] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.599s	user 0.007s	sys 0.003s
I20260709 13:10:41.471241 32396 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.471248 32006 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.471872 32083 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.473100 32367 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.475875 32359 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.476176 32343 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.476271 32298 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.464426 32404 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.478332 32382 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.479125 32358 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:41.480970 32404 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.270123 25237 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 18. Current term is 19. Ops: []
W20260709 13:10:41.448230 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 127680us
I20260709 13:10:41.401257 25242 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 13:10:41.448561 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:41.518463 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 69794.7us
I20260709 13:10:41.519578 25359 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:41.520164 25358 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:41.317883 25244 rpcz_store.cc:276] 0709 13:10:40.148492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:40.340495 (+192003us) service_pool.cc:224] Handling call
0709 13:10:40.340914 (+   419us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:40.928348 (+587434us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:40.928382 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:40.928869 (+   487us) log.cc:844] Serialized 26 byte log entry
0709 13:10:41.248662 (+319793us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:41.248724 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:41.248789 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:41.262372 (+ 13583us) raft_consensus.cc:1685] finished
0709 13:10:41.262396 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:41.262530 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":433070,"raft.queue_time_us":438056,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"thread_start_us":435488,"threads_started":2,"wal-append.queue_time_us":331711}
I20260709 13:10:41.412967 25245 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 17: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:41.512962 25212 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:41.401089 25240 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
W20260709 13:10:41.526156 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 100226us
W20260709 13:10:41.461344 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:41.534178 25936 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 13:10:41.533957 25387 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:41.536015 25945 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:10:41.538658 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:41.540232 25943 raft_consensus.cc:2410] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 22 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 13:10:41.541179 32270 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:41.542168 32270 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.545385 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:41.547376 32270 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.550484 31967 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:41.550638 32270 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:41.545578 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:41.551126 31967 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:41.551807 31967 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:41.552037 25939 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:41.552894 32006 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:41.553673 32297 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:41.554280 32297 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:41.554364 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.554728 32297 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:41.555763 25386 raft_consensus.cc:2393] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 21: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:41.555877 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:41.551067 31789 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:41.556851 25943 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:41.563997 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:41.570354 32244 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:41.574365 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 2948) took 1009 ms. Trace:
I20260709 13:10:41.574954 25321 rpcz_store.cc:276] 0709 13:10:40.565120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:40.566517 (+  1397us) service_pool.cc:224] Handling call
0709 13:10:41.573702 (+1007185us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b54002f35d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:41.574052 (+   350us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29872512}
W20260709 13:10:41.578605 32413 log.cc:927] Time spent T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 13:10:41.580217 32367 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:41.581598 32410 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 0.107s	user 0.000s	sys 0.002s
I20260709 13:10:41.581311 32367 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.583756 32338 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:41.584335 32235 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:41.584656 32338 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.584527 32367 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.551258 32270 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:41.585291 32235 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.598591 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:41.599785 25245 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
W20260709 13:10:41.600767 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:41.601439 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:41.601557 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:41.601763 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:41.602612 32343 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:41.594179 32235 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.560148 32335 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 13:10:41.614745 32338 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.618158 32133 raft_consensus.cc:3055] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:41.623586 32090 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:41.629563 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 545682us
W20260709 13:10:41.630930 25287 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 1.657s (SENT) [suppressed 7 similar messages]
W20260709 13:10:41.631194 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 108488us
I20260709 13:10:41.636358 31386 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:41.639215 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.639571 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.641228 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.641304 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:41.588559 32270 raft_consensus.cc:3060] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:41.659905 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.660434 25909 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:41.661423 25237 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 22.
I20260709 13:10:41.662057 32387 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:41.662408 32236 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:41.663328 32236 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:41.663959 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:41.664259 25909 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:41.664204 31386 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.658934 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:41.636615 32175 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.665342 32387 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:41.648573 32133 raft_consensus.cc:740] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:41.672150 32343 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.673228 31850 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:41.672940 25386 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:41.676263 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.698987 25393 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:41.676360 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:41.672961 32114 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:41.701294 32107 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:41.702013 32114 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.702383 32107 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.677382 25390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 1498) took 1027 ms. Trace:
I20260709 13:10:41.677475 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:41.706331 25243 raft_consensus.cc:2393] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 22: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:41.692142 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:41.693092 25405 tablet_copy_service.cc:140] P a2a77ecf2c0941618b5bc79d4c04db61: Received BeginTabletCopySession request for tablet b62f4877af9742298b1e149789ca3090 from peer f646dfd532974dceadfe00005d2bbf59 ({username='slave'} at 127.0.0.1:41830)
I20260709 13:10:41.693475 32031 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:41.695907 31967 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.696461 25359 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:41.693143 25238 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 21.
I20260709 13:10:41.705819 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.732847 32178 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:41.740914 32175 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:41.705561 25390 rpcz_store.cc:276] 0709 13:10:40.650295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:40.650498 (+   203us) service_pool.cc:224] Handling call
0709 13:10:40.653160 (+  2662us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:41.243130 (+589970us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:41.676822 (+433692us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x7b54001e4954. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:41.677160 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1331007232}
W20260709 13:10:41.740283 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 71139.4us
I20260709 13:10:41.740443 32031 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.741405 32178 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.751319 32178 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.752089 32178 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:41.752514 32114 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:41.701819 32133 consensus_queue.cc:260] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:10:41.754724 32133 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:41.755100 32243 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:41.755633 32243 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:41.756054 32243 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:41.757373 32120 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:41.758213 32120 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.762953 32247 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:41.763794 32247 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.766420 32219 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:41.767198 32219 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.769459 32119 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:41.770390 32119 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.743798 32343 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.776290 32244 consensus_queue.cc:1059] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:10:41.752769 32178 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.777604 32387 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:41.778523 32387 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.779577 32178 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.753597 32031 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.779886 32359 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.009s
I20260709 13:10:41.760985 32120 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:41.789578 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:41.790349 31808 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 13:10:41.794519 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:41.790915 32343 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:41.791319 32358 consensus_queue.cc:1059] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:41.792127 31952 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:41.792512 31967 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.792148 32219 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.796243 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.751691 25284 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:41.803944 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 55328us
I20260709 13:10:41.808439 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:41.809852 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 145864us
I20260709 13:10:41.809084 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:41.807283 25405 tablet_copy_service.cc:161] P a2a77ecf2c0941618b5bc79d4c04db61: Beginning new tablet copy session on tablet b62f4877af9742298b1e149789ca3090 from peer f646dfd532974dceadfe00005d2bbf59 at {username='slave'} at 127.0.0.1:41830: session id = f646dfd532974dceadfe00005d2bbf59-b62f4877af9742298b1e149789ca3090
I20260709 13:10:41.782338 32119 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.803603 32335 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:41.827278 32338 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:41.836704 32335 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.836759 32338 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:10:41.837401 32363 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:41.838009 32363 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:41.838502 32363 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:41.839183 32335 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.839181 32338 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.847755 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.849448 31952 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:41.852339 31952 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.854369 32247 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.841383 31386 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.840991 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:41.795864 32343 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:41.864174 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 56739.1us
I20260709 13:10:41.864880 32283 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:41.867007 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.874677 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.876267 25237 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:41.877377 25388 raft_consensus.cc:2468] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
W20260709 13:10:41.878268 32355 log.cc:927] Time spent T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 1.026s	user 0.005s	sys 0.006s
I20260709 13:10:41.878264 25911 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:41.881628 32382 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:41.882131 32382 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:41.882535 31898 consensus_meta.cc:287] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.764s	user 0.011s	sys 0.010s
I20260709 13:10:41.882341 32243 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.882891 32412 consensus_queue.cc:1059] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:10:41.883095 31898 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.884135 32412 raft_consensus.cc:3055] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:41.884624 32412 raft_consensus.cc:740] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:41.884786 25394 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 20: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:41.882565 32382 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:41.886775 32343 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.877408 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:41.894022 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:41.895213 25241 raft_consensus.cc:2393] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 19: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:41.894634 32412 consensus_queue.cc:260] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.896428 32412 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:41.896520 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:41.900022 32244 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:41.906451 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:41.866669 32283 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:41.927089 25241 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:41.937480 32244 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:41.939764 32114 raft_consensus.cc:3055] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:41.939932 32243 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.940166 32244 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.940316 32114 raft_consensus.cc:740] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:41.923739 32283 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:41.943962 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:41.933578 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 302823us
I20260709 13:10:41.935247 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.923791 32359 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:41.953189 25909 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:41.923691 32107 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.961205 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:41.979209 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:41.922722 32236 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:41.961652 31898 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:41.988068 25239 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:10:41.994094 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 242818us
I20260709 13:10:41.972704 32270 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:42.013669 25318 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 13:10:42.013521 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.014389 25321 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:42.014688 32408 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:10:42.015688 25212 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:42.016994 32297 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:42.019207 32297 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:42.019343 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:42.020545 32343 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:42.021109 32343 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:42.029261 32343 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:42.029418 32387 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:42.031919 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:42.025214 25240 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 18: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:42.020612 25246 raft_consensus.cc:2393] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 18: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:10:42.042439 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 94679.9us
I20260709 13:10:42.032660 25238 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:42.027225 25359 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:42.023208 31386 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:42.054442 25391 raft_consensus.cc:2393] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 19: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:42.051599 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:42.012643 32363 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:42.064656 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:42.062665 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:42.068218 25246 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:42.025502 32247 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:42.065888 25284 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:42.067487 25909 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:41.997506 32359 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:42.075619 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:42.081171 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:42.083709 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:42.018555 25242 raft_consensus.cc:2955] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Committing config change with OpId 16.11: config changed from index -1 to 11, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:10:42.089157 25238 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:42.086555 25943 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:42.025519 32114 consensus_queue.cc:260] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:42.077200 32270 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:42.057442 32284 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.163s
I20260709 13:10:42.114142 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:42.114354 31386 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:42.117625 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:42.118692 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:42.148898 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:42.149202 25944 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:42.149703 25319 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:42.149803 32359 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:42.107674 32247 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:42.152746 25942 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:42.125548 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 98462.5us
I20260709 13:10:42.153476 32247 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:42.153689 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.154980 25359 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:42.155308 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.156924 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.157393 25395 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:42.157763 32284 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:42.158142 32284 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:42.158512 32284 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:10:42.159789 32419 log.cc:927] Time spent T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 13:10:42.162210 32363 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:42.162957 31386 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:42.125730 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:42.126184 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:42.163882 25392 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:42.164217 25246 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 19: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:10:42.126645 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 69304.2us
I20260709 13:10:42.165217 25909 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:42.155977 25359 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:42.166261 25909 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:42.167163 32359 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:42.137663 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:42.141449 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 135203us
I20260709 13:10:42.165208 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:42.127067 32032 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:42.121203 32411 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:42.106971 32367 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:42.142871 25405 tablet_copy_source_session.cc:215] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:10:42.143920 25285 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:42.120051 32405 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:42.146924 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:42.147712 32298 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:42.147898 25314 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 13:10:42.148030 25318 raft_consensus.cc:1275] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:10:42.178735 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.183332 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:42.188395 25246 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:42.167595 32359 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:42.188776 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.194089 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:42.195847 25317 raft_consensus.cc:3060] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:42.198051 25939 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:42.189779 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.200997 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:42.173555 32032 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:42.201812 32032 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:42.201967 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:42.202351 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:42.202721 25241 raft_consensus.cc:3060] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:42.198652 32114 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:42.206811 32404 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:10:42.207243 32404 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:42.207587 32404 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:42.210353 25390 raft_consensus.cc:1240] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260709 13:10:42.200083 25239 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
I20260709 13:10:42.195196 25320 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:42.199357 32359 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:42.176556 32411 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:42.178841 32298 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:42.216873 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:42.218200 25313 raft_consensus.cc:2393] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 20: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:42.220589 25241 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:42.224439 25241 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:42.227854 32343 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:42.240294 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:42.240602 25395 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 18: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:10:42.241351 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 97620.9us
I20260709 13:10:42.245296 25942 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:42.251072 25943 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:42.244076 25944 raft_consensus.cc:2468] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:42.252538 25212 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:42.213501 32387 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:10:42.224450 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.256359 25212 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:42.256650 25243 raft_consensus.cc:2410] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
W20260709 13:10:42.259290 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 46614.9us
I20260709 13:10:42.259688 31386 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:42.260100 31386 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:42.260414 31386 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:42.263737 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.270128 25939 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:10:42.270715 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.271037 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.271200 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.271287 32382 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:42.271909 25939 raft_consensus.cc:2393] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 20: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:42.272192 25945 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:42.270730 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:42.273662 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:42.233767 32298 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:42.258510 32387 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:42.272197 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:42.276505 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:42.221274 32411 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:42.276959 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.302161 25319 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
W20260709 13:10:42.277843 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:42.524115 25940 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 20.
W20260709 13:10:42.316679 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:42.516294 25246 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:42.348925 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.528643 25313 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 21.
W20260709 13:10:42.464510 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:42.530400 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:42.530900 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:42.527530 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:42.522555 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.370179 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.407646 32333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b62f4877af9742298b1e149789ca3090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:10:42.464943 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.382544 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:42.537994 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:42.538610 25317 raft_consensus.cc:2468] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
W20260709 13:10:42.522018 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 225784us
I20260709 13:10:42.370647 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.261428 32235 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:42.539757 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:42.414791 32389 log.cc:927] Time spent T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.214s	user 0.004s	sys 0.000s
I20260709 13:10:42.544911 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:42.549340 25939 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:42.550585 25944 raft_consensus.cc:2368] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 19. Current term is 20.
I20260709 13:10:42.525542 32387 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:42.358328 32243 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:42.405568 32133 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:42.406248 32380 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:42.557642 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.257488 32367 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:42.564580 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.566056 25942 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 13:10:42.386188 32284 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:42.567925 25312 raft_consensus.cc:2393] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 22: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:42.544998 32235 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:42.571440 25319 raft_consensus.cc:2393] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 21: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:42.574838 32299 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:42.575428 32032 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:42.577442 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.581357 25936 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:10:42.349862 32283 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:42.585870 32437 log.cc:927] Time spent T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 13:10:42.590402 31386 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:42.593557 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.597249 25316 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 17. Ops: []
I20260709 13:10:42.600080 25246 raft_consensus.cc:1240] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:42.602473 25246 raft_consensus.cc:1275] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:10:42.604321 25246 raft_consensus.cc:1240] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:42.589634 32343 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:42.606648 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.607990 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.609589 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.610263 25321 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 20.
I20260709 13:10:42.613224 25939 raft_consensus.cc:2468] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:42.614344 32404 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:42.613238 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.615942 25321 raft_consensus.cc:2393] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 17: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:42.616693 32404 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:42.617812 32268 raft_consensus.cc:3055] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:42.618228 32268 raft_consensus.cc:740] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:42.619119 32268 consensus_queue.cc:260] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:42.620169 32268 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:42.612334 32032 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:42.623394 25945 raft_consensus.cc:2368] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 22. Current term is 23.
I20260709 13:10:42.617821 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.628731 25321 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 16: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:42.626657 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:42.629307 25393 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:42.627099 25392 raft_consensus.cc:2393] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 20: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:42.572738 32382 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:42.625264 25318 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 18: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:42.632560 31386 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:42.631081 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.632458 25387 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 20.
I20260709 13:10:42.591696 32235 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:42.635460 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.636175 25940 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:42.632035 32299 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:42.638535 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:42.639079 32299 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:42.639930 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:42.643236 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:42.643803 25319 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:42.645216 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.646210 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.641623 32133 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:42.653204 32133 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:42.653100 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:42.654645 25317 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:42.624869 32380 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:42.658413 32380 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:42.658365 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:42.645681 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 48227.1us
I20260709 13:10:42.667677 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:42.598577 32243 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:42.662515 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:42.677901 25321 raft_consensus.cc:2393] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 18: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:42.679937 25313 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:42.682992 25312 raft_consensus.cc:2410] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 19 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 14.
W20260709 13:10:42.683368 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:42.681943 25314 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 22.
I20260709 13:10:42.663689 32283 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:42.687300 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.718541 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:42.733057 25285 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:10:42.744416 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:42.747242 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.753684 25936 raft_consensus.cc:2393] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 20: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:42.756134 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:42.760526 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:42.758838 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:42.749529 32283 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:42.751561 32243 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:42.794163 25358 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:42.795035 25319 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 20.
I20260709 13:10:42.795267 32114 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 13:10:42.808514 32447 log.cc:927] Time spent T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
W20260709 13:10:42.809593 32359 consensus_meta.cc:287] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.581s	user 0.010s	sys 0.000s
I20260709 13:10:42.810039 32359 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:42.802592 25358 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:42.800571 32367 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:42.815752 25359 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:42.812283 32284 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:42.842149 32434 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20260709 13:10:42.822034 32299 raft_consensus.cc:2804] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:10:42.854512 31386 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:42.855243 31386 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:42.855665 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:42.819962 32114 raft_consensus.cc:697] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:42.856221 25246 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:42.835381 32270 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:42.835161 32133 raft_consensus.cc:2804] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:42.818912 32283 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:42.865895 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:42.871660 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:42.880728 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.880923 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.881484 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:42.882326 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.881654 25937 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:42.868125 32243 raft_consensus.cc:2804] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:10:42.867995 32274 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:42.878583 32133 raft_consensus.cc:697] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:42.863770 32297 consensus_queue.cc:1059] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.030s
I20260709 13:10:42.916559 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:42.910872 32408 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.698s	user 0.016s	sys 0.037s
W20260709 13:10:42.919021 32408 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.017s	sys 0.037s
I20260709 13:10:42.922384 32283 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:42.925697 32446 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 13:10:42.926478 32359 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:42.873310 32343 consensus_queue.cc:1059] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.036s
I20260709 13:10:42.903038 32274 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:42.917419 32243 raft_consensus.cc:697] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:42.955385 32243 consensus_queue.cc:237] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:42.944197 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.962232 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.963411 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.964146 25938 raft_consensus.cc:2393] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 19: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:42.944079 32274 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:42.966941 32270 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:42.967473 32270 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:42.969738 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:42.970070 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.970664 25937 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:42.971565 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:42.972204 25937 raft_consensus.cc:2393] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 16: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:42.974076 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:42.974627 25943 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:42.947733 32441 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.195s	user 0.000s	sys 0.001s
I20260709 13:10:42.948115 32039 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.071s
I20260709 13:10:42.977072 32114 consensus_queue.cc:237] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:42.948889 25942 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 19.
I20260709 13:10:42.977906 25945 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: []
W20260709 13:10:42.949823 32444 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 13:10:42.928552 32358 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:42.979926 32358 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:42.953779 32298 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:42.972848 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:42.964921 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:43.003942 32387 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.004312 32235 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:43.005508 25940 raft_consensus.cc:2393] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 23: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:43.006932 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.007064 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.007741 25316 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
I20260709 13:10:43.008231 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.009450 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:43.009778 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.009989 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:43.010354 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.010020 25313 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:43.011531 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.011616 32358 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:43.015472 25246 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:43.018517 25393 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:43.020179 25911 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:43.021052 25941 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:43.022516 25938 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 16: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:43.018707 25313 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:43.026232 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:43.027103 25317 raft_consensus.cc:2393] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 20: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:43.039786 32439 raft_consensus.cc:2804] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:43.040455 32439 raft_consensus.cc:697] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 19 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:43.041602 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41830 (request call id 1505) took 1245 ms. Trace:
I20260709 13:10:43.041985 25387 rpcz_store.cc:276] 0709 13:10:41.796218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:41.874287 (+ 78069us) service_pool.cc:224] Handling call
0709 13:10:43.041421 (+1167134us) spinlock_profiling.cc:243] Waited 757 ms on lock 0x7b54003241d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:43.041523 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1740487936}
I20260709 13:10:43.044494 32451 raft_consensus.cc:3055] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:43.045856 25911 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:43.045989 32439 consensus_queue.cc:237] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.046854 31850 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:43.047258 31850 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:43.047586 31850 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:43.047629 31386 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:43.044916 32451 raft_consensus.cc:740] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:43.050724 32451 consensus_queue.cc:260] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:43.052161 32452 log.cc:927] Time spent T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 13:10:43.052402 32405 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:43.052975 32451 raft_consensus.cc:3060] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:43.053027 32405 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.054214 32387 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:43.055572 32405 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:43.071108 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 232355us
I20260709 13:10:43.071997 25359 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:43.081738 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:43.081838 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 2924) took 1089 ms. Trace:
W20260709 13:10:43.083190 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:43.082286 25243 rpcz_store.cc:276] 0709 13:10:41.992582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:42.060812 (+ 68230us) service_pool.cc:224] Handling call
0709 13:10:43.080719 (+1019907us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b54003a9dd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:43.081779 (+  1060us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":391326080}
W20260709 13:10:43.085150 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 108863us
W20260709 13:10:43.086774 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 224899us
W20260709 13:10:43.087597 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 209773us
I20260709 13:10:43.087765 25943 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:43.081662 32268 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 13:10:43.097772 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 186510us
I20260709 13:10:43.099455 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:43.079988 32382 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:43.102965 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:43.104435 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:43.102170 25244 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:43.057552 32133 consensus_queue.cc:237] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.117625 25240 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:43.118666 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:43.119406 25943 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:43.090058 32299 raft_consensus.cc:697] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:43.125873 32462 consensus_queue.cc:1048] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:10:43.129696 25284 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:43.132737 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
W20260709 13:10:43.133235 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:43.133512 25314 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 16: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:43.134492 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.135121 25317 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 20.
I20260709 13:10:43.135349 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2900) took 1115 ms. Trace:
I20260709 13:10:43.135677 25238 rpcz_store.cc:276] 0709 13:10:42.019634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:42.404466 (+384832us) service_pool.cc:224] Handling call
0709 13:10:42.528806 (+124340us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:42.538390 (+  9584us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:42.538420 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:42.570874 (+ 32454us) log.cc:844] Serialized 26 byte log entry
0709 13:10:42.931249 (+360375us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:10:42.931313 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:42.931405 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:43.135157 (+203752us) raft_consensus.cc:1685] finished
0709 13:10:43.135176 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:43.135298 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14852,"raft.queue_time_us":2945,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":8670,"spinlock_wait_cycles":45952,"thread_start_us":288199,"threads_started":1,"wal-append.queue_time_us":403802}
I20260709 13:10:43.139315 32274 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.142709 32299 consensus_queue.cc:237] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:43.146562 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:43.124179 25285 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:43.149591 25393 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:43.155136 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:43.156379 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:43.157497 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:43.161445 32298 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:43.162488 32411 consensus_meta.cc:287] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.610s	user 0.012s	sys 0.000s
I20260709 13:10:43.163142 32235 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:43.162900 32411 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:43.165275 32454 log.cc:927] Time spent T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.194s	user 0.002s	sys 0.000s
I20260709 13:10:43.166287 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:43.111449 32268 raft_consensus.cc:697] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:43.168553 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:43.173264 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 49664.6us
I20260709 13:10:43.173275 31386 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:43.174453 31386 raft_consensus.cc:697] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:43.177222 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:43.177785 25238 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:43.181402 25284 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:43.181334 31386 consensus_queue.cc:237] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.111011 32382 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.171492 32411 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:43.167860 32268 consensus_queue.cc:237] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.175920 32405 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:43.195914 32405 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.198418 32405 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:43.200636 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:43.181690 32363 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:43.201327 25395 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:43.189289 25285 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:43.193230 32247 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:10:43.205497 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:43.207420 25284 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:43.208065 25392 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 21.
I20260709 13:10:43.206560 32274 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:43.209221 32274 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:10:43.210297 32408 raft_consensus.cc:3055] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:43.210953 32408 raft_consensus.cc:740] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:43.211972 32408 consensus_queue.cc:260] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:43.204591 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:43.213382 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:43.213591 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:43.210364 32382 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:43.215659 25284 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:43.215765 25911 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:43.221486 32382 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:10:43.221967 32382 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:43.222309 32382 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:43.227748 32032 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:43.229282 32270 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:10:43.232316 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:43.233171 25390 raft_consensus.cc:2410] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 13:10:43.216763 32408 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:10:43.219031 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:43.233043 25284 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:43.217227 32380 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:43.239960 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 53010.2us
I20260709 13:10:43.235677 32032 consensus_queue.cc:1059] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.026s
I20260709 13:10:43.265316 32353 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:43.266117 32380 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:43.268296 32380 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:43.271852 31850 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.275547 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:43.276156 25390 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:43.282754 32270 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:43.299057 25390 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 21.
I20260709 13:10:43.302244 25238 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:43.305956 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:43.306543 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:43.310266 25943 raft_consensus.cc:2393] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 21: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:43.252216 32247 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:43.319386 25284 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:43.319795 25911 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:43.327639 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:43.328394 25936 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:10:43.326906 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 172062us
W20260709 13:10:43.306506 25287 connection.cc:256] RPC call timeout handler was delayed by -1.4008214899999984s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1819s for a response
I20260709 13:10:43.304280 31850 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:43.336575 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:43.337381 25393 raft_consensus.cc:2368] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19.
I20260709 13:10:43.324149 32247 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:43.347357 25390 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: []
W20260709 13:10:43.348634 25287 connection.cc:256] RPC call timeout handler was delayed by -1.0720671389999552s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3194s for a response
I20260709 13:10:43.346271 25209 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:43.351753 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.352375 25317 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 22.
I20260709 13:10:43.352922 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.353708 25313 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:43.327244 32235 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:43.359746 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.361299 25318 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:43.340796 32387 raft_consensus.cc:2804] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:10:43.362772 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:43.366163 25319 raft_consensus.cc:3060] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:43.366833 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:43.364811 25943 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:43.366828 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:43.362877 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:43.368700 25317 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 16: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:43.368402 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.370363 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:43.365936 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 78979.2us
I20260709 13:10:43.373317 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.351060 32247 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:43.323697 32270 consensus_queue.cc:1059] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.028s
I20260709 13:10:43.379940 25316 raft_consensus.cc:2393] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 16: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:43.376261 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:43.371420 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.376646 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.374467 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:43.373924 25314 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 22.
I20260709 13:10:43.382169 25317 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:43.385675 32460 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:43.386193 32270 raft_consensus.cc:3055] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:43.386571 32270 raft_consensus.cc:740] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:43.387463 32270 consensus_queue.cc:260] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.388988 32270 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:10:43.390545 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:43.391664 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:43.392330 25390 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:43.393515 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:43.394307 25390 raft_consensus.cc:2393] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 19: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:43.394550 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:43.397444 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:43.400902 25313 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 20.
W20260709 13:10:43.401930 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 71527us
I20260709 13:10:43.403483 25209 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:43.357343 32178 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:43.405422 32178 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.406287 32451 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:43.407282 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:43.414225 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:43.418376 25359 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:43.381080 25315 raft_consensus.cc:2468] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:43.439368 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:43.373749 32343 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:43.378716 32387 raft_consensus.cc:697] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 21 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:43.450584 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.454674 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.458093 25319 raft_consensus.cc:2468] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:43.457019 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.459626 25316 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 22.
I20260709 13:10:43.446239 32299 raft_consensus.cc:2804] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:43.458684 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:43.431643 32298 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.022s
I20260709 13:10:43.433995 32032 consensus_queue.cc:1059] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 13:10:43.464169 25317 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 23.
I20260709 13:10:43.465617 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:43.430336 32235 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.433269 32382 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.475801 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.469964 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:43.453176 32178 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:43.472774 25320 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 16: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:10:43.473201 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 69828.7us
I20260709 13:10:43.487946 25209 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:43.472576 32299 raft_consensus.cc:697] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:43.489387 32343 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:43.490073 32299 consensus_queue.cc:237] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:43.494405 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:43.495702 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:43.496349 25320 raft_consensus.cc:3060] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:43.499574 25209 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:43.501070 32485 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:43.501860 32485 consensus_queue.cc:1059] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:10:43.502929 32451 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:43.503472 32451 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:43.503851 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:43.504498 25242 raft_consensus.cc:3060] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 21
I20260709 13:10:43.508665 32483 raft_consensus.cc:3055] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:43.503919 32363 raft_consensus.cc:3055] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:43.509615 32363 raft_consensus.cc:740] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:43.510509 32363 consensus_queue.cc:260] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.510747 32387 consensus_queue.cc:237] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 19.17, Last appended by leader: 17, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.511585 32363 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:43.513366 25287 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:43.514593 32451 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:43.515172 32451 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:43.516223 25141 catalog_manager.cc:5697] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 8 to 14. New cstate: current_term: 14 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:43.517714 25141 catalog_manager.cc:5697] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 14 to 18. New cstate: current_term: 18 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:43.519161 32270 raft_consensus.cc:3055] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:43.519634 32270 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:43.519393 25141 catalog_manager.cc:5697] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 9 to 18. New cstate: current_term: 18 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:43.520596 32270 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.521782 32270 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:43.507720 25209 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:43.537609 25141 catalog_manager.cc:5697] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 12 to 16. New cstate: current_term: 16 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:43.538473 32178 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 13:10:43.540014 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 103769us
W20260709 13:10:43.541672 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 47662.4us
I20260709 13:10:43.544209 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:43.546561 32382 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:43.546954 25209 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:43.547487 32333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:59310 (request call id 1329) took 4402 ms. Trace:
I20260709 13:10:43.540149 25141 catalog_manager.cc:5697] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 7 to 20. New cstate: current_term: 20 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
W20260709 13:10:43.548733 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:43.553292 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:43.549216 31386 consensus_queue.cc:1059] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.012s
I20260709 13:10:43.565585 25936 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 21.
I20260709 13:10:43.570079 25242 raft_consensus.cc:2468] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 21.
I20260709 13:10:43.509143 32483 raft_consensus.cc:740] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:43.571983 25284 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:43.544945 32235 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:43.585090 31386 raft_consensus.cc:2804] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:10:43.585613 31386 raft_consensus.cc:697] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:43.561766 32284 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:43.587889 25209 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:43.593809 32483 consensus_queue.cc:260] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:43.595211 32483 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:43.547920 32333 rpcz_store.cc:276] 0709 13:10:39.144712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:39.145938 (+  1226us) service_pool.cc:224] Handling call
0709 13:10:39.792874 (+646936us) ts_tablet_manager.cc:934] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
0709 13:10:42.405811 (+2612937us) tablet_copy_client.cc:368] Tablet copy session begun
0709 13:10:43.024021 (+618210us) tablet_metadata.cc:665] Metadata flushed
0709 13:10:43.027828 (+  3807us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 13:10:43.546511 (+518683us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 13:10:43.546869 (+   358us) tablet_replica.cc:206] Creating consensus instance
0709 13:10:43.547252 (+   383us) ts_tablet_manager.cc:989] Replica registered.
0709 13:10:43.547372 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":53762,"tablet-copy.queue_time_us":645737,"thread_start_us":1268605,"threads_started":2}
I20260709 13:10:43.598691 32333 tablet_copy_client.cc:806] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 0 data blocks...
I20260709 13:10:43.599305 32333 tablet_copy_client.cc:670] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Starting download of 1 WAL segments...
I20260709 13:10:43.600647 25287 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:43.602620 25287 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:43.533356 32451 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:43.606515 32353 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:43.607049 32451 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:43.607877 32380 consensus_queue.cc:1048] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:10:43.609783 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:43.610610 32243 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:43.610936 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:43.611769 32243 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:43.612264 32243 raft_consensus.cc:697] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:43.615662 25358 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:43.621183 32247 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.623677 31386 consensus_queue.cc:237] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.625094 32247 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:43.624846 25141 catalog_manager.cc:5697] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 12 to 18. New cstate: current_term: 18 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:43.626775 25141 catalog_manager.cc:5697] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 10 to 17, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 17 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:43.628821 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:43.624868 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:43.630368 25936 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 21
I20260709 13:10:43.629603 25141 catalog_manager.cc:5697] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 10 to 16. New cstate: current_term: 16 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:43.608228 25287 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:43.627942 32178 raft_consensus.cc:3055] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:43.635602 25320 raft_consensus.cc:2468] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 21.
I20260709 13:10:43.639307 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:43.586352 32235 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:43.665993 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:43.664847 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:43.668848 25392 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:43.667433 25246 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:10:43.672782 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:43.673635 25242 raft_consensus.cc:3060] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:43.679992 25936 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 21.
I20260709 13:10:43.637027 32178 raft_consensus.cc:740] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:43.638835 32353 raft_consensus.cc:2764] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
W20260709 13:10:43.682816 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:43.707515 25386 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
W20260709 13:10:43.709373 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 101210us
I20260709 13:10:43.677510 32039 raft_consensus.cc:2764] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election decision vote started in defunct term 20: won
I20260709 13:10:43.682765 25284 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:43.671653 32133 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:10:43.667176 32297 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:43.672048 32247 raft_consensus.cc:2804] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:43.677196 32451 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:10:43.734875 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 52171.8us
I20260709 13:10:43.675118 32485 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:43.738894 25242 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 21.
I20260709 13:10:43.682793 31898 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.062s
I20260709 13:10:43.613101 32243 consensus_queue.cc:237] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.718690 32405 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:43.720217 32133 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:43.770143 25392 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 22.
I20260709 13:10:43.710451 32284 consensus_queue.cc:1059] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.035s
W20260709 13:10:43.771185 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:43.734443 32451 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:43.766400 32343 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:43.767343 32363 raft_consensus.cc:2804] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 13:10:43.800725 32408 consensus_meta.cc:287] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.561s	user 0.010s	sys 0.000s
W20260709 13:10:43.865123 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 98033.5us
I20260709 13:10:43.799896 32451 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:43.768697 32133 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:43.876154 32298 raft_consensus.cc:3055] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:43.809592 32382 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:43.774994 32405 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.891275 25395 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:43.872210 32178 consensus_queue.cc:260] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:43.898319 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 269550us
I20260709 13:10:43.885218 31850 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:43.885362 32298 raft_consensus.cc:740] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:10:43.909879 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 182344us
I20260709 13:10:43.895824 32363 raft_consensus.cc:697] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:43.911087 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:43.900971 32244 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:43.931035 32405 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:43.932315 32244 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:10:43.934832 32244 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:43.935993 32298 consensus_queue.cc:260] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.936893 32178 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:43.937057 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:43.937469 32298 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:43.892256 32382 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.943962 32382 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:43.908993 32247 raft_consensus.cc:697] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:43.945945 32247 consensus_queue.cc:237] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:43.910450 31850 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:43.949863 31850 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:43.951715 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:43.953379 25387 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 19 index: 17. Preceding OpId from leader: term: 21 index: 18. (index mismatch)
I20260709 13:10:43.956230 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:43.957727 32496 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 13:10:43.976428 25209 leader_election.cc:400] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 22. Current term is 23.
I20260709 13:10:43.976827 25209 leader_election.cc:403] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:10:43.983052 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 72243.7us
I20260709 13:10:43.983789 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:43.984833 32353 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:43.985301 32490 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Advancing to term 23
I20260709 13:10:43.985435 32497 raft_consensus.cc:2764] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 13:10:43.987092 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:43.987963 25238 raft_consensus.cc:2410] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 19 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 13:10:43.989930 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:43.990482 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:43.991034 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:43.992969 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:43.995595 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:43.995910 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:43.996445 25237 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:43.996790 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:43.987129 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:43.974395 32343 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:44.003108 32343 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:44.006603 25911 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:44.024256 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:44.032282 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 304508us
I20260709 13:10:44.037473 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:44.038138 25239 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
W20260709 13:10:44.042901 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 132086us
I20260709 13:10:44.043437 25242 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 13:10:44.040982 25239 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:10:44.044162 25287 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:44.042980 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:44.047385 25212 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:44.050352 31386 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:44.050828 31386 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:44.051172 31386 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:44.051520 25287 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:44.057229 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:44.059891 32032 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:44.061595 32333 tablet_copy_client.cc:538] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:10:44.023640 32382 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 13:10:44.067382 32270 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:44.042433 32439 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:10:44.083794 25240 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:44.069684 25287 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:44.094270 25287 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:44.096285 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:44.102244 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:44.102589 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:44.101687 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:44.108270 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:44.111075 25939 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:10:44.112979 25942 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:10:44.113595 25939 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:10:44.108363 25387 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:44.117012 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:44.117933 25241 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:43.985152 32363 consensus_queue.cc:237] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:44.125278 25940 raft_consensus.cc:1275] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 13:10:44.082640 32270 raft_consensus.cc:697] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 16 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:44.105782 32247 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:44.134984 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:44.115490 32439 raft_consensus.cc:697] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 22 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:44.136554 25287 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:44.098750 32382 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.034s
I20260709 13:10:44.151471 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:44.101449 32451 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:44.125389 32178 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:44.171500 25237 raft_consensus.cc:2468] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 23.
W20260709 13:10:44.179847 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:44.180104 32508 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:44.183722 25287 leader_election.cc:400] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19.
I20260709 13:10:44.184083 25287 leader_election.cc:403] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:10:44.142403 32380 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.042s
I20260709 13:10:44.185212 32491 raft_consensus.cc:3060] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 19
I20260709 13:10:44.185339 25287 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:44.146919 32483 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:44.185992 32439 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:44.186302 32483 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:44.189735 32274 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:10:44.192325 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:44.193174 25940 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:10:44.193418 25209 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:44.193677 25287 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:44.194537 32494 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:10:44.194727 32508 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:44.195194 32494 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:44.194453 32353 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:44.196164 32353 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:44.196532 32353 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:44.195681 32494 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:44.194959 32491 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19.
I20260709 13:10:44.197927 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:44.185621 32068 consensus_queue.cc:1059] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 13:10:44.194487 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 48537.1us
I20260709 13:10:44.135926 32133 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:44.169661 32503 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:44.217707 25287 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:44.217845 25245 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:44.146801 32244 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.071s
I20260709 13:10:44.217063 31386 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:44.228623 25287 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:44.229843 25392 raft_consensus.cc:1240] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 21. Current term is 22. Ops: []
I20260709 13:10:44.233294 25287 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:44.238946 25209 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:44.239818 25936 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 22 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 21. Current term is 22. Ops: []
I20260709 13:10:44.223317 32503 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:44.243705 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:44.246052 25240 raft_consensus.cc:2393] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 19: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:44.207549 32299 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 13:10:44.217301 32508 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.029s
I20260709 13:10:44.230255 32178 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:44.230813 32491 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:44.247157 25359 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:44.248723 25141 catalog_manager.cc:5697] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 9 to 13, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 13 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
W20260709 13:10:44.241428 32512 log.cc:927] Time spent T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 13:10:44.247105 32503 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:44.225731 32502 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.027s
I20260709 13:10:44.249687 32501 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:44.255541 32497 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 13:10:44.274547 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 44696.9us
I20260709 13:10:44.275319 25287 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:44.278568 32501 raft_consensus.cc:2764] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 13:10:44.281970 25317 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:10:44.282670 25317 raft_consensus.cc:606] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:44.282459 25358 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:44.283644 32451 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:44.236111 32270 consensus_queue.cc:237] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:44.288573 25243 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16.
I20260709 13:10:44.291869 32343 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:44.292639 25391 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:44.300751 32268 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:44.301151 25358 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:44.302120 32268 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:44.302603 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:44.303511 32268 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 13:10:44.304366 32513 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 13:10:44.305408 25209 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:44.283044 25317 raft_consensus.cc:614] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:10:44.307267 31898 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:44.322984 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:44.261668 32358 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 13:10:44.309376 32451 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:44.306453 32461 consensus_queue.cc:1048] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.032s
I20260709 13:10:44.347438 32068 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:44.348054 32068 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:44.348594 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:44.350287 32068 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:44.351246 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:44.351915 25939 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:44.353462 25284 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:44.351387 32133 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:44.354029 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:44.354883 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:44.357991 32508 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:44.358565 32508 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:44.359025 32508 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:44.364495 25317 raft_consensus.cc:2468] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:44.364818 25908 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:44.376780 25908 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:44.364234 32490 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 22. Current term is 23.
W20260709 13:10:44.379495 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 43830.5us
W20260709 13:10:44.403416 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:44.466976 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 90282.8us
I20260709 13:10:44.378921 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:44.471758 25317 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:44.489732 25209 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:44.379524 32439 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:10:44.506976 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:44.383513 31850 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:44.522957 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:44.524142 25317 raft_consensus.cc:2410] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:10:44.505103 32333 tablet_bootstrap.cc:492] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59: Bootstrap starting.
I20260709 13:10:44.506371 32451 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:44.533496 25209 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:44.541599 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:44.555745 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:44.570248 25212 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:44.571216 25358 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:44.575898 25209 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:44.596863 25238 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 23.
I20260709 13:10:44.605301 32439 raft_consensus.cc:697] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 23 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:44.612028 32439 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:44.613121 32524 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
W20260709 13:10:44.614653 32526 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:10:44.570714 32270 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:44.544587 32503 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:44.635177 32503 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:44.635988 25237 raft_consensus.cc:1240] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:44.639391 25320 raft_consensus.cc:1275] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:10:44.640120 25314 raft_consensus.cc:1275] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 13:10:44.640792 25320 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 15. (term mismatch)
I20260709 13:10:44.641152 25320 pending_rounds.cc:85] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 14
I20260709 13:10:44.641031 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:44.641439 25320 pending_rounds.cc:107] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 19.15
I20260709 13:10:44.641757 25320 raft_consensus.cc:2889] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:44.642918 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:44.641672 25314 raft_consensus.cc:2376] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17. Re-sending same reply.
I20260709 13:10:44.578761 32497 raft_consensus.cc:2764] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
W20260709 13:10:44.661413 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 85614.8us
I20260709 13:10:44.661499 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:44.600149 32451 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:44.650691 32490 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:10:44.670583 25313 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:10:44.668542 32178 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:44.671803 32178 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:44.673981 32178 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:44.673844 32353 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:44.676716 32353 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:44.677883 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:44.668614 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:44.680605 32483 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:44.686913 25246 raft_consensus.cc:1275] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:10:44.687849 25392 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 13:10:44.689447 32451 raft_consensus.cc:697] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 17 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:10:44.646466 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 138893us
I20260709 13:10:44.649274 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:44.691510 25320 raft_consensus.cc:2376] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 23. Re-sending same reply.
I20260709 13:10:44.669685 25315 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:44.692698 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:44.706115 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:44.708328 25358 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:44.721052 25358 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:44.722577 25358 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:44.735050 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:44.736157 25246 raft_consensus.cc:2393] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 19: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:44.736835 25209 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:44.737494 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:44.738209 25318 raft_consensus.cc:2376] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 21. Re-sending same reply.
I20260709 13:10:44.744527 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:44.746630 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:44.751717 32244 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:44.686867 32382 consensus_queue.cc:1059] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.048s
I20260709 13:10:44.752790 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:44.754937 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:44.757062 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:44.742167 32510 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:44.755606 32497 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:10:44.742425 32359 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:44.763012 25318 raft_consensus.cc:2393] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 16: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:44.755885 32538 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.011s
W20260709 13:10:44.775362 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 227747us
I20260709 13:10:44.690811 32451 consensus_queue.cc:237] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:10:44.763749 32510 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:44.765045 32353 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:10:44.795022 25209 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:44.808955 25246 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:10:44.808869 25209 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:44.775988 32133 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:44.779798 32244 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:44.812844 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:44.794032 32510 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:44.810030 32359 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:44.813529 32270 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:44.865231 32539 raft_consensus.cc:2764] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
W20260709 13:10:44.866112 32525 log.cc:927] Time spent T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
I20260709 13:10:44.835404 32511 raft_consensus.cc:3055] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:10:44.817638 32499 consensus_queue.cc:1048] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
W20260709 13:10:44.877908 32535 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:10:44.869006 32537 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:44.861425 32359 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:44.889138 32537 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:44.891423 32359 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:44.846529 32497 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:10:44.892576 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:44.893469 25246 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:44.895008 32537 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:44.896741 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:44.903119 25320 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:44.897872 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:44.899310 25321 raft_consensus.cc:1275] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 13:10:44.913332 25319 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:10:44.839344 32508 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:44.872269 32511 raft_consensus.cc:740] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:44.930976 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:44.914252 25319 raft_consensus.cc:606] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:44.927085 32460 raft_consensus.cc:3055] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Stepping down as leader of term 19
W20260709 13:10:44.913187 32507 log.cc:927] Time spent T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 13:10:44.937876 25246 raft_consensus.cc:1275] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:10:44.945101 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:44.946702 25938 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
W20260709 13:10:44.947692 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 162049us
I20260709 13:10:44.947297 25937 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:44.949929 25316 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 20: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:44.929910 31898 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:44.950035 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 137258us
I20260709 13:10:44.951788 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:44.937595 32460 raft_consensus.cc:740] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:44.942062 32380 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:10:44.947329 32494 consensus_meta.cc:287] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.726s	user 0.012s	sys 0.000s
I20260709 13:10:44.954059 31898 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:44.968757 32458 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.003s
I20260709 13:10:44.974879 32546 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:44.975536 32546 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:44.978781 25209 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:44.962735 32451 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:44.951423 32283 consensus_queue.cc:1048] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.027s
I20260709 13:10:44.954523 32244 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.026s
I20260709 13:10:44.994815 32508 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:44.964625 32494 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:45.003512 32494 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:45.005179 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:45.001480 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:45.007023 25943 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:45.005837 25314 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:44.996613 32270 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:44.986369 32497 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:10:45.015899 25908 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:45.017879 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:45.021842 25319 raft_consensus.cc:3055] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:10:45.022221 25319 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:45.017426 32536 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:45.020323 32497 raft_consensus.cc:697] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 20 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:45.027349 32497 consensus_queue.cc:237] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:44.979933 32510 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:45.032195 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 64344.2us
I20260709 13:10:45.036281 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:45.036646 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:45.040228 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:45.036427 25319 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:45.036895 25318 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:45.041231 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:45.040625 25908 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:45.042431 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:45.044577 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:45.044946 25319 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:45.031109 32032 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:45.049366 32032 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:45.046144 32546 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:45.052237 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:44.984709 32460 consensus_queue.cc:260] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:45.041170 32499 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:45.053229 32499 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:45.041399 32283 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:45.054083 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:45.054240 32283 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:45.055239 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:45.055753 32032 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:45.047595 32439 consensus_queue.cc:1048] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:10:45.047369 32540 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:45.056764 32540 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:45.057113 32540 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:45.055902 25312 raft_consensus.cc:2393] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 17: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:45.058422 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:45.061242 32533 log.cc:927] Time spent T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.393s	user 0.000s	sys 0.003s
I20260709 13:10:44.983978 32511 consensus_queue.cc:260] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:45.065730 32511 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:45.041976 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:45.068297 32298 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:10:45.071427 32483 consensus_queue.cc:1059] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 13:10:45.071800 32531 log.cc:927] Time spent T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.522s	user 0.002s	sys 0.000s
I20260709 13:10:45.073297 32247 raft_consensus.cc:3055] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:45.073686 32247 raft_consensus.cc:740] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:45.073861 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:45.074563 32247 consensus_queue.cc:260] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:45.056332 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:45.075995 32247 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:45.076594 32387 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:45.077915 25909 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:45.079031 25313 raft_consensus.cc:1275] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 19 index: 17. Preceding OpId from leader: term: 21 index: 18. (index mismatch)
I20260709 13:10:45.056300 32283 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:45.084467 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:45.084522 25209 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:45.086717 25943 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:45.085297 25241 raft_consensus.cc:2376] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16. Re-sending same reply.
I20260709 13:10:45.091537 25940 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:45.093103 25319 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 13:10:45.099407 25321 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 15.
W20260709 13:10:45.096426 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:45.103631 25209 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:45.108691 25909 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:45.106032 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:45.113473 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:45.113945 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:45.116040 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:45.116519 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:45.055235 32499 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:45.072959 32483 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.040s
I20260709 13:10:45.092649 32494 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:45.145347 25321 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:45.148811 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:45.149613 25321 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:45.114053 32439 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:45.112759 32270 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:10:45.152647 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:45.160197 25316 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:45.125047 32538 raft_consensus.cc:2804] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:10:45.168629 25909 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
W20260709 13:10:45.083151 32536 consensus_queue.cc:1175] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "2329e428ebbf484fa3ea9452133df567" responder_term: 21 status { last_received { term: 21 index: 15 } last_committed_idx: 15 last_received_current_leader { term: 21 index: 15 } }
I20260709 13:10:45.144765 32451 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:45.185822 32510 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:45.190456 32555 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
I20260709 13:10:45.173049 32556 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:45.192638 32556 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:45.194669 32552 log.cc:927] Time spent T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.187s	user 0.001s	sys 0.000s
I20260709 13:10:45.194927 32556 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:45.198423 31898 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:10:45.200801 31898 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:45.173170 32270 raft_consensus.cc:697] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:45.175019 32405 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:10:45.210671 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 94229.4us
I20260709 13:10:45.211887 32270 consensus_queue.cc:237] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:45.146796 32363 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 13:10:45.204932 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:45.213721 25316 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:45.206931 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:45.207549 32451 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:45.218924 25320 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:45.219422 25285 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:45.220515 32439 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:10:45.220602 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 52106.5us
I20260709 13:10:45.221570 32343 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:45.224184 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:45.224967 25246 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 19: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:45.225186 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:45.225517 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:45.225695 25240 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:45.227212 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44954 (request call id 1252) took 1233 ms. Trace:
I20260709 13:10:45.227396 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:45.227573 25238 rpcz_store.cc:276] 0709 13:10:43.994541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:43.995324 (+   783us) service_pool.cc:224] Handling call
0709 13:10:45.227043 (+1231719us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b54002d0fd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:45.227138 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1171622784}
I20260709 13:10:45.229090 25393 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:10:45.229465 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:45.229854 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:45.230569 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:45.231122 25245 raft_consensus.cc:2376] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 16. Re-sending same reply.
I20260709 13:10:45.232015 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:45.233215 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:45.233285 32538 raft_consensus.cc:697] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 23 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:45.233451 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:45.234222 25941 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:45.234362 32538 consensus_queue.cc:237] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:45.235525 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:45.225057 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:45.238170 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:45.225538 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:45.239185 25241 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 21: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:45.230578 25386 raft_consensus.cc:2393] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 16: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:45.245136 25909 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:45.247100 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 70928.3us
I20260709 13:10:45.245807 32343 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
I20260709 13:10:45.248075 32405 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:45.248840 32483 consensus_queue.cc:1048] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:10:45.239624 32439 raft_consensus.cc:697] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:45.258543 32439 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.14, Last appended by leader: 14, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:45.261173 25909 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:45.220930 32451 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:45.264367 25386 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:45.265879 25320 raft_consensus.cc:2468] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:45.246887 32491 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:45.271716 25240 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:45.272699 25909 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:45.209271 32540 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:45.278260 32405 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:45.279047 32494 consensus_queue.cc:1059] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:10:45.280417 32494 raft_consensus.cc:3055] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:45.273204 32508 consensus_queue.cc:1059] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.019s
I20260709 13:10:45.290671 32566 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:45.293229 25313 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 19.
W20260709 13:10:45.293507 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 51837.8us
I20260709 13:10:45.294312 25238 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:45.300818 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:45.280865 32494 raft_consensus.cc:740] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:45.273854 32491 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:45.307420 32491 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:45.308239 32540 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:45.310051 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:45.315024 25141 catalog_manager.cc:5697] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 13 to 20. New cstate: current_term: 20 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:45.317444 25943 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:45.316846 25141 catalog_manager.cc:5697] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 11 to 14, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 14 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:45.303778 32511 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:45.332625 25316 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:45.306715 32566 raft_consensus.cc:697] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:45.339099 31386 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:45.330106 32270 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:45.350498 25237 raft_consensus.cc:1240] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 21. Current term is 22. Ops: []
I20260709 13:10:45.353025 25242 raft_consensus.cc:1275] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 13:10:45.358644 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:45.358858 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:45.359880 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:45.360462 25240 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:45.361204 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:45.365114 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 70889.9us
I20260709 13:10:45.366422 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:45.367481 25237 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:45.369787 25237 raft_consensus.cc:1275] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 15. (term mismatch)
I20260709 13:10:45.370214 25237 pending_rounds.cc:85] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 14
I20260709 13:10:45.370483 25237 pending_rounds.cc:107] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 18.15
I20260709 13:10:45.370762 25237 raft_consensus.cc:2889] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:45.378978 25209 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:45.348039 32494 consensus_queue.cc:260] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:45.399951 32566 consensus_queue.cc:237] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:45.407441 32490 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:45.409165 32502 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:45.411798 32494 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:10:45.418234 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 52410.8us
W20260709 13:10:45.515154 32568 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.215s	user 0.000s	sys 0.004s
I20260709 13:10:45.516680 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:45.517503 25209 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:45.518302 32270 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:45.519676 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:45.520412 32270 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:45.520777 32270 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:45.521265 32270 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:45.528878 32270 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:45.529963 25939 raft_consensus.cc:1275] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:10:45.530966 32270 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:45.529963 25141 catalog_manager.cc:5697] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 9 to 19. New cstate: current_term: 19 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:10:45.531574 32502 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:45.532897 32565 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:45.533385 32565 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:45.532598 25141 catalog_manager.cc:5697] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 10 to 17. New cstate: current_term: 17 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:45.535049 32501 raft_consensus.cc:3055] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:45.535406 32501 raft_consensus.cc:740] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:45.544075 25386 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
W20260709 13:10:45.544274 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 125519us
I20260709 13:10:45.551748 25209 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:45.556181 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 206842us
I20260709 13:10:45.558844 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:45.561628 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 207202us
I20260709 13:10:45.561960 32497 consensus_queue.cc:1059] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:10:45.563083 32497 raft_consensus.cc:3055] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:45.563457 32497 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:45.564293 32497 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:45.565994 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:45.561622 25242 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 13:10:45.573084 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:45.575220 25209 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:45.557443 32451 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:10:45.572439 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:45.565943 32497 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:45.555976 31898 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:45.589066 25909 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:45.593034 25284 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:45.591404 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:45.595530 25938 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:45.598276 25909 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:45.604797 25908 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:45.607702 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:45.608448 25320 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:45.578819 32556 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:10:45.612671 25315 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:10:45.558316 32490 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 13:10:45.609269 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:45.610121 25359 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:45.610852 25316 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 13:10:45.587303 32546 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:45.554611 32510 consensus_queue.cc:1059] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 13:10:45.586745 32537 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:45.613464 25212 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:45.627830 25320 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:45.630163 32387 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:45.630769 32387 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:45.631247 32387 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:45.633235 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:45.613833 32550 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:45.619115 32268 consensus_queue.cc:1048] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.022s
I20260709 13:10:45.555248 32491 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:45.649385 32451 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 13:10:45.650534 32577 log.cc:927] Time spent T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 13:10:45.649610 32380 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:45.651129 32380 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:45.651489 32380 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:45.649719 32460 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:45.614084 32511 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 13:10:45.663170 32573 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.247s	user 0.000s	sys 0.001s
I20260709 13:10:45.569618 32501 consensus_queue.cc:260] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:45.670037 32501 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:45.672722 32511 raft_consensus.cc:697] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:45.677402 25141 catalog_manager.cc:5697] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 8 to 14, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 14 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:45.678864 25141 catalog_manager.cc:5697] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 9 to 16, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 16 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:45.615986 32502 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:45.658879 25359 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:45.646126 25938 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
W20260709 13:10:45.694444 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 69139.8us
I20260709 13:10:45.698760 25358 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:45.596567 32032 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.057s
I20260709 13:10:45.611306 31898 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:45.618283 32536 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.065s
I20260709 13:10:45.653147 32353 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 13:10:45.718724 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 93498.2us
I20260709 13:10:45.730175 31898 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:45.731761 25358 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:45.733623 32114 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:45.734191 32114 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:45.734678 32114 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:45.735370 32510 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:45.737339 25315 raft_consensus.cc:606] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:45.737884 25315 raft_consensus.cc:614] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:10:45.732681 25136 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53264 (request call id 210) took 11178 ms. Trace:
W20260709 13:10:45.682324 32508 consensus_queue.cc:1175] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" responder_term: 18 status { last_received { term: 18 index: 10 } last_committed_idx: 10 last_received_current_leader { term: 18 index: 10 } }
I20260709 13:10:45.700693 32483 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:45.740914 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 147984us
I20260709 13:10:45.744575 25284 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:45.741041 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 82238.8us
I20260709 13:10:45.739461 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:45.743155 25209 consensus_peers.cc:597] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:45.746906 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 50833.9us
I20260709 13:10:45.747789 25209 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:45.647118 32537 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:45.724439 32458 raft_consensus.cc:3055] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:10:45.762063 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:45.763015 25393 raft_consensus.cc:2393] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 19: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:45.772285 25387 raft_consensus.cc:1275] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:10:45.763731 25386 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:45.752182 32502 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:45.744999 32353 raft_consensus.cc:697] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 20 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:45.792917 25240 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:45.720695 32387 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:45.817915 25320 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:10:45.778232 25395 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 21. Current term is 22. Ops: []
I20260709 13:10:45.758490 32556 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:45.771520 25358 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:45.760197 32458 raft_consensus.cc:740] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:45.823436 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:45.814554 25242 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 20. Current term is 21. Ops: []
W20260709 13:10:45.815634 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:45.810540 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:45.815901 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:45.779039 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:45.815934 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:45.779417 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:45.815439 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:45.780011 25316 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:45.778985 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:45.819347 25390 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:45.821727 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:45.806286 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:45.762302 31850 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:10:45.832913 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 61468.7us
I20260709 13:10:45.833900 25358 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:45.826459 25287 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:45.832242 25313 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:10:45.815424 32502 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:45.777704 32460 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:45.761879 32461 consensus_queue.cc:1048] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.061s
I20260709 13:10:45.754691 32483 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:45.835127 32333 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:10:45.771837 32538 consensus_queue.cc:1048] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.061s
I20260709 13:10:45.818184 32274 consensus_queue.cc:1048] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.034s
W20260709 13:10:45.878481 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 63446.6us
I20260709 13:10:45.848752 32268 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:45.848457 32551 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:45.907208 32551 raft_consensus.cc:697] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 19 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:45.843513 32359 raft_consensus.cc:2804] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:45.924166 25391 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:45.913185 25318 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:45.813194 32511 consensus_queue.cc:237] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:45.842708 31850 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.062s
I20260709 13:10:45.843932 32502 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:45.902698 32501 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:45.937721 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:45.928766 32274 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:10:45.879467 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 53127.9us
I20260709 13:10:45.927246 25908 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:45.929821 32556 raft_consensus.cc:697] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:45.951843 32567 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.174s
W20260709 13:10:45.952082 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 136711us
W20260709 13:10:45.952339 32583 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
W20260709 13:10:45.952692 32576 log.cc:927] Time spent T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 13:10:45.952585 32584 log.cc:927] Time spent T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
W20260709 13:10:45.953438 32588 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
W20260709 13:10:45.954043 32582 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
W20260709 13:10:45.954820 32585 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 13:10:45.955555 32483 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:45.952662 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:45.971587 25391 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:10:45.914798 32458 consensus_queue.cc:260] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:45.933888 32491 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:45.940404 32501 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:45.939332 32359 raft_consensus.cc:697] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:45.942975 32537 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:46.001983 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:46.000941 25242 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 13:10:46.001677 25237 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
W20260709 13:10:45.987887 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 60946.9us
I20260709 13:10:45.947613 32546 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:10:45.953254 32575 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 13:10:46.003788 25287 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:46.019618 32333 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 13:10:45.926551 32353 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:45.993891 32490 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:46.024348 32268 raft_consensus.cc:697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:46.047024 25908 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:45.995841 32387 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:45.988597 32565 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:46.059667 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 59552.7us
I20260709 13:10:46.005019 32510 consensus_queue.cc:1059] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 20.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.038s
W20260709 13:10:46.049651 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 135919us
I20260709 13:10:46.002235 32247 consensus_queue.cc:1059] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
W20260709 13:10:46.066372 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 59988.9us
I20260709 13:10:46.085881 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:46.086560 25389 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:46.059021 32537 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:46.057679 32566 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:46.083431 32458 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:46.094594 32268 consensus_queue.cc:237] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:46.000486 32551 consensus_queue.cc:237] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:45.740326 25136 rpcz_store.cc:276] 0709 13:10:34.553625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:34.582081 (+ 28456us) service_pool.cc:224] Handling call
0709 13:10:45.732390 (+11150309us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:10:40.868972 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:10:40.869580 (+   608us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:10:40.869616 (+    36us) write_op.cc:435] Acquired schema lock
0709 13:10:40.869635 (+    19us) tablet.cc:662] Decoding operations
0709 13:10:40.871275 (+  1640us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:10:40.871330 (+    55us) write_op.cc:641] Partition lock acquired for write op
0709 13:10:40.871358 (+    28us) tablet.cc:685] Acquiring locks for 21 operations
0709 13:10:41.195789 (+324431us) tablet.cc:701] Row locks acquired
0709 13:10:41.217745 (+ 21956us) write_op.cc:260] PREPARE: finished
0709 13:10:41.255908 (+ 38163us) write_op.cc:270] Start()
0709 13:10:41.279870 (+ 23962us) write_op.cc:276] Timestamp: P: 1783602641248581 usec, L: 0
0709 13:10:41.282972 (+  3102us) op_driver.cc:348] REPLICATION: starting
0709 13:10:41.343883 (+ 60911us) log.cc:844] Serialized 8319 byte log entry
0709 13:10:41.907362 (+563479us) op_driver.cc:464] REPLICATION: finished
0709 13:10:42.121388 (+214026us) write_op.cc:301] APPLY: starting
0709 13:10:42.144444 (+ 23056us) tablet.cc:1367] starting BulkCheckPresence
0709 13:10:42.520232 (+375788us) tablet.cc:1370] finished BulkCheckPresence
0709 13:10:42.520350 (+   118us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:10:43.043301 (+522951us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:10:43.043835 (+   534us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=42,key_file_lookups=42,delta_file_lookups=42,mrs_lookups=0
0709 13:10:43.043915 (+    80us) write_op.cc:312] APPLY: finished
0709 13:10:43.044633 (+   718us) log.cc:844] Serialized 188 byte log entry
0709 13:10:43.185828 (+141195us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:10:43.221364 (+ 35536us) write_op.cc:454] Released schema lock
0709 13:10:43.221445 (+    81us) write_op.cc:341] FINISH: Updating metrics
0709 13:10:43.221974 (+   529us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:10:43.222006 (+    32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":729334,"reported_tablets":35,"child_traces":[["op",{"apply.queue_time_us":213584,"apply.run_cpu_time_us":75288,"apply.run_wall_time_us":1100585,"cfile_cache_hit":86,"cfile_cache_hit_bytes":91823,"num_ops":21,"prepare.queue_time_us":461278,"prepare.run_cpu_time_us":62072,"prepare.run_wall_time_us":590288,"raft.queue_time_us":236828,"raft.run_cpu_time_us":33295,"raft.run_wall_time_us":221991,"replication_time_us":586451,"thread_start_us":541240,"threads_started":5,"wal-append.queue_time_us":281543,"wal-append.run_cpu_time_us":12331,"wal-append.run_wall_time_us":2212727}]]}
W20260709 13:10:46.109220 32579 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260709 13:10:46.109386 32592 consensus_queue.cc:1048] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.109s
I20260709 13:10:46.112347 32299 consensus_queue.cc:1059] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:10:46.114238 32461 raft_consensus.cc:3055] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:46.114665 32461 raft_consensus.cc:740] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:46.115521 32461 consensus_queue.cc:260] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:46.116788 32461 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:46.061170 32490 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:46.132542 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:46.141847 25319 raft_consensus.cc:2393] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 15: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:46.130752 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:46.143299 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:46.144052 25316 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:46.145606 25313 raft_consensus.cc:3060] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:46.139739 25312 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
W20260709 13:10:46.114686 32587 log.cc:927] Time spent T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 13:10:46.155058 25317 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260709 13:10:46.117357 32566 raft_consensus.cc:697] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:46.125754 32594 raft_consensus.cc:3055] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:46.155165 25287 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:46.178920 25313 raft_consensus.cc:1275] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 13:10:46.161929 32592 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:46.092358 32556 consensus_queue.cc:237] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:46.179283 32380 consensus_meta.cc:287] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.527s	user 0.011s	sys 0.000s
I20260709 13:10:46.169427 32594 raft_consensus.cc:740] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:46.142236 32359 consensus_queue.cc:237] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:10:46.251211 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:46.194604 32540 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:46.187448 32605 consensus_queue.cc:1059] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.114s
I20260709 13:10:46.252707 32483 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:46.252206 31850 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.054s
I20260709 13:10:46.204974 32592 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:46.249275 32380 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:46.373449 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32994 (request call id 2960) took 1013 ms. Trace:
W20260709 13:10:46.433053 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 277515us
I20260709 13:10:46.462111 25240 rpcz_store.cc:276] 0709 13:10:45.359817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:45.360981 (+  1164us) service_pool.cc:224] Handling call
0709 13:10:46.373242 (+1012261us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b5400192b54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:46.373379 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":947974912}
W20260709 13:10:46.476210 25238 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.112s	user 0.015s	sys 0.027s
W20260709 13:10:46.478722 25238 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.115s	user 0.015s	sys 0.027s
W20260709 13:10:46.370873 32604 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.198s	user 0.000s	sys 0.000s
W20260709 13:10:46.307369 32114 consensus_meta.cc:287] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.572s	user 0.014s	sys 0.000s
I20260709 13:10:46.538561 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2950) took 1129 ms. Trace:
I20260709 13:10:46.291265 32247 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:46.299685 32595 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.244s	user 0.001s	sys 0.000s
I20260709 13:10:46.539611 25386 raft_consensus.cc:2368] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20.
I20260709 13:10:46.305274 32500 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:10:46.541987 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:46.542006 25238 rpcz_store.cc:276] 0709 13:10:45.350347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:45.351870 (+  1523us) service_pool.cc:224] Handling call
0709 13:10:45.360597 (+  8727us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:45.362912 (+  2315us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:45.362940 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:45.363408 (+   468us) log.cc:844] Serialized 26 byte log entry
0709 13:10:46.479167 (+1115759us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:46.479227 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:46.479322 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:46.479386 (+    64us) raft_consensus.cc:1685] finished
0709 13:10:46.479403 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:46.479572 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1936,"raft.queue_time_us":2616,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":129,"spinlock_wait_cycles":21760,"thread_start_us":1115340,"threads_started":1,"wal-append.queue_time_us":485589}
W20260709 13:10:46.540261 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 75300.5us
I20260709 13:10:46.385365 32490 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:46.293558 32501 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:46.538300 32502 consensus_meta.cc:287] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.598s	user 0.010s	sys 0.000s
I20260709 13:10:46.372613 32451 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:46.367652 32608 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:46.540970 32358 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:46.558650 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 494295us
I20260709 13:10:46.561678 25313 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:46.549011 32500 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:46.300310 32603 consensus_queue.cc:1059] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.295s
I20260709 13:10:46.289773 32566 consensus_queue.cc:237] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:46.472609 32483 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:46.367481 32602 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.322s
I20260709 13:10:46.367787 32613 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.376s
I20260709 13:10:46.546280 32114 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:46.548345 32247 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:46.586066 32497 consensus_meta.cc:287] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.990s	user 0.013s	sys 0.000s
I20260709 13:10:46.384796 32594 consensus_queue.cc:260] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:46.558689 32502 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:46.579291 32500 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:46.571091 32451 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:46.608670 32497 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:46.613180 32619 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.204s	user 0.001s	sys 0.000s
W20260709 13:10:46.614282 32627 log.cc:927] Time spent T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 13:10:46.614812 32483 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:46.616628 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:46.616816 32502 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:46.617691 32594 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:46.618073 25387 raft_consensus.cc:2368] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20.
I20260709 13:10:46.601356 32551 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.020s
I20260709 13:10:46.622893 32551 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:46.623548 32551 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:46.574169 32608 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:46.626130 32608 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:46.579310 32358 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:46.612840 32497 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:46.655498 25284 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:46.577824 32629 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 13:10:46.692530 32333 tablet_bootstrap.cc:492] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=1 applied=10 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 13:10:46.693364 32497 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:46.693356 32380 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:46.716075 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:46.717263 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:46.720088 32651 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:46.720651 32651 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:46.756619 32247 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:46.751261 32592 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:46.758469 25241 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260709 13:10:46.764796 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 47783.6us
I20260709 13:10:46.765965 25358 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:46.782895 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 354484us
I20260709 13:10:46.782087 25315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49638 (request call id 47) took 1176 ms. Trace:
I20260709 13:10:46.784962 25315 rpcz_store.cc:276] 0709 13:10:45.587848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:45.587970 (+   122us) service_pool.cc:224] Handling call
0709 13:10:46.763987 (+1176017us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b54004399d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cac853 00007f297bc7a60a 00007f29752007d4 00007f2975200762 00007f2975200691 00007f297520060c 00007f29751ff8c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:46.764098 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":285029760}
W20260709 13:10:46.772818 32596 log.cc:927] Time spent T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.841s	user 0.004s	sys 0.007s
I20260709 13:10:46.793025 25395 raft_consensus.cc:1240] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:46.785763 32333 tablet_bootstrap.cc:492] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59: Bootstrap complete.
I20260709 13:10:46.795003 25911 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:46.794101 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:46.797072 25318 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:46.783522 32551 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:46.816816 25209 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:46.818115 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:46.810633 32566 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:10:46.808576 32114 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:46.811746 32603 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:46.947001 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 130311us
I20260709 13:10:46.904498 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:46.822404 32333 ts_tablet_manager.cc:1403] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59: Time spent bootstrapping tablet: real 2.450s	user 0.050s	sys 0.022s
I20260709 13:10:46.871326 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 2538) took 1062 ms. Trace:
W20260709 13:10:46.956223 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:46.965103 25318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:10:46.853562 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:46.842134 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:46.854310 25242 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:46.853353 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:46.988694 25245 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:46.898084 31386 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:46.813951 32644 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:46.888937 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:46.873932 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 2537) took 1068 ms. Trace:
I20260709 13:10:46.843024 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:46.883518 25940 raft_consensus.cc:1240] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 17. Current term is 18. Ops: [15.12-17.13]
I20260709 13:10:46.842326 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:46.850327 32609 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.755s	user 0.000s	sys 0.002s
I20260709 13:10:46.947962 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:46.977839 25387 rpcz_store.cc:276] 0709 13:10:45.810044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:45.810206 (+   162us) service_pool.cc:224] Handling call
0709 13:10:46.871059 (+1060853us) spinlock_profiling.cc:243] Waited 792 ms on lock 0x7b54003228d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:46.871245 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1821526272}
I20260709 13:10:46.989576 31386 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:46.903533 32621 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:46.892753 32598 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:46.981894 25240 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:46.854738 32274 raft_consensus.cc:993] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: : Instructing follower a2a77ecf2c0941618b5bc79d4c04db61 to start an election
I20260709 13:10:46.993201 25241 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:46.902671 32032 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:46.860131 32353 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:46.819880 32551 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:46.983476 32363 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:46.940486 32639 log.cc:927] Time spent T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
W20260709 13:10:46.944437 32638 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.183s	user 0.000s	sys 0.000s
I20260709 13:10:46.992401 25386 rpcz_store.cc:276] 0709 13:10:45.805391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:45.806038 (+   647us) service_pool.cc:224] Handling call
0709 13:10:46.873650 (+1067612us) spinlock_profiling.cc:243] Waited 710 ms on lock 0x7b5400481a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:46.873850 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1633362816}
W20260709 13:10:46.998281 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 163548us
I20260709 13:10:46.979576 25318 raft_consensus.cc:606] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:46.868046 31898 raft_consensus.cc:3055] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 13:10:47.022984 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 59283.4us
I20260709 13:10:47.081080 25386 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:46.907684 32387 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:46.825690 32629 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:46.940577 32633 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:46.896557 32636 raft_consensus.cc:3055] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:46.992813 32603 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.045015 32114 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:47.087461 32510 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.967s	user 0.027s	sys 0.029s
I20260709 13:10:46.943713 32649 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:47.045444 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:46.943897 32648 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:47.050642 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:47.083123 31898 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:47.025058 32644 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.122790 31898 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.103765 32636 raft_consensus.cc:740] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:47.124131 32644 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:47.127200 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 100581us
I20260709 13:10:46.983605 32358 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:47.027482 32451 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:47.082489 25937 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:47.091801 25241 raft_consensus.cc:2468] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:47.109362 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:47.109860 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:47.053262 32661 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:47.112273 25944 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 20. Current term is 21. Ops: []
I20260709 13:10:47.074679 32551 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.035140 32621 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.123787 31898 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:46.977229 32508 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:47.127964 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:47.155862 25392 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
W20260709 13:10:47.115386 32510 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.002s	user 0.028s	sys 0.030s
I20260709 13:10:47.092275 32629 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:47.137090 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:47.166505 25386 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 22: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:10:47.137771 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:46.993474 25237 raft_consensus.cc:2393] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 22: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:47.127919 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:47.167305 25387 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 20.
W20260709 13:10:47.139478 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 191586us
I20260709 13:10:47.119099 31386 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:47.047819 32032 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.108198 32270 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:47.124162 32614 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.630s	user 0.000s	sys 0.003s
I20260709 13:10:47.054814 32353 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.046833 32598 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.120060 32668 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:47.079082 32363 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.115012 32651 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:47.128458 32648 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:47.168999 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 1295) took 1041 ms. Trace:
I20260709 13:10:47.188216 25321 rpcz_store.cc:276] 0709 13:10:46.127449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:46.137009 (+  9560us) service_pool.cc:224] Handling call
0709 13:10:46.138189 (+  1180us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:46.160756 (+ 22567us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:46.160804 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:46.169117 (+  8313us) log.cc:844] Serialized 26 byte log entry
0709 13:10:46.554228 (+385111us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:10:46.554284 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:46.554378 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:47.168718 (+614340us) raft_consensus.cc:1685] finished
0709 13:10:47.168749 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:47.168925 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28401,"raft.queue_time_us":13669,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":10393,"thread_start_us":309153,"threads_started":1,"wal-append.queue_time_us":520579}
I20260709 13:10:47.144372 32460 raft_consensus.cc:1081] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Signalling peer a2a77ecf2c0941618b5bc79d4c04db61 to start an election
I20260709 13:10:47.124637 32636 consensus_queue.cc:260] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:10:47.171604 32629 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:47.148065 32451 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:47.109225 32387 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.115545 32114 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:47.187227 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 99235.1us
W20260709 13:10:47.187851 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 134039us
I20260709 13:10:47.129357 32550 consensus_queue.cc:1048] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.049s
W20260709 13:10:47.191829 32635 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.288s	user 0.001s	sys 0.000s
I20260709 13:10:47.169026 32508 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.215528 25938 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 22.
I20260709 13:10:47.129880 32649 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:47.209780 32634 log.cc:927] Time spent T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.237s	user 0.001s	sys 0.000s
I20260709 13:10:47.204530 32603 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:47.310109 32387 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:47.311174 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:47.147581 32661 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.258399 32598 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:47.205044 32601 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:47.316740 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:47.258487 32665 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.071s
I20260709 13:10:47.316962 32601 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.318969 32460 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:47.319473 32460 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.257333 32667 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:47.320464 32667 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.322713 32667 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:47.291610 32649 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:47.292376 25246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2971) took 1300 ms. Trace:
W20260709 13:10:47.261946 32611 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 1.013s	user 0.005s	sys 0.000s
I20260709 13:10:47.261636 32657 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:47.325256 32657 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:47.323982 25246 rpcz_store.cc:276] 0709 13:10:45.992337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:45.992517 (+   180us) service_pool.cc:224] Handling call
0709 13:10:45.992885 (+   368us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:45.993263 (+   378us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:10:46.057020 (+ 63757us) log.cc:844] Serialized 14 byte log entry
0709 13:10:46.555864 (+498844us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:46.571060 (+ 15196us) log.cc:844] Serialized 26 byte log entry
0709 13:10:46.571247 (+   187us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:10:46.571537 (+   290us) log.cc:844] Serialized 14 byte log entry
0709 13:10:46.689736 (+118199us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:46.690777 (+  1041us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:46.835163 (+144386us) spinlock_profiling.cc:243] Waited 3.32 ms on lock 0x7b180036c160. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc5522 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:10:46.858256 (+ 23093us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x7b180036c160. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:10:47.292081 (+433825us) raft_consensus.cc:1685] finished
0709 13:10:47.292126 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:47.292301 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27140,"prepare.queue_time_us":62820,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":79,"raft.queue_time_us":59907,"raft.run_cpu_time_us":538,"raft.run_wall_time_us":34231,"spinlock_wait_cycles":15454720,"thread_start_us":578355,"threads_started":2,"wal-append.queue_time_us":667578}
I20260709 13:10:47.325663 32657 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:47.319900 32601 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:47.327061 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44954 (request call id 1273) took 1327 ms. Trace:
I20260709 13:10:47.261435 32633 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:47.327455 25243 rpcz_store.cc:276] 0709 13:10:45.999318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:45.999436 (+   118us) service_pool.cc:224] Handling call
0709 13:10:45.999753 (+   317us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:46.000063 (+   310us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:46.000089 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:46.019659 (+ 19570us) log.cc:844] Serialized 26 byte log entry
0709 13:10:46.084863 (+ 65204us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:10:46.084898 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:46.084959 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:46.876600 (+791641us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:47.326777 (+450177us) raft_consensus.cc:1685] finished
0709 13:10:47.326799 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:47.327003 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5711,"raft.queue_time_us":173620,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":22892,"spinlock_wait_cycles":38538624,"thread_start_us":27040,"threads_started":1,"wal-append.queue_time_us":208155}
I20260709 13:10:47.328459 32633 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.330590 32633 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:47.332088 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:47.331938 32333 raft_consensus.cc:359] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [term 17 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.294205 32670 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:47.295208 32624 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:47.296070 32652 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.217s	user 0.001s	sys 0.000s
I20260709 13:10:47.296387 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:47.296720 25940 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
W20260709 13:10:47.296980 32662 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
W20260709 13:10:47.297824 32658 log.cc:927] Time spent T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.172s	user 0.000s	sys 0.001s
I20260709 13:10:47.262466 32602 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.039s
I20260709 13:10:47.338795 32625 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:47.338630 32649 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:10:47.334898 32670 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:47.301838 32660 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.194s	user 0.000s	sys 0.000s
I20260709 13:10:47.348196 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:47.315253 32661 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:47.317315 32547 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:47.262315 32654 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:47.217298 32620 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:47.183553 32270 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.291103 32551 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:47.345026 25287 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.67:34661 timed out after 2.211s (SENT) [suppressed 26 similar messages]
I20260709 13:10:47.348454 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:47.349005 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:47.377840 25241 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 13:10:47.306583 32353 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:47.317062 32032 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:47.454062 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:10:47.443912 32547 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.459368 32547 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:47.465417 32508 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:47.467604 31850 raft_consensus.cc:3055] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:47.468200 31850 raft_consensus.cc:740] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:47.471575 32270 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:47.260000 32636 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:47.206779 32651 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:47.421598 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:47.192328 32648 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:10:47.259516 32458 consensus_meta.cc:287] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 1.155s	user 0.012s	sys 0.001s
I20260709 13:10:47.464756 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:47.400678 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 219980us
I20260709 13:10:47.474339 25393 raft_consensus.cc:1240] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:47.477766 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:47.258327 32363 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:47.487412 25285 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:47.489796 25937 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 18: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:10:47.489845 25212 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:47.485503 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:10:47.491334 25389 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 22.
I20260709 13:10:47.410358 32333 raft_consensus.cc:740] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEARNER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Initialized, Role: LEARNER
I20260709 13:10:47.216926 32451 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:47.490322 32500 consensus_meta.cc:287] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.878s	user 0.004s	sys 0.007s
I20260709 13:10:47.245651 32464 consensus_queue.cc:1059] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.246s
I20260709 13:10:47.199096 32668 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.510567 25212 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:47.512665 25240 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 20. Current term is 21. Ops: []
W20260709 13:10:47.538926 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:47.539654 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:47.540323 25240 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:47.494421 32283 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:47.339418 32625 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.512722 32661 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:47.544064 32681 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:47.544534 32681 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:47.544504 32661 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.545116 32681 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:47.546959 32661 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:47.547557 32661 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:47.547900 32661 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:47.339418 32624 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.548264 32661 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:47.516788 32387 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:47.518160 32610 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 1.414s	user 0.003s	sys 0.000s
I20260709 13:10:47.518987 32599 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:47.523146 25395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60930 (request call id 3093) took 1640 ms. Trace:
I20260709 13:10:47.421105 32620 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.529376 32603 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:10:47.444028 32654 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.490845 32651 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.501068 32643 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:47.514149 32667 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:47.535318 32629 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:47.501389 32656 log.cc:927] Time spent T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.399s	user 0.001s	sys 0.000s
I20260709 13:10:47.537317 25940 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
W20260709 13:10:47.542429 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 55142.7us
I20260709 13:10:47.539398 32501 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:47.576333 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 231326us
I20260709 13:10:47.494944 31850 consensus_queue.cc:260] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.543490 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:47.543512 32283 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.550529 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:47.513788 32500 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.553294 32670 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:47.555662 32657 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.562441 25395 rpcz_store.cc:276] 0709 13:10:45.882853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:45.883349 (+   496us) service_pool.cc:224] Handling call
0709 13:10:45.929020 (+ 45671us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:46.794330 (+865310us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:47.521993 (+727663us) spinlock_profiling.cc:243] Waited 864 ms on lock 0x7b5400322dd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:47.522647 (+   654us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1986604288}
I20260709 13:10:47.585711 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:47.550457 32333 consensus_queue.cc:260] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:10:47.531054 32551 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:47.509152 32649 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:47.549428 32387 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.578245 25243 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:47.580150 25240 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:47.600945 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:47.601521 25240 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:47.563491 32624 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:47.586395 25238 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:47.565094 32538 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:47.586428 25318 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 21.
W20260709 13:10:47.581291 32494 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.006s	sys 0.025s
I20260709 13:10:47.596310 25287 leader_election.cc:400] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20.
I20260709 13:10:47.604985 32333 ts_tablet_manager.cc:1434] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59: Time spent starting tablet: real 0.604s	user 0.012s	sys 0.000s
I20260709 13:10:47.606545 32629 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:47.577852 32363 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:47.609901 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:47.578374 32603 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:47.586673 32672 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:47.580463 32501 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.620466 25246 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: [16.15-16.15]
I20260709 13:10:47.580072 32605 consensus_queue.cc:1059] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20260709 13:10:47.579177 32667 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:47.620042 32494 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.006s	sys 0.025s
I20260709 13:10:47.655790 25313 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 20.
I20260709 13:10:47.667155 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:47.620600 25287 leader_election.cc:403] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 13:10:47.664865 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:47.673724 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 77556.5us
I20260709 13:10:47.677426 25391 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:10:47.678802 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:47.612037 32683 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:47.578860 32599 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.608958 32551 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:47.714084 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:47.702118 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 111902us
I20260709 13:10:47.673467 25392 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 22.
I20260709 13:10:47.586448 32643 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.701171 25240 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
W20260709 13:10:47.626214 32333 ts_tablet_manager.cc:1467] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59: Tablet startup took 648ms
I20260709 13:10:47.663808 32628 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:47.608583 32649 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.702098 25388 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 13. (term mismatch)
I20260709 13:10:47.699229 32178 raft_consensus.cc:3060] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:47.725136 25359 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:47.726125 25911 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:47.724187 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:47.733274 25359 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:47.663389 32363 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.730968 25388 pending_rounds.cc:85] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 12
I20260709 13:10:47.751485 25388 pending_rounds.cc:107] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 17.13
I20260709 13:10:47.751811 25388 raft_consensus.cc:2889] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:47.668112 32625 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:47.625828 32538 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.742388 32696 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:47.662974 32603 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.745260 32547 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:47.745820 32566 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:47.709573 32683 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.747913 32458 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:47.748519 32650 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:47.749009 32693 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:47.749806 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:47.731652 32565 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 13:10:47.745172 32490 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:47.800670 25240 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:47.685514 32654 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:47.756565 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:47.704440 32629 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.036s
I20260709 13:10:47.744433 32613 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:47.766229 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:47.807524 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:47.808950 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:47.806111 25359 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:47.764117 32547 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:47.777496 32451 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:47.793926 32682 log.cc:927] Time spent T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 13:10:47.784082 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:47.825277 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:47.825896 25238 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:47.826762 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:47.827454 25238 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:47.828418 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:47.750926 25243 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:47.735306 32668 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:47.662231 32672 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.780129 32458 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:47.785746 32693 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.821823 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:47.763936 32696 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.823114 32677 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:47.823200 32685 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
W20260709 13:10:47.822970 25128 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f75b60146e7d4b9b9954b4fb2c414679 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica f646dfd532974dceadfe00005d2bbf59 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 18 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:47.830752 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:47.834937 25359 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:47.834065 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:47.765067 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:47.833066 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:47.797792 32494 raft_consensus.cc:3055] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:47.841919 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:47.875929 25939 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 14.
I20260709 13:10:47.786141 32268 raft_consensus.cc:3055] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:47.876601 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:47.877240 25388 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:47.840348 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:47.791070 32649 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:47.776763 32681 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.880967 25244 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:47.861846 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:47.754866 32628 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.849349 32458 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:47.873852 25321 raft_consensus.cc:3060] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:47.870682 25395 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
W20260709 13:10:47.867919 25128 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 45146.4us
I20260709 13:10:47.844570 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:47.826333 32547 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:47.883949 25312 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:10:47.852550 32620 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:47.887488 25287 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:47.813728 32490 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.864709 32651 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:47.891991 25245 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:10:47.908684 25908 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:47.910718 25318 raft_consensus.cc:2393] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 21: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:47.875319 32500 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:47.824460 32613 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.905036 32114 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:47.875808 32244 raft_consensus.cc:2804] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 13:10:47.889757 32679 log.cc:927] Time spent T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.258s	user 0.001s	sys 0.001s
I20260709 13:10:47.879818 32268 raft_consensus.cc:740] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:47.875633 32494 raft_consensus.cc:740] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:47.883181 32546 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:47.916230 32641 raft_consensus.cc:2804] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:10:47.937907 32641 raft_consensus.cc:697] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:47.938113 32546 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.938315 32494 consensus_queue.cc:260] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:47.789394 32650 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.939518 32494 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:47.928247 25245 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:10:47.906733 32657 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:47.932708 32363 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:47.916241 32675 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:47.936451 32613 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:47.937014 32695 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 13:10:47.946238 25321 raft_consensus.cc:2468] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:47.947319 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:47.947691 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:47.883801 32677 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:47.965227 32672 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:47.903108 25388 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:47.914170 32620 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:47.958045 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:47.959113 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:47.959501 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:47.959714 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:47.960158 32244 raft_consensus.cc:697] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:47.965227 25316 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:47.940541 32599 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:47.950793 32667 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:47.976090 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:48.045327 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 135159us
I20260709 13:10:47.977684 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.064678 25321 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
I20260709 13:10:48.068318 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.058974 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:48.075497 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.076118 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:47.973906 32673 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:47.978737 32691 log.cc:927] Time spent T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
I20260709 13:10:48.082293 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.083587 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.085516 32703 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:48.086057 32703 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:48.086463 32703 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:47.980180 32693 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:47.988400 32551 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:48.105234 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:48.105679 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:48.119853 25242 raft_consensus.cc:3060] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:48.105271 25393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 2545) took 1022 ms. Trace:
I20260709 13:10:48.109902 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:48.111469 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.108924 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.066351 32643 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:48.120550 25393 rpcz_store.cc:276] 0709 13:10:47.082441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:47.083572 (+  1131us) service_pool.cc:224] Handling call
0709 13:10:47.083848 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:47.492601 (+408753us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:48.104521 (+611920us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b54001ec654. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:48.105208 (+   687us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":897433984}
I20260709 13:10:47.978709 32675 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:47.939138 32641 consensus_queue.cc:237] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.117697 25321 raft_consensus.cc:1240] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 13:10:48.108518 25285 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:48.119035 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.119920 25391 raft_consensus.cc:2368] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17.
I20260709 13:10:48.113153 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:47.980688 32460 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:48.124639 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.081274 32668 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:48.130551 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.131368 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.106132 32670 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:48.111323 32546 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:48.117559 32599 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:48.126658 32648 consensus_meta.cc:287] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.630s	user 0.011s	sys 0.000s
I20260709 13:10:48.068488 32620 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.130091 32603 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:48.147018 25319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 1305) took 1212 ms. Trace:
W20260709 13:10:48.195179 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:48.141245 25245 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 12. (term mismatch)
I20260709 13:10:48.196838 25245 pending_rounds.cc:85] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 11
I20260709 13:10:48.197119 25245 pending_rounds.cc:107] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 17.12
I20260709 13:10:48.197505 25245 raft_consensus.cc:2889] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:48.126196 32675 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:48.192828 32621 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:48.138785 32244 consensus_queue.cc:237] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.139819 32460 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:48.205498 32460 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:48.205961 32460 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:48.208315 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:48.191453 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:47.978622 32268 consensus_queue.cc:260] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.118414 32032 consensus_queue.cc:1059] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 13:10:48.115077 32673 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:47.725000 32333 ts_tablet_manager.cc:1469] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59: Trace:
0709 13:10:39.144712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:39.145938 (+  1226us) service_pool.cc:224] Handling call
0709 13:10:39.792874 (+646936us) ts_tablet_manager.cc:934] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59: Initiating tablet copy from peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
0709 13:10:42.405811 (+2612937us) tablet_copy_client.cc:368] Tablet copy session begun
0709 13:10:43.024021 (+618210us) tablet_metadata.cc:665] Metadata flushed
0709 13:10:43.027828 (+  3807us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 13:10:43.546511 (+518683us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 13:10:43.546869 (+   358us) tablet_replica.cc:206] Creating consensus instance
0709 13:10:43.547252 (+   383us) ts_tablet_manager.cc:989] Replica registered.
0709 13:10:43.547372 (+   120us) inbound_call.cc:177] Queueing success response
0709 13:10:44.349116 (+801744us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 13:10:47.023308 (+2674192us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 13:10:47.076430 (+ 53122us) tablet_replica.cc:261] Starting instrumentation
0709 13:10:47.077026 (+   596us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":53762,"mutex_wait_us":32690,"raft.queue_time_us":5704,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":18037,"tablet-copy.queue_time_us":645737,"tablet-download-pool.queue_time_us":158,"tablet-download-pool.run_cpu_time_us":1216,"tablet-download-pool.run_wall_time_us":460241,"thread_start_us":1268605,"threads_started":2}
I20260709 13:10:48.192828 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:48.208653 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:48.161258 25242 raft_consensus.cc:2468] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:48.195034 25319 rpcz_store.cc:276] 0709 13:10:46.923887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:46.961608 (+ 37721us) service_pool.cc:224] Handling call
0709 13:10:46.962101 (+   493us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:47.133119 (+171018us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:47.141223 (+  8104us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:47.142422 (+  1199us) log.cc:844] Serialized 26 byte log entry
0709 13:10:47.481793 (+339371us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:10:47.500101 (+ 18308us) log.cc:844] Serialized 14 byte log entry
0709 13:10:47.539260 (+ 39159us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:47.539351 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:48.135726 (+596375us) raft_consensus.cc:1685] finished
0709 13:10:48.135749 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:48.135875 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":92158,"prepare.queue_time_us":17026,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":133932,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":2175,"thread_start_us":425444,"threads_started":2,"wal-append.queue_time_us":420441}
W20260709 13:10:48.224691 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 94018.5us
I20260709 13:10:48.226164 25285 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:48.229174 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.245021 25358 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:48.190701 32670 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:48.247107 25358 consensus_peers.cc:597] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211): Couldn't send request to peer f646dfd532974dceadfe00005d2bbf59. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:10:48.248908 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.193756 32648 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.253162 25390 raft_consensus.cc:1240] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 20. Current term is 21. Ops: []
I20260709 13:10:48.248867 32558 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:48.194368 32599 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.226982 32681 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:48.264060 32032 consensus_queue.cc:1243] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer 46de6919f85341c6a9b574dff112e35a log is divergent from this leader: its last log entry 18.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:10:48.265877 25285 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:48.234675 32490 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:48.227831 32702 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:10:48.147177 32668 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.271157 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.272084 25391 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:48.254541 32706 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:48.264866 32538 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:48.264813 32650 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:48.262686 32628 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:48.288080 32673 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:48.283107 25285 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:48.276261 32681 raft_consensus.cc:3055] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:10:48.309140 32701 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260709 13:10:48.309304 32681 raft_consensus.cc:740] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:48.310292 32681 consensus_queue.cc:260] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:48.311309 32711 log.cc:927] Time spent T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 13:10:48.266510 32683 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:48.286187 32661 consensus_meta.cc:287] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.722s	user 0.004s	sys 0.007s
I20260709 13:10:48.278265 32032 raft_consensus.cc:2804] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:48.313608 32661 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.315739 32661 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:48.315725 32460 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.311477 32681 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:48.279058 32702 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:48.319154 32702 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:10:48.305500 32178 consensus_meta.cc:287] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.567s	user 0.011s	sys 0.000s
I20260709 13:10:48.319274 32460 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:48.339418 32704 log.cc:927] Time spent T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.215s	user 0.000s	sys 0.001s
I20260709 13:10:48.331565 32178 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 19. Current term is 20.
I20260709 13:10:48.331837 32673 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 13:10:48.349176 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 61539.7us
I20260709 13:10:48.329689 32032 raft_consensus.cc:697] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:48.299496 32706 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.332653 32648 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:48.340464 32671 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:48.361800 25285 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:48.367300 32032 consensus_queue.cc:237] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.368359 32668 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:48.368913 32668 raft_consensus.cc:2804] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:48.369375 32668 raft_consensus.cc:697] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:48.370152 32668 consensus_queue.cc:237] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:48.372686 32663 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 1.184s	user 0.003s	sys 0.000s
I20260709 13:10:48.376439 25320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 2999) took 1585 ms. Trace:
I20260709 13:10:48.377703 32719 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:48.377385 32547 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.385263 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:48.376871 25320 rpcz_store.cc:276] 0709 13:10:46.791502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:46.792061 (+   559us) service_pool.cc:224] Handling call
0709 13:10:46.792510 (+   449us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:10:46.795721 (+  3211us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:46.795751 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:10:46.803426 (+  7675us) log.cc:844] Serialized 52 byte log entry
0709 13:10:46.883712 (+ 80286us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:10:47.014317 (+130605us) log.cc:844] Serialized 14 byte log entry
0709 13:10:47.138569 (+124252us) log.cc:844] Serialized 14 byte log entry
0709 13:10:47.142316 (+  3747us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:47.142413 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:48.372117 (+1229704us) spinlock_profiling.cc:243] Waited 4.75 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:48.376199 (+  4082us) raft_consensus.cc:1685] finished
0709 13:10:48.376223 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:48.376376 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":285486,"prepare.queue_time_us":143896,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":95,"raft.queue_time_us":178602,"raft.run_cpu_time_us":495,"raft.run_wall_time_us":20301,"spinlock_wait_cycles":10924800,"thread_start_us":152248,"threads_started":2,"wal-append.queue_time_us":314212}
I20260709 13:10:48.380192 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:48.350946 32673 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:48.386771 25241 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:48.395423 25241 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 21.
I20260709 13:10:48.358788 32671 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:48.400791 25245 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 17: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:48.402781 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:48.403769 25284 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:48.404042 25241 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:48.404140 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:48.404979 25242 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:48.407529 25285 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:48.409821 25386 raft_consensus.cc:2393] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 21: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:48.411082 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:48.418545 25237 raft_consensus.cc:2410] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 19 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
W20260709 13:10:48.411295 32708 log.cc:927] Time spent T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 13:10:48.385699 32620 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:48.401087 32673 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:48.423869 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.406003 32671 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:48.421440 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.434190 25320 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 20.
I20260709 13:10:48.435293 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.424641 25319 raft_consensus.cc:2393] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 21: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:48.425999 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.408385 32703 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.427127 32717 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:48.427326 32599 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:48.401732 32458 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.355970 32714 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.091s
I20260709 13:10:48.451097 32458 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:48.427640 25318 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:10:48.452941 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.453629 25318 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:48.454603 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.454829 32696 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:48.455201 25318 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 19.
I20260709 13:10:48.452868 25209 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:48.427695 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:48.424124 25285 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:48.431775 32501 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:48.458868 25317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 1302) took 1536 ms. Trace:
I20260709 13:10:48.438081 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.424772 25316 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:10:48.460413 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.460541 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.461388 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:48.461421 25321 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:48.440665 25313 raft_consensus.cc:2468] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:48.442389 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.462237 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:48.459282 25317 rpcz_store.cc:276] 0709 13:10:46.922922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:46.923050 (+   128us) service_pool.cc:224] Handling call
0709 13:10:46.983379 (+ 60329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:47.395360 (+411981us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:47.395399 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:47.410383 (+ 14984us) log.cc:844] Serialized 26 byte log entry
0709 13:10:47.708437 (+298054us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:10:47.708478 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:48.437257 (+728779us) spinlock_profiling.cc:243] Waited 228 ms on lock 0x7b5400276d54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:48.437304 (+    47us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:48.437347 (+    43us) raft_consensus.cc:1685] finished
0709 13:10:48.437360 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:48.458803 (+ 21443us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":145942,"raft.queue_time_us":197559,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":22697,"spinlock_wait_cycles":523841280,"thread_start_us":248184,"threads_started":1,"wal-append.queue_time_us":331300}
I20260709 13:10:48.443430 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:48.378259 32719 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.445813 32460 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:10:48.444156 32032 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:48.494838 32703 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:48.494983 32032 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.495944 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.497109 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.497681 25938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 1391) took 1614 ms. Trace:
I20260709 13:10:48.497109 32032 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:48.498265 25938 rpcz_store.cc:276] 0709 13:10:46.883025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:46.888567 (+  5542us) service_pool.cc:224] Handling call
0709 13:10:48.497422 (+1608855us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b540062a2d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:48.497583 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":512985856}
I20260709 13:10:48.428157 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.500860 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.503242 32460 raft_consensus.cc:697] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:48.465469 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.504226 32460 consensus_queue.cc:237] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.456070 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.469266 25940 raft_consensus.cc:1240] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 20. Current term is 21. Ops: []
I20260709 13:10:48.467962 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.464408 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.485797 32547 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:48.509974 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:48.486191 32675 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.493320 32599 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:10:48.497790 25317 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 21.
I20260709 13:10:48.468957 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.510759 25318 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:48.509225 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.514962 25319 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:48.515861 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:48.463129 32714 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:48.515594 32599 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:48.517670 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.518529 25315 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
W20260709 13:10:48.508381 32494 consensus_meta.cc:287] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.530s	user 0.008s	sys 0.003s
I20260709 13:10:48.519843 25209 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:48.526208 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:48.530280 25314 raft_consensus.cc:2368] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 19. Current term is 20.
I20260709 13:10:48.532994 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:48.533641 25237 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:48.535689 32661 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:48.536008 32706 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:48.520501 32599 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:10:48.522840 32594 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.412s	user 0.029s	sys 0.036s
I20260709 13:10:48.535518 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:48.538902 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:48.541913 25320 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:48.538420 25209 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:48.521327 32669 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:48.521514 32714 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:48.483103 32702 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.544109 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:10:48.527524 32696 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:48.546334 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:48.516626 32671 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.533183 32380 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.016s
I20260709 13:10:48.559427 32675 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:48.541553 32594 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.436s	user 0.029s	sys 0.037s
I20260709 13:10:48.561324 32671 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:48.565779 32673 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.568361 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.579285 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.568295 25212 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:48.549701 32714 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:48.569233 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.579608 25209 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:48.581477 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:48.569416 32603 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:48.583009 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.570215 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:48.583379 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.570736 32702 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:48.570681 32490 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:48.549767 32669 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:48.560081 32724 log.cc:927] Time spent T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
I20260709 13:10:48.577934 25237 raft_consensus.cc:2468] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:48.537643 32641 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.020s
I20260709 13:10:48.549028 32494 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:10:48.580260 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:48.584753 25212 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:48.553344 32696 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:48.542121 32661 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.035s
I20260709 13:10:48.589340 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:48.549503 32683 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20260709 13:10:48.591282 25359 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:48.603287 32359 raft_consensus.cc:2804] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:48.603788 32359 raft_consensus.cc:697] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:48.604573 32359 consensus_queue.cc:237] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.592329 25318 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 20. Current term is 21. Ops: []
I20260709 13:10:48.595409 32693 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:48.596022 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:48.608114 25237 raft_consensus.cc:2393] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 21: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:48.610911 32696 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:48.611055 25358 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:48.612867 32723 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:48.613291 32723 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:48.613639 32723 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:48.615111 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:48.616113 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 2548) took 1532 ms. Trace:
I20260709 13:10:48.616497 25389 rpcz_store.cc:276] 0709 13:10:47.083222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:47.083393 (+   171us) service_pool.cc:224] Handling call
0709 13:10:48.615972 (+1532579us) spinlock_profiling.cc:243] Waited 359 ms on lock 0x7b54000dded4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:48.616061 (+    89us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":826074368}
I20260709 13:10:48.621655 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.621942 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.622789 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:48.623098 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.624328 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.624661 32283 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:48.624907 25390 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:48.626463 25394 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:60946
I20260709 13:10:48.626646 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 3019) took 1037 ms. Trace:
I20260709 13:10:48.618338 32558 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:10:48.627136 32244 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:48.626971 25240 rpcz_store.cc:276] 0709 13:10:47.588947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:47.723894 (+134947us) service_pool.cc:224] Handling call
0709 13:10:48.626479 (+902585us) spinlock_profiling.cc:243] Waited 60.7 ms on lock 0x7b5400422554. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:48.626583 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":139601280}
I20260709 13:10:48.627882 32244 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.630348 32558 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:48.587312 32458 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:48.630919 32558 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.631184 32458 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.586015 32551 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:10:48.632534 32551 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:48.632900 32551 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:48.633296 25212 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:48.634233 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.635210 32693 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:48.585561 32637 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:48.636312 32637 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.636749 25212 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:48.587674 32603 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.638155 32693 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:48.639531 32603 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:48.586951 32667 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:48.639791 25212 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:48.640120 32723 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.640561 32667 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:48.587867 32490 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.641705 25212 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:48.643741 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.644484 25392 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 22.
I20260709 13:10:48.616330 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:48.650130 32114 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:10:48.650484 25358 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:48.652064 32284 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:48.653172 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:48.653256 25243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 3015) took 1078 ms. Trace:
I20260709 13:10:48.653683 25243 rpcz_store.cc:276] 0709 13:10:47.574729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:47.934475 (+359746us) service_pool.cc:224] Handling call
0709 13:10:47.934759 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:47.949795 (+ 15036us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:47.949814 (+    19us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:47.949841 (+    27us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:10:47.949861 (+    20us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:48.652985 (+703124us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b54003a8ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:48.653027 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:48.653204 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34259328}
I20260709 13:10:48.654789 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:48.655125 25392 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 20
I20260709 13:10:48.627008 25394 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:10:48.648870 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:48.656059 25394 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:48.659246 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:48.661665 25212 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:48.597798 31850 raft_consensus.cc:3060] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:48.711042 25392 raft_consensus.cc:2468] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:48.683495 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
W20260709 13:10:48.669284 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:48.691174 25287 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:48.714185 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 52643.8us
W20260709 13:10:48.708389 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 51318.3us
I20260709 13:10:48.684860 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:48.708293 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 51691.1us
I20260709 13:10:48.707834 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:48.803296 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 112217us
I20260709 13:10:48.807924 25388 raft_consensus.cc:2393] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 21: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:48.718575 32353 raft_consensus.cc:993] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: : Instructing follower a2a77ecf2c0941618b5bc79d4c04db61 to start an election
I20260709 13:10:48.592931 32494 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:48.818251 25212 leader_election.cc:400] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17.
I20260709 13:10:48.819145 25212 leader_election.cc:403] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 13:10:48.806360 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:48.810437 25287 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:48.670782 32677 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:48.650512 32114 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:48.670820 32669 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:48.676894 32670 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:48.828635 32641 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:48.837304 32641 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:48.837684 32641 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:48.720050 32661 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:48.828789 32675 raft_consensus.cc:1081] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Signalling peer a2a77ecf2c0941618b5bc79d4c04db61 to start an election
I20260709 13:10:48.667279 32380 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:48.718063 32483 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:10:48.844408 32483 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:48.679387 32718 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.292s	user 0.002s	sys 0.000s
I20260709 13:10:48.848920 32483 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:48.668865 32667 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:48.671875 32458 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:48.831797 25287 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:48.858490 32726 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:10:48.858781 25287 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:48.859083 32726 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:48.859619 32726 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:48.860052 31386 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:10:48.860548 31386 raft_consensus.cc:697] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 20 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:48.677842 32490 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:48.861330 31386 consensus_queue.cc:237] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.861690 32719 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:48.835863 25212 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:48.827728 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:48.863363 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:48.864377 32458 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:10:48.812860 32550 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:48.864989 32458 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:10:48.865370 32550 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.867345 32550 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:48.836621 32114 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:48.876071 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:48.877468 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:48.872956 32641 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.878741 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:48.879384 25243 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:48.882169 32483 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.882736 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:48.884135 32497 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 16. Current term is 17.
W20260709 13:10:48.884712 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 48912.9us
I20260709 13:10:48.884833 32497 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:10:48.885465 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:48.886591 25212 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:48.887827 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:48.888633 32683 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:48.813243 32712 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:48.889343 32683 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:48.889586 32712 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:48.890091 32712 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:48.890893 32661 consensus_queue.cc:1059] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:48.891973 32490 raft_consensus.cc:3055] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:10:48.892320 32490 raft_consensus.cc:740] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:48.893213 32490 consensus_queue.cc:260] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.894654 25212 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:48.888134 32599 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.675904 32637 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:48.896345 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:48.673007 32387 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:48.902760 25212 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:48.727851 32284 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:48.903668 25394 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.879985 32641 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:48.918639 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.920370 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.921142 25386 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 21.
I20260709 13:10:48.826436 32494 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:48.923027 25246 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:10:48.872566 32714 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.926005 25239 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 14. Current term is 15. Ops: []
I20260709 13:10:48.890087 32683 raft_consensus.cc:2804] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:48.897677 32669 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:48.931469 25244 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 3023) took 1299 ms (client timeout 1683 ms). Trace:
I20260709 13:10:48.908242 32673 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:48.913293 32497 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:10:48.932524 25244 rpcz_store.cc:269] 0709 13:10:47.631713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:47.766003 (+134290us) service_pool.cc:224] Handling call
0709 13:10:48.931255 (+1165252us) spinlock_profiling.cc:243] Waited 74.1 ms on lock 0x7b54000a00d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:48.931411 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":170437376}
I20260709 13:10:48.937706 25212 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:48.939167 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:48.921926 32603 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:48.940148 25392 raft_consensus.cc:2393] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 19: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:48.930444 32683 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:48.932253 32667 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:48.947762 25319 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:48.922061 32284 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.940421 32497 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:48.970885 25390 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260709 13:10:48.943696 32353 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:48.976666 32497 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.977170 32551 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:48.978252 32284 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:48.979009 32497 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:48.979445 32551 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:48.974375 32667 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.980420 25240 raft_consensus.cc:2393] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 23: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:48.937587 32669 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.957417 32683 raft_consensus.cc:3060] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:48.987001 32669 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:48.931785 32114 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:48.954151 32603 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:10:49.022770 32603 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:49.023281 32603 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:48.974462 32483 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:49.021720 32696 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:49.029749 32599 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:49.031489 32712 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.032709 32667 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:49.028738 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:49.035133 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:49.036530 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:49.053038 25312 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:49.071381 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:49.076349 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 138730us
I20260709 13:10:49.086122 32114 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:49.086808 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:49.085954 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:49.087996 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:49.088340 25392 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:49.088511 25315 raft_consensus.cc:3055] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:49.095093 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:49.088914 25315 raft_consensus.cc:740] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:49.110575 25908 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:49.112748 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:49.120038 25312 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 23.
I20260709 13:10:49.121663 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:49.122591 25312 raft_consensus.cc:2393] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 23: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:49.123204 25316 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 19: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:49.134404 25392 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 23.
I20260709 13:10:49.140867 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:49.141609 25316 raft_consensus.cc:3060] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Advancing to term 23
I20260709 13:10:49.141777 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:49.142807 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:49.140866 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:49.114226 25315 consensus_queue.cc:260] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:49.144415 25315 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:49.106781 32508 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:49.154043 32490 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:49.109293 32387 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.049s
I20260709 13:10:49.159464 32511 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:10:49.145365 32677 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:49.145210 32494 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:49.164031 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:49.181849 32494 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:49.145012 25405 tablet_copy_service.cc:342] P a2a77ecf2c0941618b5bc79d4c04db61: Request end of tablet copy session f646dfd532974dceadfe00005d2bbf59-b62f4877af9742298b1e149789ca3090 received from {username='slave'} at 127.0.0.1:41830
I20260709 13:10:49.182471 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:49.165738 25321 raft_consensus.cc:1275] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 13:10:49.183795 25394 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:49.168306 25940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 1424) took 1060 ms. Trace:
I20260709 13:10:49.173825 25239 raft_consensus.cc:1275] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 13:10:49.183429 25315 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:49.162456 25320 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:49.184806 25237 raft_consensus.cc:1240] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 13:10:49.175096 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:49.187686 25244 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:49.187770 25318 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:10:49.178557 25387 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60946 (request call id 2563) took 1345 ms. Trace:
I20260709 13:10:49.189320 25387 rpcz_store.cc:276] 0709 13:10:47.833247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:47.833357 (+   110us) service_pool.cc:224] Handling call
0709 13:10:49.178366 (+1345009us) spinlock_profiling.cc:243] Waited 518 ms on lock 0x7b5400481a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:49.178500 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1190579456}
I20260709 13:10:49.184337 25940 rpcz_store.cc:276] 0709 13:10:48.107670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:48.110551 (+  2881us) service_pool.cc:224] Handling call
0709 13:10:48.126809 (+ 16258us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:48.513445 (+386636us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:49.168001 (+654556us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b5400703854. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:49.168215 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":787015040}
I20260709 13:10:49.178845 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:49.193220 25394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:60946 (request call id 2559) took 1836 ms. Trace:
I20260709 13:10:49.192951 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:49.193670 25394 rpcz_store.cc:276] 0709 13:10:47.348052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:48.626335 (+1278283us) service_pool.cc:224] Handling call
0709 13:10:49.183695 (+557360us) spinlock_profiling.cc:243] Waited 7.54 ms on lock 0x7b4c00384440. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2974f7ea20 00007f2974f8fa29 00007f2974f9013f 00007f2974f542ab 00007f2974f81f3c 00007f2974f823fe 00007f297520f88b 00007f2976c0e495 00007f2976c5f30f 00007f2976ca9a55 00007f297bc79cd0
0709 13:10:49.184340 (+   645us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17339776}
I20260709 13:10:49.207234 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:49.175244 32677 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:49.176254 32558 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:49.210150 25387 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 23: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:10:49.210713 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:49.210856 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 214412us
I20260709 13:10:49.209880 25316 raft_consensus.cc:2468] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 23.
I20260709 13:10:49.211592 25359 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:49.174437 32696 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:49.212280 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:49.213457 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:49.213969 25392 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 21.
I20260709 13:10:49.148888 32665 consensus_queue.cc:1059] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 16.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 13:10:49.201040 32408 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:49.217571 32243 consensus_queue.cc:1059] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.002s
I20260709 13:10:49.220162 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:49.202960 25405 tablet_copy_service.cc:434] P a2a77ecf2c0941618b5bc79d4c04db61: ending tablet copy session f646dfd532974dceadfe00005d2bbf59-b62f4877af9742298b1e149789ca3090 on tablet b62f4877af9742298b1e149789ca3090 with peer f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:49.227094 25244 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:49.199502 32359 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:49.237545 25359 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:49.179764 32683 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:49.213013 32677 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:49.189685 32508 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.245440 32494 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:49.245863 32243 raft_consensus.cc:3055] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:10:49.239322 32408 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.249675 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:49.210355 32299 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:49.255859 25209 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:10:49.245016 32734 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 13:10:49.209188 32726 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.236286 32274 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:49.265901 25316 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 23.
I20260709 13:10:49.241995 32641 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:10:49.268061 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 122038us
I20260709 13:10:49.273794 25944 raft_consensus.cc:3060] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [term 17 LEARNER]: Advancing to term 18
I20260709 13:10:49.256886 32243 raft_consensus.cc:740] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:10:49.280012 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 139347us
I20260709 13:10:49.280206 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:49.259615 32645 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:49.282848 25359 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:49.243166 32359 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.299170 32408 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:49.299675 32633 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:49.260524 32494 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.301445 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:49.315690 25392 raft_consensus.cc:2368] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 20. Current term is 21.
I20260709 13:10:49.316172 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:49.302734 32243 consensus_queue.cc:260] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:49.265945 32737 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
I20260709 13:10:49.305807 32683 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:49.306566 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:49.265148 32133 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 13:10:49.268288 31898 consensus_queue.cc:1059] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.040s
I20260709 13:10:49.274945 32274 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.267841 32299 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.269814 32696 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:10:49.312709 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:49.283541 32645 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:49.268198 32603 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.320910 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:49.325661 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:49.322705 25246 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:10:49.322878 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:49.322878 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:49.325169 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:49.326946 25392 raft_consensus.cc:2368] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 22. Current term is 23.
I20260709 13:10:49.326810 25244 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:49.326761 32645 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:49.331435 31898 raft_consensus.cc:3055] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:10:49.331853 31898 raft_consensus.cc:740] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:49.333361 32665 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:10:49.333757 32299 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:49.325227 32502 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:49.334724 32502 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.336416 25358 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:49.336535 32274 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:49.328778 32363 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:10:49.336723 32502 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:49.338040 32464 consensus_queue.cc:1048] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:10:49.338363 32624 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:10:49.339226 32508 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:49.340672 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:49.341272 25239 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 23.
I20260709 13:10:49.342319 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:49.316896 32633 raft_consensus.cc:697] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:49.337162 32696 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 23 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:49.341221 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:49.332580 31898 consensus_queue.cc:260] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.323700 32641 consensus_queue.cc:1059] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.023s
I20260709 13:10:49.346781 31898 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:10:49.346846 32243 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:49.348162 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:49.349248 32268 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:49.349826 32510 consensus_queue.cc:1175] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f646dfd532974dceadfe00005d2bbf59" responder_term: 14 status { last_received { term: 14 index: 13 } last_committed_idx: 13 last_received_current_leader { term: 14 index: 13 } }
I20260709 13:10:49.350847 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:49.353482 32633 consensus_queue.cc:237] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:49.356446 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 2986) took 1823 ms. Trace:
I20260709 13:10:49.356832 25238 rpcz_store.cc:276] 0709 13:10:47.533180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:47.886945 (+353765us) service_pool.cc:224] Handling call
0709 13:10:48.629086 (+742141us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:48.629712 (+   626us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:48.629751 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:48.629810 (+    59us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:48.629853 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:48.629903 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:49.356295 (+726392us) spinlock_profiling.cc:243] Waited 742 ms on lock 0x7b5400422550. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb18ad 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:49.356393 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1706055936}
I20260709 13:10:49.357697 32359 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:49.358758 32359 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:10:49.358697 32696 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:49.362322 32726 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:49.363332 25318 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 13:10:49.347054 32641 raft_consensus.cc:3055] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:10:49.363965 25359 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:49.368085 25237 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 13:10:49.379029 32625 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:10:49.380229 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:49.364527 32641 raft_consensus.cc:740] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:49.385247 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:49.386287 25237 raft_consensus.cc:2393] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 20: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:49.386682 25321 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 20.
I20260709 13:10:49.369393 32133 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.021s
I20260709 13:10:49.368907 25908 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:49.391084 32538 raft_consensus.cc:2804] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:10:49.381608 32408 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:49.401621 32500 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:49.402951 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:49.403929 25392 raft_consensus.cc:2393] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 20: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:49.391598 32538 raft_consensus.cc:697] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 23 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:49.411798 25287 leader_election.cc:400] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 22. Current term is 23.
I20260709 13:10:49.413255 25287 leader_election.cc:403] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 13:10:49.415931 32603 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:49.406025 32743 consensus_queue.cc:1059] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.011s
I20260709 13:10:49.421169 32743 raft_consensus.cc:3055] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:49.421609 32743 raft_consensus.cc:740] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:49.422619 32743 consensus_queue.cc:260] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.412751 32133 consensus_queue.cc:1059] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.005s
I20260709 13:10:49.417946 32677 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.419870 25392 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:60946
I20260709 13:10:49.425614 25392 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:10:49.425411 32641 consensus_queue.cc:260] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.402108 32408 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:49.426105 25392 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:49.426568 32408 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:49.426828 32641 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:49.405838 32500 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.429749 32538 consensus_queue.cc:237] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.432492 32743 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:49.436232 25390 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 21.
I20260709 13:10:49.430279 32500 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:49.437978 32671 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 22. Current term is 23.
I20260709 13:10:49.441049 32494 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:49.440951 25392 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:49.429773 32133 raft_consensus.cc:3055] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:49.449527 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:49.438458 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:49.437815 32677 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:49.485982 25285 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:49.492635 25285 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:49.493842 25395 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:49.490522 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: true dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:49.494879 25245 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:49.495831 25285 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:49.496589 25392 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:49.461247 32668 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.072s	user 0.007s	sys 0.041s
I20260709 13:10:49.498296 32363 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:10:49.501798 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 77899.3us
I20260709 13:10:49.504576 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:49.482350 32133 raft_consensus.cc:740] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
W20260709 13:10:49.507117 25314 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36876 (request call id 3035) took 1555 ms (client timeout 1966 ms). Trace:
W20260709 13:10:49.507580 25314 rpcz_store.cc:269] 0709 13:10:47.945980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:47.946135 (+   155us) service_pool.cc:224] Handling call
0709 13:10:49.500205 (+1554070us) spinlock_profiling.cc:243] Waited 561 ms on lock 0x7b5400443254. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:49.501699 (+  1494us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1289362816}
I20260709 13:10:49.508343 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:49.509414 25392 raft_consensus.cc:2393] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 19: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
W20260709 13:10:49.499761 32668 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.111s	user 0.008s	sys 0.041s
I20260709 13:10:49.497789 32671 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:49.514801 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:49.515915 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:49.516824 25358 leader_election.cc:400] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 19. Current term is 20.
I20260709 13:10:49.518127 25392 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 22.
I20260709 13:10:49.519240 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:49.519708 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:49.520403 25244 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 22: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:49.522192 25238 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 20. Current term is 21. Ops: []
I20260709 13:10:49.518119 25358 leader_election.cc:403] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:10:49.520262 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:49.525090 25392 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:49.529170 25285 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:49.500895 32500 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:49.524842 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:49.531592 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:49.542212 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:49.545547 31850 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:49.558015 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:49.557974 31850 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.547055 32665 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:49.559731 32665 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.560307 31850 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:49.562489 32665 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:49.563782 25245 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:49.565101 25359 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:49.566098 32244 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:49.566850 32244 raft_consensus.cc:2804] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:49.567277 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:49.567443 32244 raft_consensus.cc:697] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:49.547459 32387 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:49.548105 32538 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:49.549512 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:49.550046 25938 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 13:10:49.551043 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:49.552155 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:49.553284 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:49.554327 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:49.556788 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:49.572975 25937 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:49.570719 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:49.573372 25939 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 20.
I20260709 13:10:49.572914 25943 raft_consensus.cc:2468] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 21.
W20260709 13:10:49.575269 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 50564.9us
I20260709 13:10:49.574928 32538 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.575135 32133 consensus_queue.cc:260] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:49.579001 32133 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:49.579897 32538 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:49.580503 32592 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:10:49.580917 32592 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:49.581326 32592 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:49.555488 32710 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 19. Current term is 20.
I20260709 13:10:49.584673 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:49.587910 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:49.589322 25321 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:49.589104 25285 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:49.593168 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:49.595402 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:49.595480 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:49.597357 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:49.588507 25939 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 21.
I20260709 13:10:49.571979 32601 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:49.599813 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:49.602325 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:49.603169 25939 raft_consensus.cc:2410] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 13:10:49.737777 25393 raft_consensus.cc:1240] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 13:10:49.717900 32714 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:49.715778 32723 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:49.727870 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:49.737748 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:49.708173 25287 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:49.634688 32299 raft_consensus.cc:2955] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Committing config change with OpId 16.11: config changed from index -1 to 11, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
W20260709 13:10:49.733707 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 62398.1us
I20260709 13:10:49.791646 32668 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:10:49.568679 32244 consensus_queue.cc:237] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:49.681535 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:49.815657 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 213294us
I20260709 13:10:49.586694 32508 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.018s
I20260709 13:10:49.655898 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:49.692196 25390 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:10:49.610230 32645 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:49.742421 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:49.672503 25319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 3049) took 1114 ms. Trace:
I20260709 13:10:49.817368 25237 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:49.700555 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:49.685731 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:49.817430 25319 rpcz_store.cc:276] 0709 13:10:48.557571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:48.557697 (+   126us) service_pool.cc:224] Handling call
0709 13:10:48.573998 (+ 16301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:48.966847 (+392849us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:49.672184 (+705337us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b5400352354. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:49.672427 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":855224192}
W20260709 13:10:49.647729 25395 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60930 (request call id 3106) took 1686 ms (client timeout 1973 ms). Trace:
I20260709 13:10:49.732651 25911 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:49.818663 25395 rpcz_store.cc:269] 0709 13:10:47.961116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:48.118828 (+157712us) service_pool.cc:224] Handling call
0709 13:10:49.647504 (+1528676us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54003e4254. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:49.647650 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3140963200}
W20260709 13:10:49.818943 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 86407.6us
I20260709 13:10:49.692111 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:49.670770 25240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32994 (request call id 3006) took 1012 ms. Trace:
I20260709 13:10:49.576622 32387 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.695180 25246 raft_consensus.cc:1240] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:49.701534 32696 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:49.686793 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:49.821863 25936 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:49.746872 32363 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:49.738467 32665 consensus_queue.cc:1059] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 16.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:10:49.748339 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:49.663254 32408 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:49.729902 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 126777us
I20260709 13:10:49.651072 25388 raft_consensus.cc:1275] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:10:49.683990 25386 raft_consensus.cc:1275] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 13:10:49.691931 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:49.603259 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:49.671402 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:49.661041 32502 raft_consensus.cc:3055] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:49.751116 32675 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:49.826987 32714 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:49.827126 32502 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:49.827514 32714 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:49.828084 32502 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:49.829723 32714 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:49.720165 32712 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:49.696760 32511 raft_consensus.cc:3055] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:49.691082 25389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 2572) took 1071 ms. Trace:
I20260709 13:10:49.830420 32511 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:49.824419 32678 consensus_queue.cc:1059] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.007s
I20260709 13:10:49.831735 32678 consensus_queue.cc:1243] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer a2a77ecf2c0941618b5bc79d4c04db61 log is divergent from this leader: its last log entry 19.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 13:10:49.831336 32511 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 16.15, Last appended by leader: 15, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.830575 25389 rpcz_store.cc:276] 0709 13:10:48.619963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:48.623932 (+  3969us) service_pool.cc:224] Handling call
0709 13:10:48.624196 (+   264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:48.665993 (+ 41797us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:48.666050 (+    57us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:48.666109 (+    59us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:10:48.666154 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:49.690864 (+1024710us) spinlock_profiling.cc:243] Waited 41.5 ms on lock 0x7b54001e3a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:49.690910 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:49.691021 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":95550848}
I20260709 13:10:49.832194 32678 raft_consensus.cc:3055] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:49.832553 32511 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:49.834988 32387 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:49.835919 25911 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:49.787146 25387 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
W20260709 13:10:49.787978 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 79955.1us
I20260709 13:10:49.837719 32665 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:49.838972 32408 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:49.839587 32405 consensus_queue.cc:1175] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46de6919f85341c6a9b574dff112e35a" responder_term: 17 status { last_received { term: 15 index: 14 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 16. Current term is 17. Ops: [16.15-16.15]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:10:49.842264 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:49.845293 32603 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:49.847528 25312 raft_consensus.cc:1275] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:10:49.848539 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:49.849575 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:49.849929 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:49.850787 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:49.851163 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:49.851634 25389 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:49.851907 25387 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 17: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:49.721714 25392 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 22.
I20260709 13:10:49.816461 25285 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:49.818310 25942 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:49.855261 25285 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:49.856467 32712 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:49.819796 25242 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:49.819761 25240 rpcz_store.cc:276] 0709 13:10:48.658712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:48.659036 (+   324us) service_pool.cc:224] Handling call
0709 13:10:49.670488 (+1011452us) spinlock_profiling.cc:243] Waited 320 ms on lock 0x7b54003ab1d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:49.670696 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":736813952}
I20260709 13:10:49.857350 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:49.811273 32668 raft_consensus.cc:697] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
W20260709 13:10:49.823460 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 75212us
I20260709 13:10:49.824797 32696 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:49.848609 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:49.868960 25319 raft_consensus.cc:2393] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 23: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:49.868803 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:49.858402 25359 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:49.859140 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:49.861176 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:49.870024 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:49.870012 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:49.808102 25909 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:49.872334 25239 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
W20260709 13:10:49.779608 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 98512.9us
I20260709 13:10:49.862066 25942 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 22.
I20260709 13:10:49.865054 25212 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:49.874501 25212 leader_election.cc:400] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 20. Current term is 21.
I20260709 13:10:49.874773 25212 leader_election.cc:403] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 13:10:49.875164 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:49.865193 25237 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
I20260709 13:10:49.880522 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:49.872643 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 64674.8us
I20260709 13:10:49.888087 25909 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:49.906306 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:49.907586 25240 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:49.832589 32678 raft_consensus.cc:740] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:49.871951 32508 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:10:49.913407 25212 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:49.915540 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:49.876020 32550 raft_consensus.cc:2764] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 13:10:49.875958 32387 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:10:49.890687 32603 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 20. Current term is 21.
I20260709 13:10:49.887406 32133 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:10:49.922825 25942 raft_consensus.cc:2955] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEARNER]: Committing config change with OpId 16.11: config changed from index -1 to 11, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:10:49.919005 32536 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:49.915611 32665 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:49.919188 32668 consensus_queue.cc:237] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:49.921023 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:49.936371 25317 raft_consensus.cc:3055] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:49.936777 25317 raft_consensus.cc:740] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:49.916025 25909 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:49.878306 32494 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:49.884433 32714 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 13:10:49.864923 32511 consensus_queue.cc:1059] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.057s
I20260709 13:10:49.940092 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:49.944291 32696 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:49.883489 32592 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:49.917860 32508 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:49.960178 25319 raft_consensus.cc:3060] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:49.968822 25358 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:49.972604 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:49.975319 25358 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:49.960134 25317 consensus_queue.cc:260] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:49.976543 25317 raft_consensus.cc:3060] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:49.982524 25317 raft_consensus.cc:2468] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:49.922426 32538 consensus_queue.cc:1059] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.034s
I20260709 13:10:49.934309 32762 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 13:10:49.993297 25358 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:49.991176 25319 raft_consensus.cc:2468] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
I20260709 13:10:49.973816 32380 consensus_queue.cc:1048] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.025s
I20260709 13:10:49.970203 32508 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:10:49.967753 32536 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:49.983717 32494 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:10:50.023401 32133 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:49.999120 32547 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:49.981889 32408 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:10:50.046854 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 133513us
I20260709 13:10:50.064915 25212 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:50.065148 25359 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:50.052698 25358 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.097586 25386 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 20. Current term is 21. Ops: []
I20260709 13:10:49.973104 32460 consensus_queue.cc:1048] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.074s
I20260709 13:10:49.970504 32274 consensus_queue.cc:1048] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.073s
I20260709 13:10:50.098819 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:50.001748   301 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:50.105223   301 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:50.101887 25320 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 22. Current term is 23. Ops: []
W20260709 13:10:50.106320 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 53751.6us
I20260709 13:10:50.107172 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.102658 25312 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 22. Current term is 23. Ops: []
I20260709 13:10:50.046967 32494 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:50.111352 25242 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:10:50.113094 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.114970 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:50.115415 25358 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.063776 32603 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:50.109078   301 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:50.114255 32283 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:10:50.104395 32461 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:50.119186 25358 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.117830 32464 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:50.095467   303 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader election won for term 22
W20260709 13:10:50.110599 32763 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 13:10:50.104184 32133 raft_consensus.cc:2804] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:10:50.094583 32671 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.051s
I20260709 13:10:50.109880 32547 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:10:50.114154 32494 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:10:50.114569 32645 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:50.121325 32380 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:50.145469   303 raft_consensus.cc:697] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 22 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:50.155318 32547 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.005s
I20260709 13:10:50.140910 32461 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:50.167284 32743 consensus_meta.cc:287] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.734s	user 0.012s	sys 0.000s
I20260709 13:10:50.127774 32603 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:50.169948 32508 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:50.175767 25285 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:50.151504 32464 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:10:50.145920 32133 raft_consensus.cc:697] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:50.134634 32353 consensus_queue.cc:1059] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 13:10:50.181951   305 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:50.219275 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 344164us
I20260709 13:10:50.220888 25209 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:10:50.223100 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 307136us
I20260709 13:10:50.223920 25909 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:50.184454 32592 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:50.218334 32247 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:10:50.253897 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 213313us
I20260709 13:10:50.255607 25909 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:50.258569 25909 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:50.252607 32538 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:50.255053   306 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:50.253654 32247 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:50.222697   305 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:50.316864 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 63434.4us
W20260709 13:10:50.322129 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 186079us
I20260709 13:10:50.314183 25313 raft_consensus.cc:2955] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Committing config change with OpId 16.11: config changed from index -1 to 11, NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:10:50.306121 32247 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:10:50.331310 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 212200us
W20260709 13:10:50.343794 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 85306.6us
W20260709 13:10:50.345644 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:50.262403 32678 consensus_queue.cc:260] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:50.348588 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.330369   311 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:50.343897 32743 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:50.360256 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:50.351250 32299 consensus_meta.cc:287] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.529s	user 0.013s	sys 0.000s
I20260709 13:10:50.363672 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:50.319893 32133 consensus_queue.cc:237] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:50.284423   303 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:50.371284   322 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:50.497623 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.505733 25937 raft_consensus.cc:2410] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 13:10:50.442893 32508 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:50.415169 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:50.354485 32387 consensus_queue.cc:1175] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46de6919f85341c6a9b574dff112e35a" responder_term: 19 status { last_received { term: 17 index: 14 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 18. Current term is 19. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:10:50.471634 25936 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:10:50.489683 32603 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:50.358054 32678 raft_consensus.cc:3060] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:50.514424 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.363389 32743 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:50.458750 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:50.443276   308 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.111s	user 0.000s	sys 0.001s
I20260709 13:10:50.411176 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:50.446350 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.451989 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.519761 25941 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 20: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:50.521371 32743 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:50.523447 32677 raft_consensus.cc:3055] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:10:50.527560 25246 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:50.532032 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
W20260709 13:10:50.536672   314 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 13:10:50.536526 25359 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:50.539875 32723 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:50.540306 32723 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:50.540663 32723 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:50.542289 32665 raft_consensus.cc:3055] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:50.542755 32665 raft_consensus.cc:740] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:50.542906 32494 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:50.543717 32665 consensus_queue.cc:260] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:50.549769 25941 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:50.554658 25940 raft_consensus.cc:1275] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 15. (term mismatch)
I20260709 13:10:50.593542 25940 pending_rounds.cc:85] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 14
I20260709 13:10:50.593799 25940 pending_rounds.cc:107] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 18.15
I20260709 13:10:50.594028 25940 raft_consensus.cc:2889] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:50.556562 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.562562 32299 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:44966 (request call id 4) took 13056 ms. Trace:
I20260709 13:10:50.568271 25942 raft_consensus.cc:1275] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 13. (term mismatch)
I20260709 13:10:50.570107 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:50.571337 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.599880 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.600605 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.599313 25942 pending_rounds.cc:85] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 12
I20260709 13:10:50.601974 25942 pending_rounds.cc:107] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 16.13
I20260709 13:10:50.602294 25942 raft_consensus.cc:2889] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:50.601497 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:50.604701 25937 raft_consensus.cc:2410] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 13:10:50.572532 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.532971 32677 raft_consensus.cc:740] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:50.512513   322 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:50.574807 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.575212 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.577437 32502 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:10:50.581175   309 log.cc:927] Time spent T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 13:10:50.566833   305 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:50.582245 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:50.583258 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:50.615751 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.584324 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:50.580511   301 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:50.586884 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:50.589104 25237 raft_consensus.cc:1240] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 20. Current term is 21. Ops: []
I20260709 13:10:50.617241 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.590202 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:50.617985 25937 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 20: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:50.618726 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.619663 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.619753 25209 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.620599 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:50.621310 32603 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:50.621423 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:50.621763 32603 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:50.622148 32603 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:50.622519 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.623528 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.624435 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.624719   313 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:50.627949   301 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:50.596485 25938 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 18: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:50.629889 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:50.630853 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.629029 25241 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:50.633104   326 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:50.633620 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.568423 32247 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:50.633730   326 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:50.620993 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.635701 32247 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:50.635828   326 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:50.636122 25209 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.636400   326 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:50.637642 25386 raft_consensus.cc:1240] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:50.636849 25392 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 13:10:50.639233 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:50.639932 25937 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 17: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:50.641434 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.642022 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:50.624933 25943 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 20.
I20260709 13:10:50.643431 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:50.633283 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:50.597270 32494 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:50.644019 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.645529 25942 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 18: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:50.646309 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:50.649343 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.649747 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.653741 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:50.637006   326 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:10:50.654458 25284 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.653440   313 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:50.599296 32696 consensus_queue.cc:1059] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.049s
I20260709 13:10:50.643589 32723 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:50.658838   313 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:50.663854   313 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:50.664009 32353 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:50.656244   305 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:10:50.665383 25284 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.676043 32247 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:50.677541 32461 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:50.678088 32247 consensus_queue.cc:1048] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:50.681526 25239 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 17.
I20260709 13:10:50.682675 32494 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:50.683211 32494 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:50.658109 32671 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.015s
I20260709 13:10:50.672582 32723 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:50.598752 32299 rpcz_store.cc:276] 0709 13:10:37.505592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:37.505737 (+   145us) service_pool.cc:224] Handling call
0709 13:10:37.685927 (+180190us) log.cc:844] Serialized 506 byte log entry
0709 13:10:38.349478 (+663551us) spinlock_profiling.cc:243] Waited 1.75 ms on lock 0x7b1800128c20. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b932a6 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976c32b26 00007f2976c2c49a 00007f2976c2b714
0709 13:10:38.350058 (+   580us) spinlock_profiling.cc:243] Waited 1.89 ms on lock 0x7b540007ea20. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976c32b26 00007f2976c2c49a 00007f2976c2b714 00007f2976c0beaa 00007f2976c12d53 00007f2976c12cd9 00007f2976c12c69 00007f2976c12c31 00007f2976c11dfd 000000000038f354
0709 13:10:38.355737 (+  5679us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x7b540007ea20. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976c2b142 00007f2976c0beaa 00007f2976c12d53 00007f2976c12cd9 00007f2976c12c69 00007f2976c12c31 00007f2976c11dfd 000000000038f354 000000000038f199 00007f2973bc5475
0709 13:10:38.446164 (+ 90427us) spinlock_profiling.cc:243] Waited 9.59 ms on lock 0x7b540007ea20. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976c2b142 00007f2976c0beaa 00007f2976c12d53 00007f2976c12cd9 00007f2976c12c69 00007f2976c12c31 00007f2976c11dfd 000000000038f354 000000000038f199 00007f2973bc5475
0709 13:10:38.934335 (+488171us) spinlock_profiling.cc:243] Waited 4.6 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:50.562337 (+11628002us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2391,"raft.queue_time_us":7791,"raft.run_cpu_time_us":17823,"raft.run_wall_time_us":1547462,"spinlock_wait_cycles":45229312,"thread_start_us":109191,"threads_started":1,"wal-append.queue_time_us":234035,"wal-append.run_cpu_time_us":7071,"wal-append.run_wall_time_us":1989114}
I20260709 13:10:50.698437 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:50.699069 25314 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:50.673549 25909 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:50.686347 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:50.654758 32642 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.016s
I20260709 13:10:50.699590 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:50.702314 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:50.686703 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:50.703061 25313 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 17.
I20260709 13:10:50.664942 32677 consensus_queue.cc:260] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:50.704546 32677 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:50.705668 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:50.706439 25392 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 18: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:50.707511 25209 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:50.707717 25386 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 13:10:50.688417 25319 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 13:10:50.708524   326 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:50.708910   326 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:50.709259   326 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:50.692329 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:50.713093 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:50.713459 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:50.713838 25943 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 18: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:50.713979 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:50.714359 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:50.690652 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:50.698275 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:50.713667 25909 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:50.693532 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:50.687669 25285 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:50.722537 25388 raft_consensus.cc:2393] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 21: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:50.722358 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:50.685408 32494 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:50.730149 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.722179 32511 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:50.722153 32696 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.011s
I20260709 13:10:50.734266   322 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:50.725904 32712 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:50.735921 32712 raft_consensus.cc:697] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:50.736475 32483 raft_consensus.cc:3055] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:10:50.737030 32483 raft_consensus.cc:740] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:50.738113 32483 consensus_queue.cc:260] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:50.736876 32712 consensus_queue.cc:237] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:50.750618 32483 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:50.760303 32642 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:50.756717 32743 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:50.760902 32642 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:50.761750 25939 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:50.764519 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:50.765206 25315 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 21.
I20260709 13:10:50.771719 25284 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.774622 25209 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:50.782006 25284 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.783767 25212 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:50.783936 25209 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:50.784617 25286 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:50.785535 25284 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.751783 25386 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:50.788009 25212 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:50.791800 25284 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.792021 25212 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:50.793022 32723 consensus_queue.cc:1048] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:10:50.795723 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:50.803426 25284 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.804309 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:50.803097 25387 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 21.
I20260709 13:10:50.808162 25284 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.807896 25209 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.783792 32508 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:10:50.810173 25284 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.789222 32671 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:50.795295 32696 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 13:10:50.832479 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:50.852902 25908 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:50.801788   312 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:50.854828 31386 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:10:50.857226 25192 auto_rebalancer.cc:629] Failed to schedule move for tablet 72001570ee2c41cb88f0e9eac74992c7: Illegal state: Replica 2329e428ebbf484fa3ea9452133df567 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 20 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:50.780792   326 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:50.857987   328 log.cc:927] Time spent T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
I20260709 13:10:50.859114 25237 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 18: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:50.805615 32547 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:50.867760 32547 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:50.805759 32601 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:10:50.876895 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:50.877564 25387 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:50.807381 32247 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:50.878782 25911 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:50.879735 32536 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:50.880102 32536 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:50.880407 32536 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:50.846407 32762 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:10:50.872030 32603 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:50.791337 32494 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.055s
I20260709 13:10:50.812052   301 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:10:50.859082 32696 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:50.875481 32642 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:50.875833 32665 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:50.811611 32274 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:50.807544   322 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:50.893864 25284 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.894693 25314 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
W20260709 13:10:50.870600 32763 log.cc:927] Time spent T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 13:10:50.906038 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:50.901226   312 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:50.922447 31386 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:50.869959 32547 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:50.925475 25936 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
I20260709 13:10:50.916092 32696 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:50.932991 25284 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.939473 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:50.948086 25284 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:50.948351 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 156376us
I20260709 13:10:50.952163 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:50.953222 25386 raft_consensus.cc:2393] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 18: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:50.936592 32671 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:50.921702   313 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:50.941664   322 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:50.948231 32743 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:50.965842 25284 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:50.961421 32550 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:50.985416 32675 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 13:10:50.951582 32547 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:10:50.950642 32761 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:50.993899 25391 raft_consensus.cc:1240] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17. Ops: []
W20260709 13:10:50.997069 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 189252us
I20260709 13:10:50.997855 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 3071) took 1138 ms. Trace:
I20260709 13:10:50.999230 25239 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:51.001646 32671 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:51.002283 25209 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:51.003422 32671 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:51.004081 32671 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:51.004837   322 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:51.005595 32672 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:51.006023 32603 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:51.006068 32672 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:51.006803 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:51.007252   338 log.cc:927] Time spent T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 13:10:51.008198 25246 raft_consensus.cc:1240] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:51.008514 32672 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:51.103820 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:51.104308 25312 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:51.104629 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:51.107553 25389 raft_consensus.cc:1240] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 20. Current term is 21. Ops: []
I20260709 13:10:51.008726   323 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:51.010146 32603 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:51.138208 25317 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 13:10:51.011112 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:51.015074 32380 raft_consensus.cc:1064] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61: attempting to promote NON_VOTER f646dfd532974dceadfe00005d2bbf59 to VOTER
I20260709 13:10:51.139321 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:51.139735 25319 raft_consensus.cc:1275] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 13:10:51.140146 25316 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:51.016400 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:51.141095 25243 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:50.998246 25318 rpcz_store.cc:276] 0709 13:10:49.859354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:49.860362 (+  1008us) service_pool.cc:224] Handling call
0709 13:10:49.868307 (+  7945us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:49.891693 (+ 23386us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:49.891734 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:49.902604 (+ 10870us) log.cc:844] Serialized 26 byte log entry
0709 13:10:50.681202 (+778598us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b1800172d80. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973ba7e07 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b 00007f2976851d23 00007f29768520d1 00007f2976c6da0f
0709 13:10:50.886796 (+205594us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:10:50.886868 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:50.887082 (+   214us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:50.987500 (+100418us) raft_consensus.cc:1685] finished
0709 13:10:50.987519 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:50.997793 (+ 10274us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30681,"raft.queue_time_us":24357,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":12781,"spinlock_wait_cycles":6417664,"thread_start_us":205127,"threads_started":1,"wal-append.queue_time_us":1030567}
I20260709 13:10:51.032099 32671 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:51.035050 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:51.143093 25936 raft_consensus.cc:2393] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 21: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:51.036015 32274 consensus_queue.cc:1059] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:10:51.061728   313 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:51.071463 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:51.110707 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:51.144568 32274 raft_consensus.cc:3055] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:51.145113 25390 raft_consensus.cc:2368] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 17. Current term is 18.
I20260709 13:10:51.145040 32274 raft_consensus.cc:740] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:51.146201 32274 consensus_queue.cc:260] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:51.147746 32274 raft_consensus.cc:3060] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:50.992699 32547 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:51.149384 32547 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:51.114903 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:51.114782 25314 raft_consensus.cc:2468] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
I20260709 13:10:51.119143 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:51.139667 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 128616us
I20260709 13:10:51.155588 25320 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:51.155822 25209 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:51.122716 25313 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 23. Current term is 24. Ops: []
I20260709 13:10:51.159982 25285 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:51.079563   312 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:51.164516 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:51.091092 32502 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:51.173497 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:51.177145 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:51.177495 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:51.178692 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:51.158547   301 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:51.162108   306 raft_consensus.cc:3055] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:51.190328   306 raft_consensus.cc:740] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:10:51.190611   346 log.cc:927] Time spent T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.226s	user 0.000s	sys 0.001s
I20260709 13:10:51.191476   306 consensus_queue.cc:260] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:51.198490   312 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 13:10:51.199362   313 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:51.200191   322 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:51.200662   322 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:51.201050   322 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:51.165665 32039 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:51.206418 32039 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:51.208518 32039 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:51.166241 32283 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:51.166528 32671 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:51.184402 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:51.171325 32661 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:51.186607 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:51.187343 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:51.189361 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:51.210232 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:51.214169 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:51.224255 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:51.225317 25936 raft_consensus.cc:2393] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 18: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:51.230814 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:51.240413 25209 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:51.247179 25287 leader_election.cc:400] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 17 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 17. Current term is 18.
I20260709 13:10:51.220146 32671 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:51.252061 25246 raft_consensus.cc:1240] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 24 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 23. Current term is 24. Ops: []
I20260709 13:10:51.182158 32502 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:51.284759 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:51.303973 32536 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:51.159055   323 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:51.306609 32502 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:51.308583   326 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:51.285456 32661 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:51.247068 32714 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:10:51.288722 32696 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:51.246932   313 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:51.292351   312 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:10:51.323464 32677 consensus_queue.cc:1059] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:10:51.324404 32547 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:51.285966 32671 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:51.227065 32283 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:51.336571   323 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:51.349810   322 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:51.417610 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:51.418530 25237 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:51.420112 25237 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:10:51.421736 25245 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:10:51.346877   313 raft_consensus.cc:697] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:51.424603 25393 raft_consensus.cc:2368] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 21. Current term is 22.
I20260709 13:10:51.426024 31386 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:51.426668 31386 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:51.429502 25245 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 13:10:51.341688 32661 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:51.430383 25243 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 20. Current term is 21. Ops: []
I20260709 13:10:51.434326 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:51.436127 31386 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:51.438081 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:51.439810 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:51.440433 25395 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:51.443781 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:51.444761 25320 raft_consensus.cc:1275] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:10:51.410657   301 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:51.446108 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:51.446228 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:51.447242 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:51.447739 25320 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
I20260709 13:10:51.432930 32592 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:51.454798 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:51.455076 25359 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:51.453322   301 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:51.460067 32592 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } }
I20260709 13:10:51.487139   364 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:51.487016   313 consensus_queue.cc:237] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:51.487749   364 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:51.489396   355 log.cc:927] Time spent T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 13:10:51.489650 32714 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:51.490186 32714 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:51.492094 32714 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:51.495465   322 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:51.498306 32547 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:51.498622 32408 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:51.499456 32114 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:51.499173 32408 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:51.499857 32114 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:51.500303 32114 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:51.501080 32661 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:51.501307 32408 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:51.506556 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 51564.6us
I20260709 13:10:51.510650 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:51.511345 25392 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 20.
I20260709 13:10:51.517467 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:51.518041 25390 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:51.519560 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:51.525454 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:51.499789 32536 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:51.535867 25390 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 20.
I20260709 13:10:51.537587 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:51.538221 25242 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:51.514703 32284 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:51.540424 25911 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:51.544596 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:51.545356 25241 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:51.552286 25313 raft_consensus.cc:2468] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:51.530006 32714 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:51.514161 32645 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.036s
I20260709 13:10:51.568600 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:51.554108 32536 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:51.574450 25242 raft_consensus.cc:2393] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 21: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:51.553902 32387 raft_consensus.cc:2804] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:10:51.529650 32299 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.031s
I20260709 13:10:51.539165 32723 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
I20260709 13:10:51.561969 32696 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:51.561800 32714 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:51.602780 31386 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:51.603387 31386 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:51.554841 32284 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:51.605460 32114 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:51.609503   306 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:51.587163 32536 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:51.611717   364 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:51.614228 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 44581.8us
I20260709 13:10:51.604079 32387 raft_consensus.cc:697] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 20 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:51.604377 32714 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:51.626484 25395 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 13:10:51.628509 25359 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:51.631464 31386 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:51.631866 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 88391.1us
I20260709 13:10:51.644508 25359 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:51.643610 32645 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:51.627971   301 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:51.661590 25358 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:51.663179 25358 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:51.655622 32645 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:51.666796 32558 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:51.664950 32592 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:51.696218 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:51.699268 25241 raft_consensus.cc:1240] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 17. Current term is 18. Ops: []
I20260709 13:10:51.700935 25243 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 13:10:51.701164 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:51.701900 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:51.708516 25237 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:51.704403   320 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:51.713138 32511 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:51.713456   320 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:51.713718 32511 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:51.715729 32511 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:51.716095 32677 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:51.716307 32671 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:51.703773 32645 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:51.717437 32536 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:51.703112   366 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:10:51.718928 32637 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:51.719396 32637 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:51.669694   365 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20260709 13:10:51.704473 32558 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:51.722672 32558 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:51.722968 32637 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:51.725088 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 60665.3us
I20260709 13:10:51.727633   366 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:51.728242   366 consensus_queue.cc:1048] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:10:51.709878   368 raft_consensus.cc:2804] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:10:51.729681   368 raft_consensus.cc:697] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:51.730756 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:51.730525   368 consensus_queue.cc:237] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:51.732034 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:51.732192 32283 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:51.714108   330 log.cc:927] Time spent T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 1.050s	user 0.003s	sys 0.004s
I20260709 13:10:51.733798 32114 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:51.659360 32387 consensus_queue.cc:237] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:51.735256 25391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60946 (request call id 2593) took 2327 ms. Trace:
I20260709 13:10:51.735606 25391 rpcz_store.cc:276] 0709 13:10:49.407572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:49.682038 (+274466us) service_pool.cc:224] Handling call
0709 13:10:49.682355 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:50.995709 (+1313354us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:51.735047 (+739338us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54001f59d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:51.735200 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3015711104}
I20260709 13:10:51.705281 32363 consensus_queue.cc:1059] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.031s
I20260709 13:10:51.737051 32363 raft_consensus.cc:3055] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:10:51.737417 32363 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:51.712605 25285 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:51.738166 32363 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:51.716634 32677 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:51.750432 25285 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:51.741770 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:51.743409 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:51.739588 31386 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:51.764137 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:51.764374 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:51.764704 25315 raft_consensus.cc:3060] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:51.764930 25320 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:51.763312 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:51.770463 25389 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:51.770520 25358 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:51.774844 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 49105.9us
I20260709 13:10:51.782727 25908 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:51.774490 32550 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:51.739796 32483 consensus_queue.cc:1059] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.049s
I20260709 13:10:51.788709 32550 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:51.789064 32483 raft_consensus.cc:3055] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:10:51.789525 32483 raft_consensus.cc:740] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:51.790318 32483 consensus_queue.cc:260] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:51.790843 32550 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:51.791424 32483 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:51.744892 32665 raft_consensus.cc:3055] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:10:51.791960 32665 raft_consensus.cc:740] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:51.740362 32460 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 13:10:51.799968 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:51.800917 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:51.804281 25316 raft_consensus.cc:1240] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 13:10:51.804263 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:51.805697 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:51.805997 25320 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 22
I20260709 13:10:51.810931 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:51.814575 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:51.815367 25319 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:51.817749 31386 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:51.819025 32637 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:51.824347 32677 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:51.824856   369 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:51.793061 32665 consensus_queue.cc:260] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:51.825785   369 consensus_queue.cc:1048] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:10:51.826597 32665 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:10:51.835536 25319 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
I20260709 13:10:49.920364 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:10:51.923734 25212 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:51.925952 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:51.836634 25314 raft_consensus.cc:1275] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:10:51.832114 32536 consensus_queue.cc:1059] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.019s
W20260709 13:10:51.990356 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 67089.1us
I20260709 13:10:51.991487 32536 raft_consensus.cc:3055] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:10:51.992082 32536 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:51.992475   306 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:51.850474 32558 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:51.993058   306 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:51.993665   306 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:51.793445   311 consensus_queue.cc:1059] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.049s
W20260709 13:10:51.861009   371 log.cc:927] Time spent T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 13:10:51.852784 32380 consensus_queue.cc:237] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 11, Current term: 18, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:51.837893 32405 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:52.000491 32405 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.001471 32762 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:52.002220 32762 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:52.003507 32405 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:51.746822 32671 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:52.004264 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:52.005549 25246 raft_consensus.cc:2393] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 15: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:52.006520 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:51.789115 32645 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:51.754528 32714 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:52.025238 32645 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:51.858354   374 log.cc:927] Time spent T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.189s	user 0.000s	sys 0.000s
I20260709 13:10:52.029842 25316 raft_consensus.cc:2393] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 21: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:51.749634   320 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:52.026134 32714 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:51.942324 25239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32992 (request call id 3080) took 1189 ms. Trace:
I20260709 13:10:52.033982 25239 rpcz_store.cc:276] 0709 13:10:50.752273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:50.752452 (+   179us) service_pool.cc:224] Handling call
0709 13:10:50.753073 (+   621us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:50.999632 (+246559us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:51.941648 (+942016us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b5400364a54. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:51.942051 (+   403us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":565723520}
I20260709 13:10:52.041280 32536 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.043375 32536 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:51.943821 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:52.045053 25240 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
W20260709 13:10:51.940672 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 116211us
I20260709 13:10:51.946169 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:51.951506 25139 catalog_manager.cc:5697] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 9 to 14. New cstate: current_term: 14 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:10:51.948807 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:51.962525 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:51.967672 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:51.960253 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 163414us
I20260709 13:10:51.980724 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:52.010382 31386 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:51.996169 32558 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:52.017701   378 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:51.940366 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:52.050179 31386 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:52.052430 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 44994.5us
W20260709 13:10:52.055540 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 93471.2us
I20260709 13:10:52.061201 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:52.062917 32603 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:10:52.064129 25237 raft_consensus.cc:3060] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:52.078222 25908 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:52.085651 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:52.091480 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:52.094842 25393 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 13:10:52.095311 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:52.096182 25395 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:52.100274 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:52.100935 25238 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:52.105221 32665 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:52.090248 25910 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:52.079262 32762 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:52.120329   311 raft_consensus.cc:3055] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:10:52.121378 32665 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.124218 32665 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:52.106698 32405 raft_consensus.cc:2804] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:52.132265 32405 raft_consensus.cc:697] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:52.133637 32405 consensus_queue.cc:237] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.133062 25238 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:52.141813 25395 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 22.
W20260709 13:10:52.141831 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 63815.9us
W20260709 13:10:52.146680 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 53928.1us
I20260709 13:10:52.120898   311 raft_consensus.cc:740] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:52.145241   326 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.011s
I20260709 13:10:52.157284 31386 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:52.168246   318 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:52.169024   318 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.171079 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:52.171818 25391 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:52.171679   318 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:52.174397 25908 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:52.174568 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:52.175619   320 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:52.176086   320 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:52.176525   320 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:52.185926 25911 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:52.187726   369 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:52.188248   369 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:52.188678   369 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:52.189885 25938 raft_consensus.cc:1240] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 13:10:52.190522 25237 raft_consensus.cc:2468] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
I20260709 13:10:52.180339 25139 catalog_manager.cc:5697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 11 to 15, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 15 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:52.197563 32284 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:52.201483 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:52.203663 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:52.204135 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:52.204861 25312 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:52.206346 25359 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:52.196882 32363 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:52.211350   306 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:52.204890 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:52.212491 25245 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:52.213300 25314 raft_consensus.cc:2368] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 20. Current term is 21.
I20260709 13:10:52.145396   378 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.214984 32665 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:52.215616 32665 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.216300   378 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:52.216960 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:52.217851 32665 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:52.218616 32502 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:10:52.217868 25393 raft_consensus.cc:2368] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16.
I20260709 13:10:52.219203 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:52.220813 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:52.226415 25245 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
I20260709 13:10:52.222401 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:52.229638 32502 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:52.245224 32672 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:52.265317 32665 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:52.266083 32665 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.212615   311 consensus_queue.cc:260] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.269951 25358 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:52.270303   311 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:52.272465 25212 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:52.273562 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 52988.2us
I20260709 13:10:52.273432   359 consensus_queue.cc:1059] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:52.274900 25287 leader_election.cc:400] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 21. Current term is 22.
I20260709 13:10:52.275211 25287 leader_election.cc:403] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:10:52.276273   359 consensus_queue.cc:1048] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:52.277557 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:52.277873 25359 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:52.278214 25386 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 22.
I20260709 13:10:52.278939   316 raft_consensus.cc:3055] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:52.279328 25911 leader_election.cc:400] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16.
I20260709 13:10:52.279665 25911 leader_election.cc:403] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:10:52.280066 32408 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:10:52.280460 25359 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:52.280694 32408 raft_consensus.cc:697] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:52.285405 25941 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 13:10:52.277976 32461 raft_consensus.cc:2804] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:10:52.289774 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:52.287843 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:52.238633 32502 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:52.290617 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:52.244309 25285 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:52.248448   369 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.281920 32637 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:10:52.311652 32637 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:52.296190 25938 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 22.
I20260709 13:10:52.295024 25911 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:52.288944 32642 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:52.281998   313 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:10:52.303572 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:52.304153 32461 raft_consensus.cc:697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:52.279358   316 raft_consensus.cc:740] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:52.295321 25943 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
I20260709 13:10:52.304654 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:52.305002 32672 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:52.306149   320 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.284485 32536 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 14 FOLLOWER]: Advancing to term 16
I20260709 13:10:52.317535 25392 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 20.
W20260709 13:10:52.308734   385 log.cc:927] Time spent T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 13:10:52.309476 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:52.313027 25312 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 22.
I20260709 13:10:52.287070 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:52.316918 25320 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 22.
I20260709 13:10:52.326318 25936 raft_consensus.cc:2393] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 20: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:52.326484 31386 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 21. Current term is 22.
I20260709 13:10:52.332059 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:52.335734 25316 raft_consensus.cc:2393] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 21: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:52.317003   313 raft_consensus.cc:697] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 22 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:52.334947 25911 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:52.328241   378 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:10:52.341543 25942 raft_consensus.cc:3060] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:52.312968 32637 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:52.349776   313 consensus_queue.cc:237] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:52.351261 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:52.334517 32642 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.352672 32726 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:52.353108   379 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:10:52.353540   379 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:52.353880   379 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:52.354616 32461 consensus_queue.cc:237] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:52.341308 32672 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.359148 32672 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:52.362701 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 179541us
I20260709 13:10:52.363981 25285 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:52.363946 32726 consensus_queue.cc:1059] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:10:52.366787 32500 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:10:52.367354 32726 raft_consensus.cc:3055] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:52.381597 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:52.347194   378 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:52.397459 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 98338us
I20260709 13:10:52.400785 32408 consensus_queue.cc:237] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.396566 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:52.406229   320 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:52.408030 32642 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:52.408769   322 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:52.409444   322 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:52.409813 25392 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:10:52.413858 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:52.416846 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:52.417454   369 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:52.419461 32536 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 15. Current term is 16.
I20260709 13:10:52.420203 25940 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
W20260709 13:10:52.421386 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 57493.3us
I20260709 13:10:52.369494 32726 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:52.426282 25909 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:52.432152 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:52.432930 25246 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:52.435537 32726 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.437209 32726 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:52.438141 25909 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:52.439026   369 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:52.439258 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:52.440636   369 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:52.428334   322 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:52.417884 32642 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.011s
I20260709 13:10:52.442313 25909 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:52.443434 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:52.415977   378 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:52.414909 32665 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:52.459570 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:52.464507 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:52.466845 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 50214.7us
I20260709 13:10:52.468001 25319 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 21.
I20260709 13:10:52.470296 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:52.471753 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:52.472402 25242 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 21.
I20260709 13:10:52.448478 32536 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:52.474619 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:52.475173 25387 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:52.487885 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:52.484411 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:52.491180 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:52.491983 25245 raft_consensus.cc:3060] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 20
I20260709 13:10:52.483958   306 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:52.499039 25939 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEARNER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:10:52.502978   322 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:52.508713   316 consensus_queue.cc:260] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:52.514081 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:52.515740 25141 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39572 (request call id 294) took 19662 ms (client timeout 15000 ms). Trace:
I20260709 13:10:52.531349   322 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:52.533629   322 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:52.534406 25139 catalog_manager.cc:5697] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 7 to 10. New cstate: current_term: 10 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:52.537787 25240 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 21: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
W20260709 13:10:52.537711 25315 consensus_meta.cc:287] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.772s	user 0.009s	sys 0.009s
I20260709 13:10:52.538446 25315 raft_consensus.cc:2468] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 20.
I20260709 13:10:52.539053 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:52.539731 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:52.537669 25320 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:10:52.556005 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:52.539816 25242 raft_consensus.cc:2393] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 21: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:52.557549 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:52.561568   379 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.536569 25139 catalog_manager.cc:5697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 9 to 13, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 13 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:52.562873   316 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:52.566371 25238 raft_consensus.cc:3060] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:52.529840 25209 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:52.575579 25245 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 20.
I20260709 13:10:52.590929 25237 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:10:52.579496 25387 raft_consensus.cc:2468] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 20.
I20260709 13:10:52.603295 32673 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:52.605067 25911 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:52.604874 25358 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:52.612279 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 114735us
I20260709 13:10:52.612915   311 raft_consensus.cc:2804] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:10:52.648703 25139 catalog_manager.cc:5697] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:52.652894 32592 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:52.653486 32592 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:52.655560 32592 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:52.658370   379 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:52.603883 32673 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.668670 31386 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:52.666496 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 53367.7us
I20260709 13:10:52.674556 31386 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:52.675252 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 70604.3us
I20260709 13:10:52.675904 32502 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:52.676215 25358 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:52.646356 32665 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:52.679555 31386 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:52.652355 32637 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:10:52.679970 25139 catalog_manager.cc:5697] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 14 to 17. New cstate: current_term: 17 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:10:52.683645 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:52.686039 25390 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 22
I20260709 13:10:52.688208 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:52.683307   369 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:52.689832   369 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.684274 32637 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:52.690672 32637 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:52.692092   369 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:10:52.694456 32502 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:52.694772 25287 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:52.696316 25389 raft_consensus.cc:1240] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 13:10:52.697561 31386 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:52.698308 25285 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:52.699447 25285 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:52.700464 32712 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:10:52.701085 25285 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:52.701958 32712 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:52.689047 32696 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:10:52.705916 25390 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 22.
I20260709 13:10:52.707360 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:52.708156 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:52.710695 25911 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:52.712076 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:52.667341   311 raft_consensus.cc:697] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 20 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:52.728734 32762 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:10:52.729771 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 46245.7us
I20260709 13:10:52.728754 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:52.746555 25209 leader_election.cc:400] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 20. Current term is 21.
I20260709 13:10:52.683799 32672 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:52.720613 32645 raft_consensus.cc:2764] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 13:10:52.753726 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:52.762409 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:52.773268   364 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.019s
W20260709 13:10:52.778923 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 102871us
I20260709 13:10:52.789655   311 consensus_queue.cc:237] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.791349 25358 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:52.794382 32673 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:52.796535   366 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:52.737823 32500 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:52.797518   368 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 13:10:52.522459 25141 rpcz_store.cc:269] 0709 13:10:32.853678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:32.853937 (+   259us) service_pool.cc:224] Handling call
0709 13:10:52.515507 (+19661570us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:10:45.956446 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:10:45.957273 (+   827us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:10:45.957341 (+    68us) write_op.cc:435] Acquired schema lock
0709 13:10:45.957378 (+    37us) tablet.cc:662] Decoding operations
0709 13:10:45.989024 (+ 31646us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:10:45.989150 (+   126us) write_op.cc:641] Partition lock acquired for write op
0709 13:10:45.989193 (+    43us) tablet.cc:685] Acquiring locks for 15 operations
0709 13:10:45.990821 (+  1628us) tablet.cc:701] Row locks acquired
0709 13:10:45.990861 (+    40us) write_op.cc:260] PREPARE: finished
0709 13:10:45.991092 (+   231us) write_op.cc:270] Start()
0709 13:10:45.991289 (+   197us) write_op.cc:276] Timestamp: P: 1783602645991001 usec, L: 0
0709 13:10:45.991313 (+    24us) op_driver.cc:348] REPLICATION: starting
0709 13:10:45.992259 (+   946us) log.cc:844] Serialized 5974 byte log entry
0709 13:10:46.900729 (+908470us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:10:47.631833 (+731104us) op_driver.cc:464] REPLICATION: finished
0709 13:10:47.919827 (+287994us) write_op.cc:301] APPLY: starting
0709 13:10:47.923951 (+  4124us) tablet.cc:1367] starting BulkCheckPresence
0709 13:10:48.621798 (+697847us) tablet.cc:1370] finished BulkCheckPresence
0709 13:10:48.621827 (+    29us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:10:49.026991 (+405164us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:10:49.027312 (+   321us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=30,key_file_lookups=30,delta_file_lookups=30,mrs_lookups=0
0709 13:10:49.027365 (+    53us) write_op.cc:312] APPLY: finished
0709 13:10:49.027767 (+   402us) log.cc:844] Serialized 139 byte log entry
0709 13:10:49.210583 (+182816us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:10:49.236050 (+ 25467us) write_op.cc:454] Released schema lock
0709 13:10:49.262827 (+ 26777us) write_op.cc:341] FINISH: Updating metrics
0709 13:10:49.304498 (+ 41671us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:10:49.304967 (+   469us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2214930,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":250071,"apply.run_cpu_time_us":56746,"apply.run_wall_time_us":1373685,"cfile_cache_hit":62,"cfile_cache_hit_bytes":66767,"num_ops":15,"prepare.queue_time_us":216283,"prepare.run_cpu_time_us":12350,"prepare.run_wall_time_us":105353,"raft.queue_time_us":253712,"raft.run_cpu_time_us":10505,"raft.run_wall_time_us":246474,"replication_time_us":1608734,"spinlock_wait_cycles":44711552,"thread_start_us":706834,"threads_started":5,"wal-append.queue_time_us":370435,"wal-append.run_cpu_time_us":24321,"wal-append.run_wall_time_us":3523843}]]}
I20260709 13:10:52.802134 32672 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.797521 32500 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.804291 32672 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:52.804447 32500 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:52.809306 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:52.810160 25313 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
W20260709 13:10:52.811535   395 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
I20260709 13:10:52.737536   369 raft_consensus.cc:2804] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:10:52.813056   369 raft_consensus.cc:697] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 22 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:52.814275   369 consensus_queue.cc:237] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.816180   378 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.712738 32665 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.819478   378 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:52.819569 32665 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:52.820765 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:52.821512 25942 raft_consensus.cc:2468] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 22.
W20260709 13:10:52.823086 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 82762.6us
I20260709 13:10:52.823303 32637 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.824121 32726 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:52.825949 32726 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:52.824333 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:52.827380 25245 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:52.812160 25318 raft_consensus.cc:1275] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:10:52.827478 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:52.827668 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:52.828406 25244 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 24.
I20260709 13:10:52.829362 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:52.829567 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 72117.9us
I20260709 13:10:52.829398 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:52.831920 25320 raft_consensus.cc:2393] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 21: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:52.829584 32726 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:52.834487 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:52.837491 25285 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:52.838335 25285 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:52.830894 32637 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:52.839789 31386 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:52.840255 31386 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:52.840618 31386 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:52.842710 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:52.849516 25908 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:52.866086 32642 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:10:52.867475 25908 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:52.851548 25315 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:10:52.852608 25284 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:52.824764 25209 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:52.852977 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:52.856264 25238 raft_consensus.cc:1275] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:10:52.827556 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:52.903118 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:52.912896 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 60578.8us
I20260709 13:10:52.912853 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:52.914013 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:52.911579 25939 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 24.
I20260709 13:10:52.918573 25942 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:10:52.915149 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:52.903048 32726 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:10:52.907701 32642 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:52.925719 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:52.928882 25285 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:52.915751 32592 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:10:52.932770 25315 raft_consensus.cc:1275] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:10:52.930539 32712 consensus_queue.cc:1059] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.003s
I20260709 13:10:52.934053 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:52.917605 32696 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:52.917747 32603 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:52.921893 32726 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:10:52.937067 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 112490us
I20260709 13:10:52.937463 25358 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:52.920836 25908 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:52.950619 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:52.964627 32762 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:52.969328 32637 raft_consensus.cc:2804] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:10:52.969975 32637 raft_consensus.cc:697] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:52.945739 32726 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:10:52.952823 32405 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:52.936838 32645 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 13:10:52.964874   378 consensus_queue.cc:1048] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.032s
I20260709 13:10:52.993050 32405 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:52.995304 32696 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election won for term 21
W20260709 13:10:53.005628 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 76849.3us
I20260709 13:10:53.007351 32405 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:53.043292 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 133928us
I20260709 13:10:53.053220 32665 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:10:52.995697 32603 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:53.038780   405 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:53.076294 25242 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
W20260709 13:10:53.077365 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 73857.1us
I20260709 13:10:53.077911 25241 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:10:53.080554 25909 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:53.081773 25942 raft_consensus.cc:1240] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:53.085055 32502 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:53.090946 32380 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 13:10:53.092629 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 113036us
I20260709 13:10:53.097738 32637 consensus_queue.cc:237] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:53.114432 31386 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:53.118223 25320 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 21: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:10:53.129472 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 48983.9us
I20260709 13:10:53.131366 25909 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:53.132938 25242 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:10:53.139870 32672 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:53.140467 32672 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:53.058248 32500 raft_consensus.cc:3055] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:10:53.045253 32461 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:53.148293 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:53.154136 31386 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:53.157696 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:53.158047 25287 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:53.161978 25358 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:53.172966 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:53.175985 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:53.176260 25939 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:53.127233   405 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:53.179220 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:53.180009 32603 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:53.180455   405 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:53.182250   301 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:53.182811   301 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:53.185226 32461 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:53.185663   366 consensus_queue.cc:1048] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:10:53.186817 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:53.187728 32408 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:10:53.188645 32762 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:10:53.174968 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:53.146690   378 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:53.193406 25316 raft_consensus.cc:2368] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 20. Current term is 21.
I20260709 13:10:53.193485 25312 raft_consensus.cc:2393] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 21: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:53.135358   369 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:53.199640 25939 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 21.
W20260709 13:10:53.203081 25210 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:53.148523 32500 raft_consensus.cc:740] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:53.153051 32363 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:53.201697 25284 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:53.177019 32696 raft_consensus.cc:697] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 21 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:53.157538 32645 consensus_queue.cc:1048] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.053s
I20260709 13:10:53.232911 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:53.234519 25393 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 13:10:53.238502 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:53.239125 25239 raft_consensus.cc:3060] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:53.244036 25390 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 22. Current term is 23. Ops: []
I20260709 13:10:53.261528   418 raft_consensus.cc:2804] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:10:53.262120   418 raft_consensus.cc:697] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:53.218585 32380 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.029s
I20260709 13:10:53.268179 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:53.269196 25936 raft_consensus.cc:2393] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 23: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:10:53.265652 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 76509.3us
I20260709 13:10:53.286257 32500 consensus_queue.cc:260] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:53.290661   306 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:53.291855 32500 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:53.262585 32637 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:53.259115 32363 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:53.286442 32408 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:53.315773 32673 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:53.322654 32408 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:53.322779 32673 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:53.323463   378 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:53.324884 32673 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:53.326098 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:53.326685 25390 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 22.
I20260709 13:10:53.321983   416 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:10:53.341007 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:53.304003 32672 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:53.382339 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 70344.9us
W20260709 13:10:53.383104   410 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 13:10:53.391659   306 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 13:10:53.393677 25239 raft_consensus.cc:2468] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 21.
I20260709 13:10:53.394572 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:53.396212 25314 raft_consensus.cc:1275] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:10:53.393397   427 consensus_queue.cc:1059] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:53.397377 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:53.398030   427 raft_consensus.cc:3055] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:53.398537   427 raft_consensus.cc:740] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:53.399019 32637 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:53.399595   427 consensus_queue.cc:260] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:53.401202 32637 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:53.401809 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:53.402525 25241 raft_consensus.cc:3060] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:53.398190 25244 raft_consensus.cc:1275] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:10:53.398190 25239 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 17.
I20260709 13:10:53.324817 32408 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:53.426851 25358 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:53.438596 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:53.440659 25321 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:10:53.429397   427 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:53.450045 25209 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:53.456786 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:53.457729 25209 leader_election.cc:400] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 20. Current term is 21.
I20260709 13:10:53.458793 25209 leader_election.cc:403] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 13:10:53.463337 25244 raft_consensus.cc:2393] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 21: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:53.339066 32696 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:53.470212   316 consensus_queue.cc:1048] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:10:53.472355   316 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:10:53.481802   424 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315) with higher term. Message: Invalid argument: T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 20. Current term is 21.
I20260709 13:10:53.487102   420 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:10:53.493362 32461 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:53.494086 25209 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:53.494825   422 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:53.397397   418 consensus_queue.cc:237] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:53.461979 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:53.435559   306 consensus_queue.cc:1059] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.027s
I20260709 13:10:53.493254 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:53.454537   322 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:53.519315 25321 raft_consensus.cc:2393] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 23: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:53.520756 25287 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:53.519621 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:53.533493 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 226771us
I20260709 13:10:53.532949 32284 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:53.542878 25939 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:10:53.540307 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:53.544138 25319 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 23: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:53.544137 25936 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 23.
I20260709 13:10:53.531261 32363 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:53.555498 25389 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:53.539706   424 raft_consensus.cc:3055] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:10:53.542892 32762 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:10:53.551491 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:53.566149 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:53.569303 25941 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 23.
I20260709 13:10:53.574116 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:53.574936 25319 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 15.
I20260709 13:10:53.560842   424 raft_consensus.cc:740] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
W20260709 13:10:53.588359 32726 consensus_meta.cc:287] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.590s	user 0.007s	sys 0.005s
W20260709 13:10:53.600494 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 63846.2us
I20260709 13:10:53.562254 25321 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:10:53.603250 25321 raft_consensus.cc:606] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:53.603724 25321 raft_consensus.cc:614] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:10:53.625018 25241 raft_consensus.cc:2468] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 22.
I20260709 13:10:53.551151 32284 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:53.641429 25389 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:10:53.650884 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
W20260709 13:10:53.665442 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 117389us
I20260709 13:10:53.631141 32665 consensus_queue.cc:1048] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.028s
I20260709 13:10:53.670136 25358 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:53.632220   368 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 13:10:53.678609 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:53.675112 25287 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:53.675743 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 88531.9us
I20260709 13:10:53.624984 32726 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:53.685235 32726 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 25 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:53.677547 32461 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:10:53.687703 32461 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:53.688186 32461 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:10:53.689683   437 log.cc:927] Time spent T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260709 13:10:53.681815 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:53.697794 32637 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:10:53.729517 32637 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting leader election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:53.729981 32637 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:53.739337   301 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:53.693215 32500 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:53.741358 32500 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:53.741695 32500 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:53.741978 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:53.742534 32405 consensus_meta.cc:287] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Time spent flushing consensus metadata: real 0.685s	user 0.010s	sys 0.000s
I20260709 13:10:53.742924 32405 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:53.735415 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 53685.5us
I20260709 13:10:53.747123 32642 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:10:53.753719 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:53.740928 32284 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:53.769443 25318 raft_consensus.cc:2410] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for term 18 because replica has last-logged OpId of term: 16 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 14.
I20260709 13:10:53.744925 32405 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
W20260709 13:10:53.747068   426 log.cc:927] Time spent T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.231s	user 0.001s	sys 0.000s
I20260709 13:10:53.774085 25936 raft_consensus.cc:1240] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 22. Current term is 23. Ops: [22.18-22.18]
I20260709 13:10:53.774744 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:10:53.795364 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 148419us
W20260709 13:10:53.798674   444 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 13:10:53.807876 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:53.810092 25386 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:10:53.699177   424 consensus_queue.cc:260] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:53.787348 32461 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:53.808529 25318 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 20.
W20260709 13:10:53.806277   435 log.cc:927] Time spent T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.237s	user 0.001s	sys 0.000s
I20260709 13:10:53.811662   369 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:53.821995   369 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:53.824657   369 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:53.825500 32642 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:10:53.826208 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:53.827476 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:53.828593 25312 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:10:53.829893   430 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:53.830500   436 log.cc:927] Time spent T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.239s	user 0.001s	sys 0.001s
I20260709 13:10:53.831424 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:53.833182   432 log.cc:927] Time spent T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.245s	user 0.001s	sys 0.000s
I20260709 13:10:53.834769 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:53.835074   443 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.229s	user 0.000s	sys 0.003s
I20260709 13:10:53.835448 25239 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:53.842065 25945 raft_consensus.cc:1275] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:10:53.855279 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:53.855684 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:53.856571 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:53.817361 32665 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.056s
I20260709 13:10:53.857095 25314 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:53.873746 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 58185.7us
I20260709 13:10:53.817144 32363 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:53.869827 25285 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:53.869956   416 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:53.870293   424 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:10:53.931319 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:53.861699 32762 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:53.999248 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
W20260709 13:10:54.025759 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 157618us
I20260709 13:10:53.870148 32500 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:53.997711 25320 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 13:10:53.864332   430 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:53.882822 32363 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:53.884367   418 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:54.025673 32762 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:10:54.111002 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 112999us
I20260709 13:10:53.886833   416 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:54.105196 25245 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:10:54.102689   301 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:54.236083   418 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:54.037961   460 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:54.116871 32726 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:54.242015 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 140268us
I20260709 13:10:54.212205 32696 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:54.247238 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 365191us
I20260709 13:10:54.248268 25285 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:54.257771 25314 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:54.240088   378 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:54.263789 25245 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 25.
I20260709 13:10:54.262022   431 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:54.258016   450 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.010s
I20260709 13:10:54.240339   301 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:54.267871   431 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:54.268342   431 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:54.269374 32500 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:54.245308   418 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:54.272701   418 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:54.275674 32363 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:54.276698 32637 consensus_meta.cc:287] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61: Time spent flushing consensus metadata: real 0.546s	user 0.000s	sys 0.010s
I20260709 13:10:54.283174 32762 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:54.285705 32762 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:54.247648   460 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:54.285723   301 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:54.287177   454 log.cc:927] Time spent T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.183s	user 0.001s	sys 0.000s
I20260709 13:10:54.287995   460 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:54.289402 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:54.290084 25321 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:54.291409 25936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:54.292248 25936 raft_consensus.cc:3060] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:10:54.299141 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:54.301492 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:54.256592 32696 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:54.248231 32726 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:54.266263 32461 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:54.321781 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:54.322618 25939 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:10:54.337761 25909 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:54.364648 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:54.302095 32637 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:54.395498 32637 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:54.395814 32726 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:54.397624   458 log.cc:927] Time spent T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.160s	user 0.000s	sys 0.000s
W20260709 13:10:54.400528   427 consensus_meta.cc:287] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.943s	user 0.011s	sys 0.000s
I20260709 13:10:54.300239   378 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:54.401767   416 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:54.402683 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:54.403103   430 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:54.403515   461 log.cc:927] Time spent T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 13:10:54.403669 25943 raft_consensus.cc:2393] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 18: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
W20260709 13:10:54.405906 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 76415.5us
I20260709 13:10:54.309674   451 raft_consensus.cc:2955] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Committing config change with OpId 18.13: config changed from index 11 to 13, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:10:54.412087 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 74438.7us
I20260709 13:10:54.419757 25939 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 24.
I20260709 13:10:54.421406 25284 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:54.424669 25936 raft_consensus.cc:2468] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 25.
I20260709 13:10:54.425752 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:54.430366 25319 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 23: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
W20260709 13:10:54.430488 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:54.431173 25284 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:54.431268 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:54.402704   378 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:54.416339 25943 raft_consensus.cc:2955] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Committing config change with OpId 18.13: config changed from index 11 to 13, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:54.433071 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:54.415302 32665 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:54.427929   430 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:10:54.448333 25942 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 23.
W20260709 13:10:54.449244 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 184113us
I20260709 13:10:54.436079   416 raft_consensus.cc:2804] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:10:54.457490 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:54.458298 25239 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:54.460629 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:54.461232 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:54.473107 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:54.469573 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:54.478322 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:54.479099 25389 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 20.
I20260709 13:10:54.476702 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:54.480584 25944 raft_consensus.cc:2368] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 23. Current term is 24.
I20260709 13:10:54.450664   430 raft_consensus.cc:697] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 24 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:54.489987 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:54.469172   416 raft_consensus.cc:697] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 25 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:54.484881 32696 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:54.497440   301 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:54.606719 32645 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:54.607997   301 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:54.629702 25240 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:10:54.635579 25285 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:54.630029   301 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:54.661902   431 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:54.663447   456 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:54.664199   456 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:54.666718   456 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:54.639396   474 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:54.682909 25239 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:54.632917   416 consensus_queue.cc:237] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:54.625844   430 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:54.634465 32645 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:54.705000 32645 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:54.718014 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:54.722638 25941 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 21.
W20260709 13:10:54.725286   424 consensus_meta.cc:287] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.728s	user 0.008s	sys 0.004s
I20260709 13:10:54.726327 25284 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:54.727962 32762 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:10:54.728531 32762 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:54.728941 32762 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:54.737277 25237 raft_consensus.cc:1275] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 13:10:54.738471 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 101712us
I20260709 13:10:54.739470 25318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36876 (request call id 3136) took 1156 ms. Trace:
I20260709 13:10:54.744274 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:54.743723 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:54.820504 25242 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:54.821424 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 82810.7us
I20260709 13:10:54.821623 32558 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:54.822343 32558 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:54.819445 25318 rpcz_store.cc:276] 0709 13:10:53.582764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:53.858057 (+275293us) service_pool.cc:224] Handling call
0709 13:10:53.858364 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:54.246252 (+387888us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:54.246281 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:54.246318 (+    37us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:10:54.246349 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:54.739275 (+492926us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x7b54000627d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb2e01 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2 00007f2974fed48c 00007f2974fec8c6
0709 13:10:54.739316 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:54.739420 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":891649920}
W20260709 13:10:54.823819 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 85872.6us
W20260709 13:10:54.824548 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 86979.9us
W20260709 13:10:54.825021 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 79178.8us
I20260709 13:10:54.826607 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:54.828087 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:54.828771 25320 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 19.
I20260709 13:10:54.813804 25240 raft_consensus.cc:1275] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:10:54.774701   301 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:54.829471 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:54.831274 25211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:54.821448 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:54.833683 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:54.833971 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:54.834275 25941 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
I20260709 13:10:54.826596 25285 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:54.832793 25244 raft_consensus.cc:2393] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 19: Already voted for candidate 46de6919f85341c6a9b574dff112e35a in this term.
I20260709 13:10:54.839566 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:54.832683 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:54.844640 25320 raft_consensus.cc:2393] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 20: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:54.846266 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:54.849370 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:54.830780   430 raft_consensus.cc:493] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:54.847077   456 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:54.831112 32696 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:54.853868   456 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:54.854548 32696 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:54.855166 25358 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:54.856038   456 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:54.856298 25358 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:54.856518 32696 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:54.853716   430 raft_consensus.cc:515] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:54.857914 32380 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:54.858224   301 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:54.858305 32380 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:54.858836   357 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:10:54.859403   357 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:54.859623   430 leader_election.cc:290] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:54.859786   357 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:54.858911 32380 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:54.866437 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:54.868276 25238 raft_consensus.cc:2955] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Committing config change with OpId 18.13: config changed from index 11 to 13, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:54.858947 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:54.873034   364 consensus_queue.cc:1048] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:10:54.876804   450 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:54.877374   450 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:54.891388 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:54.845995   431 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:54.907403   431 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:54.910158   431 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:54.912163   450 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:54.843245   422 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:54.913280   422 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:54.915717   422 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:54.834685 32762 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:54.847365   464 raft_consensus.cc:2804] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:54.919929 25314 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 21. Current term is 22. Ops: []
I20260709 13:10:54.920403 25320 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:10:54.846338   474 raft_consensus.cc:493] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
W20260709 13:10:54.924819 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 99160.3us
I20260709 13:10:54.930953 25908 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:54.934458 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:54.945611 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:54.946209 25387 raft_consensus.cc:3060] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:54.948541 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:54.948947 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:54.847144 32726 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.077s
I20260709 13:10:54.958839 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:54.928918   423 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:54.935375   464 raft_consensus.cc:697] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:54.954244   422 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:54.951246   431 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:54.945384 32500 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:55.014129 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 135271us
I20260709 13:10:54.985095 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:55.015137 25393 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:10:54.933465   474 raft_consensus.cc:515] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:54.988600 32558 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:54.991362   423 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:54.993806   424 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:54.996196   481 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:54.997320   422 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:54.999320   418 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:55.000860 25941 raft_consensus.cc:1275] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:10:55.005541   427 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:55.016808 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:55.017478 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 171258us
I20260709 13:10:55.020673 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:55.027374 25242 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 18.
I20260709 13:10:55.027812 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:55.036126 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 178130us
I20260709 13:10:55.025224   424 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:10:55.040660 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 108532us
I20260709 13:10:55.041369 25908 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:55.043576 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:55.044485 25395 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 23.
I20260709 13:10:55.028816 32696 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:55.028788   481 raft_consensus.cc:697] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:55.062572 25387 raft_consensus.cc:2468] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:55.019307   431 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:55.025344   423 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.030920   418 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.018676 32500 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.031105   427 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.080444 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:55.080538 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:55.052508   480 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:55.082268 25242 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
I20260709 13:10:55.082849 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:55.091919 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:55.050310 32726 consensus_queue.cc:1048] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 13:10:55.093811 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f406a4d41bb9441bba0a83c1d6a56aa3" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:55.064337   482 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:10:55.096009 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:55.099958 25287 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:55.101454 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:55.102412   474 leader_election.cc:290] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:55.106356   418 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:55.063911   301 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.030s
W20260709 13:10:55.108150   479 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 13:10:55.071028   464 consensus_queue.cc:237] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:55.099646   482 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:55.117421   482 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:55.118530   423 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:55.083808   480 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:55.122037   480 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:55.126778   431 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:55.130831 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 48058.1us
I20260709 13:10:55.133725 25358 leader_election.cc:400] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 election: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 23. Current term is 24.
I20260709 13:10:55.134116 25358 leader_election.cc:403] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
W20260709 13:10:55.134927 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 59596.9us
I20260709 13:10:55.140637 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:55.141465 25312 raft_consensus.cc:2393] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 20: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:55.141326 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:55.142449 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:55.142603   366 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:10:55.143085 25321 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 22.
I20260709 13:10:55.143146   366 raft_consensus.cc:697] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:55.144402   423 consensus_queue.cc:1059] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:10:55.144254   366 consensus_queue.cc:237] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.145304   423 raft_consensus.cc:3055] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:10:55.145426 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:55.145756   423 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:55.149078 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:55.147307 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 47422.9us
I20260709 13:10:55.137333 25909 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:55.168869 25287 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:55.155251 25209 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:10:55.147508 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 48478.7us
I20260709 13:10:55.098547   445 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:55.172376 25359 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:55.171792 25386 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41830 (request call id 1645) took 1176 ms. Trace:
I20260709 13:10:55.112675 32380 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:55.132823   468 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:55.178968 25287 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:55.145018   481 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 16.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:55.200482 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 68747.8us
W20260709 13:10:55.179792   485 log.cc:927] Time spent T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 13:10:55.175413 25386 rpcz_store.cc:276] 0709 13:10:53.987814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:53.988638 (+   824us) service_pool.cc:224] Handling call
0709 13:10:53.989074 (+   436us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:54.036408 (+ 47334us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:10:54.036456 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:54.722877 (+686421us) spinlock_profiling.cc:243] Waited 64.6 ms on lock 0x7b2c000e1828. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f29755aba11 00007f29755bb372 00007f2976ceda01 00007f2976cb243a 00007f2976cb183e 00007f297bc76419 00007f29751edbd4 00007f29751edb62 00007f29751eda91 00007f29751eda0c 00007f29751eccc2
0709 13:10:54.723290 (+   413us) log.cc:844] Serialized 26 byte log entry
0709 13:10:54.887645 (+164355us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:10:54.887726 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:54.887869 (+   143us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:55.128073 (+240204us) raft_consensus.cc:1685] finished
0709 13:10:55.128105 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:55.163863 (+ 35758us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":180642,"raft.queue_time_us":47441,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":76631,"spinlock_wait_cycles":148589440,"thread_start_us":163851,"threads_started":1,"wal-append.queue_time_us":222270}
I20260709 13:10:55.198820   489 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:55.214900 25941 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:10:55.214730 25909 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:55.214640 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:55.177897   474 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:55.189045 32500 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:55.200686   492 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:10:55.231348   481 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:55.234956 25320 raft_consensus.cc:606] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:55.216100 32558 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:55.224370   463 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:10:55.171196   423 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.190388   357 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:55.183018   445 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.242828   492 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:55.260916   481 consensus_queue.cc:1048] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.002s
I20260709 13:10:55.216403   488 consensus_queue.cc:1048] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.046s
I20260709 13:10:55.252009   463 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:55.260634   422 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:55.292658   445 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:55.293900 32380 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:55.266444   492 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:55.300176   502 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:55.300966   502 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:55.303778   502 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:55.304028 25393 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 24.
I20260709 13:10:55.305429 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:55.305855 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:55.279037   463 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:10:55.306870 25243 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 25 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 13:10:55.310125 32645 raft_consensus.cc:2749] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211) with higher term. Message: Invalid argument: T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 23. Current term is 24.
I20260709 13:10:55.315287 32762 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:55.318775   480 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.258132   424 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:55.323764   357 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:55.391315 25238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:55.391973   465 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:55.392204   499 raft_consensus.cc:493] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:55.392570   465 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:55.392772   499 raft_consensus.cc:515] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.394898   499 leader_election.cc:290] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:55.395735   464 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:55.395905 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:55.396267   499 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:55.396478 25320 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 21.
I20260709 13:10:55.396718   430 raft_consensus.cc:493] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:55.397384   422 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:55.397279   430 raft_consensus.cc:515] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.397924   422 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.399360   430 leader_election.cc:290] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:55.394825   465 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:55.400014   422 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:55.400895 25285 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:55.403206 32726 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:55.405524 32762 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:10:55.282279   423 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:10:55.409366 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 155162us
I20260709 13:10:55.410820 25358 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:55.411761 25209 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:55.412940 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:55.343253   480 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:55.423802 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:55.424710 25393 raft_consensus.cc:2393] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 22: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:55.425966 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:55.415695   484 log.cc:927] Time spent T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.402s	user 0.000s	sys 0.005s
I20260709 13:10:55.396984   499 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.438241 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:55.441886 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:55.430991 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 252086us
I20260709 13:10:55.396237   464 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:55.410486   427 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:55.447288 25386 raft_consensus.cc:2368] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 20. Current term is 21.
I20260709 13:10:55.454303 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:55.548592   379 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.028s	sys 0.077s
W20260709 13:10:55.538539 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 91767.1us
W20260709 13:10:55.453835 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 224507us
I20260709 13:10:55.539106 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:55.564466 25393 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 23.
W20260709 13:10:55.512192 25209 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:55.432716   500 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:10:55.564615 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:55.445288   317 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:10:55.566238 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 312547us
I20260709 13:10:55.578227 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:55.568104   379 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.028s	sys 0.081s
W20260709 13:10:55.582419 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 339410us
I20260709 13:10:55.588496 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:55.592136 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:55.595796 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:55.598290 25386 raft_consensus.cc:1240] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Rejecting Update request from peer 46de6919f85341c6a9b574dff112e35a for earlier term 15. Current term is 16. Ops: []
I20260709 13:10:55.574533   465 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:55.567714   500 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:55.578065 32665 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:55.609413 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:55.588394   445 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:55.613260 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:55.593845 32380 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:55.645500 25314 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:10:55.650672 25312 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
I20260709 13:10:55.654017 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:55.591109   502 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:55.609772   500 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:55.650520 25314 raft_consensus.cc:606] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:55.605002   481 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:55.664001 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:55.664407 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:55.665907 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:55.664759 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 59502.7us
I20260709 13:10:55.668897 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:55.669523 25937 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:55.669754 25941 raft_consensus.cc:1275] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 13:10:55.673640   503 log.cc:927] Time spent T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
W20260709 13:10:55.676312   513 log.cc:927] Time spent T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:10:55.618309 32665 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.669437 25312 raft_consensus.cc:2955] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Committing config change with OpId 18.13: config changed from index 11 to 13, f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:55.681833 25937 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 22.
I20260709 13:10:55.682762 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:55.684594 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:55.608670   465 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:55.687624 25908 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:55.683269   482 consensus_meta.cc:287] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.565s	user 0.008s	sys 0.004s
I20260709 13:10:55.723119 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:55.724915 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:55.724927 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:55.725679 25938 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 in current term 19: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:55.721704 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:55.727331 25317 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:55.728351 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:55.728956 25942 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 23.
I20260709 13:10:55.645871   445 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.663260 32380 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.721879 32637 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:55.734630 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:55.685779 32665 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:55.736181 25945 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 13:10:55.740059 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:55.744601 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:55.720901   488 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:55.749253 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:55.750250 25240 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 24 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:55.751098 25240 raft_consensus.cc:2410] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 24 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 17.
I20260709 13:10:55.751966 25287 leader_election.cc:304] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:55.749058 25209 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:55.723100   499 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:55.762332 25358 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:55.762877   511 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:55.763814 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:55.764575 25246 raft_consensus.cc:3060] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:10:55.671943   481 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.758767 25284 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:55.668336   502 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.781157 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:55.769518 25938 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:55.757658   369 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:55.769457 32726 raft_consensus.cc:2749] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:55.761234   487 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:55.731130   482 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.836416 25246 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
W20260709 13:10:55.848124 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 85840.2us
I20260709 13:10:55.850241 25358 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:55.791035   505 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:55.759424 32637 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:55.736177   492 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:55.864158 25244 raft_consensus.cc:2368] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 19. Current term is 20.
I20260709 13:10:55.786458   488 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.887334   482 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:55.892918   492 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:55.844934   487 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.878481   505 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:55.875942   369 raft_consensus.cc:697] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 19 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:55.916556   487 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:55.916287   369 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.908124   445 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:55.922367   463 consensus_meta.cc:287] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: Time spent flushing consensus metadata: real 0.615s	user 0.005s	sys 0.004s
I20260709 13:10:55.922900   463 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.893282   519 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:55.923943   519 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:55.911495   502 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:55.911980   488 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:55.912513   481 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:55.913960   465 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:55.914048 32380 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:55.925079   463 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:55.926182   519 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:55.908785 32637 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:55.934999 25317 raft_consensus.cc:2468] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 21.
I20260709 13:10:55.938876   500 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:55.942505   500 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:55.943288   423 consensus_meta.cc:287] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a: Time spent flushing consensus metadata: real 0.527s	user 0.008s	sys 0.002s
I20260709 13:10:55.944252 25238 raft_consensus.cc:2410] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for term 22 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 13:10:55.955220   378 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:55.957643   378 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:56.017535 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 408045us
I20260709 13:10:56.022228 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:56.026697 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 158892us
I20260709 13:10:56.047295 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 24 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.048204 25392 raft_consensus.cc:2410] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 24 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 17.
I20260709 13:10:56.049111 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d45c0f8b003848989966aa931c6a7c11" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.022423 32672 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:56.073191   505 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:56.068545 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.069010 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:56.067226 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.096517 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.098030 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.103235 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.103562 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd399e2f6384e98ada57177a6010d14" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:56.105104   529 log.cc:927] Time spent T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:10:56.108249 25284 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:56.095947   369 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:56.112589   369 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.114920   369 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:56.119299   502 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:56.120054   502 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.101244   463 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:56.121608   463 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:56.129537   378 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:56.133054 32726 consensus_queue.cc:1048] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:10:56.134052 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:56.134868 25239 raft_consensus.cc:2410] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 23 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 13:10:56.135761 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:56.140228 25389 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 13:10:56.140749 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:56.142369 32762 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:10:56.149408 32762 raft_consensus.cc:697] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 22 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:56.150434 32762 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.153024 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.156476 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.157076 25389 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 24.
I20260709 13:10:56.159072   464 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:56.167609 25911 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:56.175041 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 64227.4us
I20260709 13:10:56.170257 25359 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:56.181384 32380 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:10:56.186419   536 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:10:56.186708   519 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:56.190613   519 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.187214   536 raft_consensus.cc:493] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:56.198187 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:56.198899 25242 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
W20260709 13:10:56.211985 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 462999us
I20260709 13:10:56.215909 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:56.216499 25320 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:56.208604 25245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32994 (request call id 3175) took 1129 ms. Trace:
I20260709 13:10:56.193791 25359 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:56.201081 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.219404 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:56.220458 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:56.217700 25245 rpcz_store.cc:276] 0709 13:10:55.079079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:55.083383 (+  4304us) service_pool.cc:224] Handling call
0709 13:10:55.083703 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:10:55.083932 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:10:55.435312 (+351380us) log.cc:844] Serialized 14 byte log entry
0709 13:10:55.586086 (+150774us) log.cc:844] Serialized 14 byte log entry
0709 13:10:56.208154 (+622068us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b1800258be0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b3fc0a 00007f297d27f219 00007f2973bc7a40 00007f2973bc24e4 00007f2973bc2455 00007f2976caa060 00007f2976cbc716 00007f2976ccd287 00007f2976ccd208
0709 13:10:56.208290 (+   136us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:10:56.208334 (+    44us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:10:56.208380 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:56.208441 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:56.208547 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":613258,"prepare.queue_time_us":118196,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":5267,"raft.queue_time_us":624569,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":3217,"spinlock_wait_cycles":14608384,"thread_start_us":481177,"threads_started":2,"wal-append.queue_time_us":231075}
I20260709 13:10:56.224150 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:56.226794 25245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:56.212316   536 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:10:56.232302 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:56.227002   366 raft_consensus.cc:2804] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:10:56.236168 25212 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:56.236785   514 log.cc:927] Time spent T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.559s	user 0.002s	sys 0.000s
I20260709 13:10:56.222014   492 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:10:56.244618 25388 raft_consensus.cc:2393] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 22: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:56.250248 25212 leader_election.cc:400] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 20. Current term is 21.
I20260709 13:10:56.250615 25212 leader_election.cc:403] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 13:10:56.255879 25212 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:56.270310   516 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a for earlier term 20. Current term is 21.
I20260709 13:10:56.258245   422 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:56.249051   492 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:56.285988 25212 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:56.261067   366 raft_consensus.cc:697] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:56.281145   500 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:56.297601 25320 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 20.
I20260709 13:10:56.286970   492 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:10:56.285293   519 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:56.301501 25209 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
W20260709 13:10:56.352531 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 51158.6us
I20260709 13:10:56.353402   494 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:10:56.354022   494 raft_consensus.cc:697] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 20 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:56.354997   494 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:56.361874   551 log.cc:927] Time spent T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 13:10:56.366731 32696 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:56.367393 32696 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.370220 32696 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:56.371971 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.374171   366 consensus_queue.cc:237] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:56.388304 32672 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:56.350257   516 consensus_queue.cc:1059] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
I20260709 13:10:56.393772 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.304684   423 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:10:56.395803   423 raft_consensus.cc:3055] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:10:56.396198   423 raft_consensus.cc:740] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:56.397063   423 consensus_queue.cc:260] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:56.398272   423 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:56.401331 25237 raft_consensus.cc:1240] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 21. Current term is 22. Ops: []
I20260709 13:10:56.402429 25313 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 22.
I20260709 13:10:56.403621 25209 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:56.405311 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.406616 25393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.412308 25238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32992 (request call id 3160) took 1100 ms. Trace:
I20260709 13:10:56.417404 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.412765 25238 rpcz_store.cc:276] 0709 13:10:55.301595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:55.342683 (+ 41088us) service_pool.cc:224] Handling call
0709 13:10:56.402290 (+1059607us) spinlock_profiling.cc:243] Waited 552 ms on lock 0x7b54003a98d4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cb62fa 00007f297bc77b17 00007f29751f26d4 00007f29751f2662 00007f29751f2591 00007f29751f250c 00007f29751f17c2 00007f2974fed48c 00007f2974fec8c6 00007f2974fec26f
0709 13:10:56.402429 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1269231872}
W20260709 13:10:56.468668 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 62424.8us
I20260709 13:10:56.476511 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:56.388903 32672 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.351635   316 raft_consensus.cc:493] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:56.464075 25287 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:56.414105   482 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:10:56.465881   431 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:56.492555 25287 leader_election.cc:304] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:56.497723 25287 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:56.411791   536 raft_consensus.cc:515] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.503324 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:56.502009   535 raft_consensus.cc:2749] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:56.497715   316 raft_consensus.cc:515] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.494493   474 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:10:56.526461 25287 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:56.497848   482 raft_consensus.cc:697] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 22 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:56.533388 25287 leader_election.cc:304] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:56.525532   544 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:56.530501   535 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:56.497007   431 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.498157   492 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.547533   492 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:56.555505 25387 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:10:56.565004 32672 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:56.567265   378 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:56.567423   482 consensus_queue.cc:237] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:56.568132   378 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.571286   378 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:56.535477   549 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:56.577659   527 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:10:56.539116   468 raft_consensus.cc:2749] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:10:56.588436   536 leader_election.cc:290] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 25 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:56.588629   464 raft_consensus.cc:993] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567: : Instructing follower 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:10:56.607219 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.592564 25244 raft_consensus.cc:1275] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:10:56.578368 32665 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:56.603432   492 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:56.626433 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b5c9f65605b4dfca1bbfe0f09629c6d" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.582217   316 leader_election.cc:290] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:56.639822   536 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:56.640513   536 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.613026 32672 raft_consensus.cc:1081] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Signalling peer 46de6919f85341c6a9b574dff112e35a to start an election
I20260709 13:10:56.614099   423 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:56.649904   460 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:56.653986   460 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.656157   460 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:56.656720   431 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:56.657912   500 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:56.666531 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:56.667147 25390 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:10:56.644330   492 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:56.651063   509 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:56.674954 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:56.675695 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:56.646934 32665 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.663964   486 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
W20260709 13:10:56.685541 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 146896us
I20260709 13:10:56.687163 25938 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 13:10:56.692780 25909 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:56.694306 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:56.696991   369 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:56.677659   500 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.699966   536 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:56.700587   536 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:10:56.704867 25390 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 24.
I20260709 13:10:56.681254   509 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.709393   492 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:56.711097   486 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.660641   423 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.714141 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.714686   423 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:56.725317   486 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:56.726163   502 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:56.748553 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 61960.9us
I20260709 13:10:56.755117 25287 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:56.759202 25287 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:56.761117 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:56.762138 25242 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:10:56.759954 25246 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615"
dest_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:32992
I20260709 13:10:56.765555 25246 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:10:56.765882 25246 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:56.783263 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:56.782346 25246 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:56.794361   418 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:10:56.799935 25285 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:56.801575 32672 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:56.801964 32672 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:56.802559   463 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:56.802774 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 43656.6us
I20260709 13:10:56.803153   463 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:10:56.786444   527 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.025s
I20260709 13:10:56.763403 32726 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:56.813221 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:56.813766 25391 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:10:56.815966 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.818302 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.802326 32672 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:56.826962 25287 leader_election.cc:304] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:56.823634 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 128367us
I20260709 13:10:56.828328 25909 leader_election.cc:400] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for earlier term 19. Current term is 20.
I20260709 13:10:56.829212 25246 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:56.829808 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:56.830742 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98122796f52e40968d1d8ccd7b032b32" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:56.826237   506 consensus_queue.cc:1059] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.007s
I20260709 13:10:56.833227 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:56.833799 25943 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:10:56.833873   506 raft_consensus.cc:3055] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:10:56.835605   506 raft_consensus.cc:740] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:56.836436   500 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:56.836753   431 raft_consensus.cc:2749] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:10:56.836402   506 consensus_queue.cc:260] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:56.837965   506 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:56.839895 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:56.840682 25313 raft_consensus.cc:2410] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a77ecf2c0941618b5bc79d4c04db61 for term 23 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 13:10:56.831576   527 consensus_queue.cc:1048] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.012s
I20260709 13:10:56.853688 25943 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 24.
I20260709 13:10:56.858318 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:56.860832 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:56.861665 25317 raft_consensus.cc:2393] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 24: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:10:56.862818 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.864876 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:56.866900 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e29e9ab7eda4242964fa76e29c2a221" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:56.863425 25392 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:56.869228 25237 pending_rounds.cc:85] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a: Aborting all ops after (but not including) 15
I20260709 13:10:56.870178 25317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:56.872253 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.872452 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.872259 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:56.874830 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46ab4fd91d3f4766a7b97848f5244360" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:56.869529 25237 pending_rounds.cc:107] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a: Aborting uncommitted NO_OP operation due to leader change: 21.16
I20260709 13:10:56.877489 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898e00cfe03c4fc68dc9d6e92a55e996" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:56.877758 25237 raft_consensus.cc:2889] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:56.887506 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:56.909698 25212 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:56.911160 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:56.912071 25943 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 19: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:56.913242 25284 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:56.914345 25284 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:56.915835 25284 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:56.918092 32672 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.919132 25320 raft_consensus.cc:1240] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 18. Current term is 19. Ops: []
I20260709 13:10:56.919485 32665 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:56.920482   516 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:10:56.921626   463 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:10:56.923091   463 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:10:56.923802   463 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:56.921793 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.921033   516 raft_consensus.cc:697] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 24 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:56.928226   516 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:56.932350 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:56.932070 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:56.924970   463 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:10:56.938350 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:56.941726 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:56.942203 25238 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:10:56.943176 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:56.943557 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:56.944592 25241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:56.922609   509 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:56.945287 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:56.947806 25358 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:56.950282 25941 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:56.950649 25942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:56.950910 25941 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 22.
I20260709 13:10:56.951861 25939 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 13:10:56.960861 25358 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:56.962309 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:56.965978 25391 raft_consensus.cc:2468] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 25.
I20260709 13:10:56.967305 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:10:56.939791   533 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.180s	user 0.003s	sys 0.000s
I20260709 13:10:56.971673 25386 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 25 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 13:10:56.982407 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:56.983196 25937 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:10:56.984895 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:56.991432 25243 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 13:10:56.992682 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:56.999104 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:56.999686 25319 raft_consensus.cc:3055] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:10:56.999979 25319 raft_consensus.cc:740] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:57.003419 32672 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:56.981619   504 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:57.004241   317 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:10:57.008424 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:57.008808 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: true dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:57.009241 25244 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:57.009418 25395 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:57.000694 25319 consensus_queue.cc:260] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:57.022300 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:57.023444 25319 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:10:56.971143   451 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:10:57.024338 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:57.020884 25312 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:10:56.990352   511 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.038s
I20260709 13:10:57.028988 25209 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:57.030215   494 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:57.030903 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:57.030159 25209 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:57.032171 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:57.032855 32696 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:57.033257 32696 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting leader election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:57.033612 32696 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:57.034518 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:57.035676 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2730c370dd9a4a1ea5adfb543e3ba120" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:57.025398 25246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:57.038874   492 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:10:57.044154 25320 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:10:57.051707 25358 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:57.057345 25319 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 22.
I20260709 13:10:57.060212 25908 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:57.063871 25395 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 22.
I20260709 13:10:57.064502 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:57.065022 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:56.958478   463 raft_consensus.cc:697] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 24 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:57.066004 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:57.066402   463 consensus_queue.cc:237] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:57.068341   566 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 13:10:57.068485   494 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:10:57.069043   508 consensus_queue.cc:1059] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 13:10:57.069751 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:10:57.070892   564 log.cc:927] Time spent T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 13:10:57.076373 25908 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:57.077164   460 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:10:57.088781 25937 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 24.
I20260709 13:10:57.089869   480 consensus_queue.cc:1059] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:10:57.090751   480 raft_consensus.cc:3055] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:10:57.090517 25908 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:57.091133   480 raft_consensus.cc:740] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:57.091655   369 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:57.092000   480 consensus_queue.cc:260] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:57.094393 25244 raft_consensus.cc:2468] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:57.094652   480 raft_consensus.cc:3060] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:10:57.095429 25911 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:57.096534 32696 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:57.098733 32696 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:57.101310 25908 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:57.109464 25395 raft_consensus.cc:1240] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 21. Current term is 22. Ops: []
I20260709 13:10:57.110847 25944 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 13:10:57.078346   511 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:10:57.082616   509 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:57.113770 25358 leader_election.cc:304] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:57.121464 25287 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:57.104794   533 log.cc:927] Time spent T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 13:10:57.103617   519 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:57.126358 25319 raft_consensus.cc:1275] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:10:57.126469 25287 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:57.128369 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:57.129696 25287 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:57.132365 25312 raft_consensus.cc:1240] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 24 LEADER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 23. Current term is 24. Ops: []
I20260709 13:10:57.133106 25320 raft_consensus.cc:606] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 24 LEADER]: Received request to transfer leadership to TS a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:57.129685 25212 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:57.134791   463 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:57.135128 32696 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:10:57.135205 32672 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:10:57.135706 32696 raft_consensus.cc:697] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 22 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:57.145275 25319 raft_consensus.cc:1240] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 19. Current term is 20. Ops: []
I20260709 13:10:57.150897 25321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36864 (request call id 3081) took 1982 ms. Trace:
I20260709 13:10:57.156462 25321 rpcz_store.cc:276] 0709 13:10:55.150138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:55.150307 (+   169us) service_pool.cc:224] Handling call
0709 13:10:55.173233 (+ 22926us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:10:55.407964 (+234731us) spinlock_profiling.cc:243] Waited 2.14 ms on lock 0x7b180034d740. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f297685a4bc 00007f2973ba848b 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc2455 00007f2976caa060 00007f2976caabfb 00007f2976cb564b
0709 13:10:55.408103 (+   139us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:10:55.408401 (+   298us) log.cc:844] Serialized 14 byte log entry
0709 13:10:55.902737 (+494336us) log.cc:844] Serialized 14 byte log entry
0709 13:10:55.930200 (+ 27463us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b180034d740. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:10:55.930967 (+   767us) spinlock_profiling.cc:243] Waited 20.2 ms on lock 0x7b180034d740. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:10:56.740351 (+809384us) spinlock_profiling.cc:243] Waited 30 ms on lock 0x7b180034d740. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973b3fc0a 00007f297d27f219 00007f2973bc7a40 00007f2973bc24e4 00007f2973bc2455 00007f297a6aa3c7 00007f2976cbc9a7 00007f2976ccc0bf 00007f2976ccc038
0709 13:10:56.740557 (+   206us) log.cc:844] Serialized 14 byte log entry
0709 13:10:56.740757 (+   200us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:10:56.764248 (+ 23491us) log.cc:844] Serialized 26 byte log entry
0709 13:10:56.764445 (+   197us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:10:56.770723 (+  6278us) log.cc:844] Serialized 14 byte log entry
0709 13:10:56.793924 (+ 23201us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:10:56.794015 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:10:56.837528 (+ 43513us) spinlock_profiling.cc:243] Waited 5.84 ms on lock 0x7b180034d740. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc5522 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:10:56.925035 (+ 87507us) spinlock_profiling.cc:243] Waited 3.69 ms on lock 0x7b2400124ec8. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973a4b9c4 00007f2973a49acb 00007f2973c04dc9 00007f2973c04a55 00007f2973c0409d 00007f297683610e 00007f297684d88c 00007f297684cbb6 00007f297684c7d8 00007f2976856221
0709 13:10:57.131826 (+206791us) raft_consensus.cc:1685] finished
0709 13:10:57.131853 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:10:57.132016 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":903339,"prepare.queue_time_us":924964,"prepare.run_cpu_time_us":162,"prepare.run_wall_time_us":4482,"raft.queue_time_us":95359,"raft.run_cpu_time_us":537,"raft.run_wall_time_us":29463,"spinlock_wait_cycles":168858880,"thread_start_us":546215,"threads_started":2,"wal-append.queue_time_us":635477}
I20260709 13:10:57.136596 32696 consensus_queue.cc:237] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:57.130824   508 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:10:57.172374   508 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:57.149982   504 raft_consensus.cc:2749] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:57.162889 25911 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:57.174883 25911 leader_election.cc:304] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567
I20260709 13:10:57.176645   489 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:57.177369   489 raft_consensus.cc:2804] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:10:57.178896   489 raft_consensus.cc:697] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 25 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:57.179824   489 consensus_queue.cc:237] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:57.172891   508 raft_consensus.cc:3060] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:10:57.188678   416 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:57.189313   416 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:57.192664 25246 raft_consensus.cc:1240] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Rejecting Update request from peer a2a77ecf2c0941618b5bc79d4c04db61 for earlier term 16. Current term is 17. Ops: []
I20260709 13:10:57.194307   416 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:57.204744 25144 catalog_manager.cc:5697] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 18 to 22, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 22 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:57.211907 25144 catalog_manager.cc:5697] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:57.214874   494 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:10:57.226969 25139 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39582 (request call id 310) took 51285 ms (client timeout 15000 ms). Trace:
I20260709 13:10:57.165016   481 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:57.238102 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
W20260709 13:10:57.232867 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 198397us
I20260709 13:10:57.244442 25209 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:57.246304 25285 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:57.250834 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:57.239979   569 log.cc:927] Time spent T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.117s	user 0.004s	sys 0.000s
I20260709 13:10:57.256484 25320 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a"
dest_uuid: "2329e428ebbf484fa3ea9452133df567"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:49638
I20260709 13:10:57.260380 25320 raft_consensus.cc:606] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:57.256780   511 consensus_queue.cc:1059] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.004s
I20260709 13:10:57.260733 25320 raft_consensus.cc:614] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:10:57.263418   502 raft_consensus.cc:3055] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:10:57.263839   502 raft_consensus.cc:740] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:57.265440   509 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:57.265974   509 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:57.267478   519 consensus_queue.cc:1059] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:10:57.268155   509 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:57.268389   519 raft_consensus.cc:3055] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:10:57.268805   519 raft_consensus.cc:740] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:57.269994   423 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:57.269763   519 consensus_queue.cc:260] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:57.271404   519 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:10:57.286655 32672 consensus_queue.cc:1059] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:10:57.288189 32380 consensus_queue.cc:1048] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:10:57.241957 32645 consensus_queue.cc:1059] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260709 13:10:57.290664 32645 raft_consensus.cc:3055] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:10:57.291028 32645 raft_consensus.cc:740] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:57.291930 32645 consensus_queue.cc:260] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:57.234236   494 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:10:57.245355   567 log.cc:927] Time spent T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.195s	user 0.000s	sys 0.004s
W20260709 13:10:57.303128 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 56897.7us
I20260709 13:10:57.248500 25144 catalog_manager.cc:5697] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 11 to 15, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 15 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:57.299317 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:10:57.296463   570 log.cc:927] Time spent T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
W20260709 13:10:57.231534 25139 rpcz_store.cc:269] 0709 13:10:05.941727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:05.941989 (+   262us) service_pool.cc:224] Handling call
0709 13:10:57.226856 (+51284867us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:10:53.105283 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:10:53.311132 (+205849us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:10:53.328574 (+ 17442us) write_op.cc:435] Acquired schema lock
0709 13:10:53.328601 (+    27us) tablet.cc:662] Decoding operations
0709 13:10:53.443081 (+114480us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:10:53.459176 (+ 16095us) write_op.cc:641] Partition lock acquired for write op
0709 13:10:53.462775 (+  3599us) tablet.cc:685] Acquiring locks for 6 operations
0709 13:10:53.606256 (+143481us) tablet.cc:701] Row locks acquired
0709 13:10:53.606304 (+    48us) write_op.cc:260] PREPARE: finished
0709 13:10:53.606488 (+   184us) write_op.cc:270] Start()
0709 13:10:53.606639 (+   151us) write_op.cc:276] Timestamp: P: 1783602653606428 usec, L: 0
0709 13:10:53.606676 (+    37us) op_driver.cc:348] REPLICATION: starting
0709 13:10:53.769089 (+162413us) log.cc:844] Serialized 2472 byte log entry
0709 13:10:54.667509 (+898420us) op_driver.cc:464] REPLICATION: finished
0709 13:10:55.009996 (+342487us) spinlock_profiling.cc:243] Waited 2.64 ms on lock 0x7b18002c25a0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
0709 13:10:55.010250 (+   254us) write_op.cc:301] APPLY: starting
0709 13:10:55.010370 (+   120us) tablet.cc:1367] starting BulkCheckPresence
0709 13:10:55.322044 (+311674us) tablet.cc:1370] finished BulkCheckPresence
0709 13:10:55.322095 (+    51us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:10:55.595555 (+273460us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:10:55.720697 (+125142us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=12,key_file_lookups=12,delta_file_lookups=12,mrs_lookups=0
0709 13:10:55.730116 (+  9419us) write_op.cc:312] APPLY: finished
0709 13:10:55.844660 (+114544us) log.cc:844] Serialized 65 byte log entry
0709 13:10:55.997243 (+152583us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:10:56.020459 (+ 23216us) write_op.cc:454] Released schema lock
0709 13:10:56.046332 (+ 25873us) write_op.cc:341] FINISH: Updating metrics
0709 13:10:56.100601 (+ 54269us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:10:56.102362 (+  1761us) write_op.cc:454] Released schema lock
0709 13:10:56.551584 (+449222us) spinlock_profiling.cc:243] Waited 5.23 ms on lock 0x7b18002c25a0. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297ca8dde1 00007f297ca9a5b1 00007f2973bc53f3 00007f2973bc77c1 00007f2973bc7779 00007f2973bc7709 00007f2973bc76d1 00007f2973bc69cd 000000000038f354 000000000038f199 00007f2973ba883a
Metrics: {"mutex_wait_us":5017928,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":155801,"apply.run_cpu_time_us":30550,"apply.run_wall_time_us":1069400,"cfile_cache_hit":26,"cfile_cache_hit_bytes":29183,"num_ops":6,"prepare.queue_time_us":310255,"prepare.run_cpu_time_us":98336,"prepare.run_wall_time_us":1031197,"raft.queue_time_us":219428,"raft.run_cpu_time_us":21725,"raft.run_wall_time_us":182616,"replication_time_us":1027073,"spinlock_wait_cycles":18088576,"thread_start_us":728565,"threads_started":5,"wal-append.queue_time_us":716874,"wal-append.run_cpu_time_us":32071,"wal-append.run_wall_time_us":1279534}]]}
I20260709 13:10:57.334061 25144 catalog_manager.cc:5697] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 14 to 18, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 18 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:57.336251 25144 catalog_manager.cc:5697] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 13 to 21, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 21 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:57.307472   418 raft_consensus.cc:2804] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:57.309916 32645 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:57.348959 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cff7b4cb7184e4c9a69372d5901e19a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:57.248179   481 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:57.359432 25944 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679"
dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
mode: GRACEFUL
new_leader_uuid: "46de6919f85341c6a9b574dff112e35a"
 from {username='slave'} at 127.0.0.1:45428
I20260709 13:10:57.359993 25937 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:10:57.360025 25944 raft_consensus.cc:606] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Received request to transfer leadership to TS 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:57.362273 25944 raft_consensus.cc:614] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:10:57.354615 25144 catalog_manager.cc:5697] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 16 to 17, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 17 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:57.364571 25144 catalog_manager.cc:5697] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 11 to 20, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 20 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:57.365540 25285 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:57.368243 25908 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:57.376297 25144 catalog_manager.cc:5697] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:57.355939   418 raft_consensus.cc:697] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:57.389460   418 consensus_queue.cc:237] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:57.373772   460 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:10:57.369875 32726 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.035s
I20260709 13:10:57.379408   463 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:57.424839 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:10:57.399705   431 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:57.428555 25144 catalog_manager.cc:5697] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 11 to 19, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 19 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:10:57.429651 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 177193us
I20260709 13:10:57.430377 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:10:57.439313 25909 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:57.443082 25909 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:57.445282 25320 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 13:10:57.445029 25909 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:57.453600 25909 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:57.456558 25909 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:57.457372 25144 catalog_manager.cc:5697] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:57.465727 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:57.466562 25243 raft_consensus.cc:2410] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 13:10:57.431413   431 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:57.468230 25209 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:57.449229   486 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:57.478652   486 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:10:57.452363   460 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:57.435614   492 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.028s
I20260709 13:10:57.462952 32665 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:10:57.462082   565 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:57.460331 25144 catalog_manager.cc:5697] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 12 to 20. New cstate: current_term: 20 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:57.490769 25909 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:57.476061   482 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:57.491900 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:57.491780   482 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:57.493436   502 consensus_queue.cc:260] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:57.494787   502 raft_consensus.cc:3060] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:10:57.498581   576 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 13:10:57.475817   511 consensus_queue.cc:1243] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer 2329e428ebbf484fa3ea9452133df567 log is divergent from this leader: its last log entry 24.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 13:10:57.499783   508 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:57.501833 25321 raft_consensus.cc:2410] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 13:10:57.502492 25144 catalog_manager.cc:5697] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 14 to 21, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 21 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:57.506762 25909 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:57.521312 32696 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:57.524385   578 consensus_queue.cc:1059] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:10:57.532657 25395 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 13:10:57.542616 25144 catalog_manager.cc:5697] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:57.558599   582 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.012s
I20260709 13:10:57.550208   460 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:10:57.509119   480 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:10:57.573282   431 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:57.574853   508 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:57.584144   505 raft_consensus.cc:3055] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:10:57.584632   505 raft_consensus.cc:740] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:57.589390 25144 catalog_manager.cc:5697] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 8 to 18, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 18 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:57.593706 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:57.594695 25244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:57.602669 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:57.603516 25944 raft_consensus.cc:2393] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 19: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:10:57.604838 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:57.605119 25244 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:57.600163 25144 catalog_manager.cc:5697] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 19 to 20, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 20 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:57.610809   482 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:10:57.614301 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 53702.3us
I20260709 13:10:57.617581 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:57.585564   505 consensus_queue.cc:260] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:57.621657 25943 raft_consensus.cc:1240] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 25 LEADER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 13:10:57.623608 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:57.624809 25387 raft_consensus.cc:2410] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 13:10:57.624007 25209 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:57.629307 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:57.600265   481 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:57.648034 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:57.651880 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:57.652608   492 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:10:57.662786 25908 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:10:57.669453 25287 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:57.670601   565 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:57.679638   494 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:57.680840 25246 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Rejecting Update request from peer 2329e428ebbf484fa3ea9452133df567 for earlier term 22. Current term is 23. Ops: []
W20260709 13:10:57.681655 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 57740.1us
I20260709 13:10:57.683720   492 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:10:57.680074 25387 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 25 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 13:10:57.686375   565 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:10:57.688392 25937 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 15. (term mismatch)
I20260709 13:10:57.688808 25937 pending_rounds.cc:85] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 14
I20260709 13:10:57.689007 25937 pending_rounds.cc:107] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 19.15
I20260709 13:10:57.689240 25937 raft_consensus.cc:2889] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:57.689405   505 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:10:57.691371 25944 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 13:10:57.694582 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:57.698148 25392 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:10:57.706665 25321 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:10:57.700596   431 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:10:57.713806 25943 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 13:10:57.715391 32726 consensus_queue.cc:1059] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.003s
I20260709 13:10:57.706115   423 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.011s
I20260709 13:10:57.705780   482 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.015s
I20260709 13:10:57.702114   492 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:10:57.714589 25144 catalog_manager.cc:5697] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 11 to 14, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 14 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:57.732923 25287 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:57.734392 25392 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
W20260709 13:10:57.778893 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 46886.3us
W20260709 13:10:57.784395 25209 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 44201.6us
I20260709 13:10:57.786855 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:57.835516   582 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:10:57.837703   582 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:10:57.754323   418 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.086s
I20260709 13:10:57.841571   494 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:57.843153   543 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:57.843693   543 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:57.845762   543 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:57.846470 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:57.779150   527 raft_consensus.cc:3055] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:10:57.847139 25237 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:57.847285   527 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:57.848169   527 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:57.849355   527 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:57.849817 25285 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:57.851924   543 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:10:57.852295   543 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:10:57.852630   543 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:10:57.854893 25390 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 15. (term mismatch)
I20260709 13:10:57.855261 25390 pending_rounds.cc:85] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Aborting all ops after (but not including) 14
I20260709 13:10:57.855468 25390 pending_rounds.cc:107] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Aborting uncommitted NO_OP operation due to leader change: 23.15
I20260709 13:10:57.855693 25390 raft_consensus.cc:2889] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:57.783646 32726 raft_consensus.cc:2804] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:10:57.858579   418 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:10:57.866050 25313 raft_consensus.cc:1275] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 25 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 16. (term mismatch)
I20260709 13:10:57.866417 25313 pending_rounds.cc:85] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: Aborting all ops after (but not including) 15
I20260709 13:10:57.866616 25313 pending_rounds.cc:107] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260709 13:10:57.866834 25313 raft_consensus.cc:2889] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:57.869048 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:57.869843 25390 raft_consensus.cc:2393] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 16: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:57.874848 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:57.862610 25144 catalog_manager.cc:5697] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 16 to 18, now has a pending config: VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65), VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66), VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67), NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 18 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } } pending_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: NON_VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: true } } }
I20260709 13:10:57.884058 25144 catalog_manager.cc:5697] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 10 to 18, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 18 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:57.889160 25144 catalog_manager.cc:5697] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 15 to 19, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 19 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:57.889729 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:10:57.890614 25144 catalog_manager.cc:5697] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:57.873308 32726 raft_consensus.cc:697] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:57.905822 25390 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 13:10:57.907335 25144 catalog_manager.cc:5697] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 19 to 22, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 22 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:57.909543 32726 consensus_queue.cc:237] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:57.877737   480 consensus_queue.cc:1048] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.030s
I20260709 13:10:57.931558 25144 catalog_manager.cc:5697] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:57.916577 32696 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:10:57.929512   482 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:57.941696   482 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:57.943912   482 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:57.933204 25144 catalog_manager.cc:5697] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 18 to 19, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 19 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:10:57.966086 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:57.966509 25939 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 19 index: 16. (term mismatch)
I20260709 13:10:57.980891 25939 pending_rounds.cc:85] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 15
I20260709 13:10:57.981294 25939 pending_rounds.cc:107] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 18.16
I20260709 13:10:57.981614 25939 raft_consensus.cc:2889] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:10:57.984167 25144 catalog_manager.cc:5697] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:57.986553 25144 catalog_manager.cc:5697] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 10 to 18. New cstate: current_term: 18 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:58.009003 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:58.009985 25390 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 22.
I20260709 13:10:58.012008 25212 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567
W20260709 13:10:58.077486 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 61379.2us
W20260709 13:10:58.080858 25212 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 68976.8us
W20260709 13:10:58.073148   592 log.cc:927] Time spent T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 13:10:58.079511   451 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:57.997964   423 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.082s
W20260709 13:10:58.085444   589 log.cc:927] Time spent T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:10:58.003614   463 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.085s
I20260709 13:10:58.087116   500 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:10:58.101493   451 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:58.129387   451 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:58.153729 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:58.105001   500 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:58.163275 25242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:58.163935 25242 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
W20260709 13:10:58.168880   594 log.cc:927] Time spent T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
W20260709 13:10:58.188591   595 log.cc:927] Time spent T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.109s	user 0.000s	sys 0.002s
W20260709 13:10:58.193856   599 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
W20260709 13:10:58.201472   605 log.cc:927] Time spent T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:10:58.204487 25359 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:58.206295   450 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:10:58.206676   450 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:58.207031   450 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:58.208544   543 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:58.159699   500 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:58.223749   450 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:58.226303 25312 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
I20260709 13:10:58.237700   500 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:58.239652   543 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:10:58.249243 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 88522.5us
W20260709 13:10:58.252203 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 79202.1us
W20260709 13:10:58.261458 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 95055.1us
W20260709 13:10:58.263353 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 58970.4us
I20260709 13:10:58.281608 25243 raft_consensus.cc:1275] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
W20260709 13:10:58.290164 25287 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:10:58.302155 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:58.321411 25387 raft_consensus.cc:2368] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17.
W20260709 13:10:58.335192 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 43979.4us
I20260709 13:10:58.349390   431 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:58.396082 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:58.396829 25321 raft_consensus.cc:3060] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:10:58.404835 25243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:58.397816   450 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:58.427209 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:58.375025   431 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:58.447257   431 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:58.449497   500 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:58.439594   583 consensus_queue.cc:1048] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.005s
W20260709 13:10:58.442466   603 log.cc:927] Time spent T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 13:10:58.457988 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:58.458657 25387 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:10:58.462391 25319 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Advancing to term 23
I20260709 13:10:58.463445 25321 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 17.
I20260709 13:10:58.469099 25237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a"
I20260709 13:10:58.475090   563 raft_consensus.cc:993] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: : Instructing follower a2a77ecf2c0941618b5bc79d4c04db61 to start an election
I20260709 13:10:58.488247   563 raft_consensus.cc:1081] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 24 LEADER]: Signalling peer a2a77ecf2c0941618b5bc79d4c04db61 to start an election
I20260709 13:10:58.469434 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:58.493608 25287 leader_election.cc:400] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17.
I20260709 13:10:58.493974 25287 leader_election.cc:403] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 13:10:58.495131   431 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661) with higher term. Message: Invalid argument: T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 16. Current term is 17.
I20260709 13:10:58.501617 25358 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:58.504331   504 raft_consensus.cc:2804] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:10:58.504833   504 raft_consensus.cc:697] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:58.505733   504 consensus_queue.cc:237] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:58.476532 25243 raft_consensus.cc:2468] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 16.
I20260709 13:10:58.484107 25945 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:10:58.485255 25239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46de6919f85341c6a9b574dff112e35a" is_pre_election: true
I20260709 13:10:58.509794 25239 raft_consensus.cc:2368] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 22. Current term is 23.
I20260709 13:10:58.486207 25240 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:10:58.533663 25285 leader_election.cc:400] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 22. Current term is 23.
I20260709 13:10:58.534044 25285 leader_election.cc:403] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:10:58.534556 25390 raft_consensus.cc:1275] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:10:58.536113 25387 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 23.
I20260709 13:10:58.510177 25319 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 23.
I20260709 13:10:58.540223 25212 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:10:58.545007   494 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:10:58.545608   494 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 23 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:10:58.546484   494 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:58.526283   543 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.014s
I20260709 13:10:58.557480 25387 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa"
dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
 from {username='slave'} at 127.0.0.1:60946
I20260709 13:10:58.558305 25387 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:10:58.558779 25387 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:10:58.537817   502 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313) with higher term. Message: Invalid argument: T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 for earlier term 22. Current term is 23.
I20260709 13:10:58.570700 25387 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 25 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:58.573201 25387 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 25 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:58.585054 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: true dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:58.562870   431 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 13:10:58.604883 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:10:58.607013   463 consensus_queue.cc:1048] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 13:10:58.626935 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 283783us
W20260709 13:10:58.654881 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 117335us
I20260709 13:10:58.666518 25313 raft_consensus.cc:3055] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:10:58.667147 25313 raft_consensus.cc:740] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:10:58.667924 25313 consensus_queue.cc:260] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:58.671051 25313 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:10:58.672508 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de96ce5d28144cf3894e100bfc4959fa" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: true dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:58.673146 25939 raft_consensus.cc:3060] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:10:58.685679 25313 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 25.
W20260709 13:10:58.737468   624 log.cc:927] Time spent T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 13:10:58.744447 25358 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:10:58.754204 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 50324.5us
I20260709 13:10:58.757992 25939 raft_consensus.cc:2468] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 25.
I20260709 13:10:58.764007   610 raft_consensus.cc:2804] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 25 FOLLOWER]: Leader election won for term 25
/apache/dev/git/kudu/src/kudu/master/auto_leader_rebalancer-test.cc:278: Failure
Failed
Bad status: Illegal state: leader_transfer_task size should be 0, but 6
I20260709 13:10:58.799252   610 raft_consensus.cc:697] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 25 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:58.810104   610 consensus_queue.cc:237] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:58.820796   607 log.cc:927] Time spent T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.084s	user 0.002s	sys 0.000s
/apache/dev/git/kudu/src/kudu/util/test_util.cc:402: Failure
Failed
Timed out waiting for assertion to pass.
I20260709 13:10:58.864856 24113 tablet_server.cc:179] TabletServer@127.23.140.65:0 shutting down...
W20260709 13:10:58.869819 25909 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 22 similar messages]
W20260709 13:10:58.892639 25285 consensus_peers.cc:597] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:58.889483 25359 consensus_peers.cc:597] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:58.924966 25285 consensus_peers.cc:597] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:58.925623 25909 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 55823.6us
I20260709 13:10:58.948203 25391 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 13:10:58.950837   482 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.014s
I20260709 13:10:58.978284 25144 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53264 (request call id 211) took 8719 ms. Trace:
I20260709 13:10:58.979831   451 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:58.980389   451 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:58.982991   451 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:58.978868 25144 rpcz_store.cc:276] 0709 13:10:50.259028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:50.259252 (+   224us) service_pool.cc:224] Handling call
0709 13:10:58.978208 (+8718956us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:10:58.431383 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:10:58.431995 (+   612us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:10:58.432035 (+    40us) write_op.cc:435] Acquired schema lock
0709 13:10:58.432054 (+    19us) tablet.cc:662] Decoding operations
0709 13:10:58.435279 (+  3225us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:10:58.435360 (+    81us) write_op.cc:641] Partition lock acquired for write op
0709 13:10:58.435387 (+    27us) tablet.cc:685] Acquiring locks for 25 operations
0709 13:10:58.506266 (+ 70879us) tablet.cc:701] Row locks acquired
0709 13:10:58.506288 (+    22us) write_op.cc:260] PREPARE: finished
0709 13:10:58.506443 (+   155us) write_op.cc:270] Start()
0709 13:10:58.506574 (+   131us) write_op.cc:276] Timestamp: P: 1783602658506398 usec, L: 0
0709 13:10:58.506599 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 13:10:58.507323 (+   724us) log.cc:844] Serialized 9981 byte log entry
0709 13:10:58.776348 (+269025us) op_driver.cc:464] REPLICATION: finished
0709 13:10:58.791878 (+ 15530us) write_op.cc:301] APPLY: starting
0709 13:10:58.793223 (+  1345us) tablet.cc:1367] starting BulkCheckPresence
0709 13:10:58.826650 (+ 33427us) tablet.cc:1370] finished BulkCheckPresence
0709 13:10:58.826739 (+    89us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:10:58.841415 (+ 14676us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:10:58.841764 (+   349us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=50,key_file_lookups=50,delta_file_lookups=50,mrs_lookups=0
0709 13:10:58.841814 (+    50us) write_op.cc:312] APPLY: finished
0709 13:10:58.842281 (+   467us) log.cc:844] Serialized 220 byte log entry
0709 13:10:58.843089 (+   808us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:10:58.843304 (+   215us) write_op.cc:454] Released schema lock
0709 13:10:58.843370 (+    66us) write_op.cc:341] FINISH: Updating metrics
0709 13:10:58.853799 (+ 10429us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:10:58.853831 (+    32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1440735,"reported_tablets":31,"child_traces":[["op",{"apply.queue_time_us":15241,"apply.run_cpu_time_us":30203,"apply.run_wall_time_us":61893,"cfile_cache_hit":102,"cfile_cache_hit_bytes":108527,"num_ops":25,"prepare.queue_time_us":63957,"prepare.run_cpu_time_us":11495,"prepare.run_wall_time_us":131100,"raft.queue_time_us":96398,"raft.run_cpu_time_us":7289,"raft.run_wall_time_us":15697,"replication_time_us":269598,"spinlock_wait_cycles":59648,"thread_start_us":198127,"threads_started":4,"wal-append.queue_time_us":111560}]]}
I20260709 13:10:58.984408 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:58.985092 25937 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 16.
W20260709 13:10:58.986831 25285 consensus_peers.cc:597] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:10:58.992712 25136 catalog_manager.cc:5697] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 10 to 19, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 19 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:10:58.995185 25136 catalog_manager.cc:5697] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 17 to 20. New cstate: current_term: 20 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:10:58.997092 25136 catalog_manager.cc:5697] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
W20260709 13:10:58.991811 25359 consensus_peers.cc:597] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.007714 25285 consensus_peers.cc:597] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:10:59.003342 25358 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:59.016348 25136 catalog_manager.cc:5697] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:59.017292   504 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:59.037597 25136 catalog_manager.cc:5697] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 5 to 22, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 22 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:10:59.040328 25136 catalog_manager.cc:5697] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 19 to 20, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 20 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:10:59.062556 25909 consensus_peers.cc:597] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.063375 25909 consensus_peers.cc:597] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:10:59.023998   504 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:59.066259   504 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:10:59.085520 25136 catalog_manager.cc:5697] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 18 to 19, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 19 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
W20260709 13:10:59.116003 25359 consensus_peers.cc:597] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.117791 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 96976.7us
I20260709 13:10:59.116292 25136 catalog_manager.cc:5697] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 15 to 21, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 21 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:10:59.125722 25316 raft_consensus.cc:1275] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:10:59.132190 25136 catalog_manager.cc:5697] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 17 to 21. New cstate: current_term: 21 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:59.149320 25316 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
W20260709 13:10:59.142820 25285 consensus_peers.cc:597] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:10:59.128733   364 consensus_queue.cc:1048] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.010s
W20260709 13:10:59.162286 25359 consensus_peers.cc:597] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.166175 25359 leader_election.cc:336] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:10:59.173458 25285 consensus_peers.cc:597] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.174669 25285 consensus_peers.cc:597] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:10:59.155165   492 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.015s
I20260709 13:10:59.178582 25136 catalog_manager.cc:5697] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 13 to 21, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 21 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
W20260709 13:10:59.175287 25285 consensus_peers.cc:597] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.209259 25285 consensus_peers.cc:597] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.207823 25359 consensus_peers.cc:597] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:10:59.208257   618 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:59.219895 25136 catalog_manager.cc:5697] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 9 to 16. New cstate: current_term: 16 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:10:59.234230   618 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:59.256053 25136 catalog_manager.cc:5697] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 19 to 21, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 21 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:10:59.258170 25136 catalog_manager.cc:5697] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 15 to 17, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 17 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:10:59.280754 25285 consensus_peers.cc:597] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.303457 25285 consensus_peers.cc:597] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:10:59.305550   618 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:10:59.351629 25359 consensus_peers.cc:597] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:10:59.359088 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:59.359768 25312 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 23.
W20260709 13:10:59.363798 25285 consensus_peers.cc:597] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:10:59.373739 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:10:59.369845 25909 consensus_peers.cc:597] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.375685 25909 consensus_peers.cc:597] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.381474 25909 consensus_peers.cc:597] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:10:59.382625   369 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:10:59.389501   369 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:59.389873   369 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:10:59.382082 25909 leader_election.cc:336] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:10:59.410315 25319 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 25 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 13:10:59.411566 25285 consensus_peers.cc:597] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.419111 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 49483.8us
W20260709 13:10:59.425350 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 68569us
W20260709 13:10:59.431289 25359 consensus_peers.cc:597] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.436960 25359 consensus_peers.cc:597] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.462302 25359 consensus_peers.cc:597] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.463853 25359 consensus_peers.cc:597] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:10:59.465191 25939 raft_consensus.cc:1275] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 13:10:59.466804 25359 consensus_peers.cc:597] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.470955 25359 consensus_peers.cc:597] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.480635 25359 consensus_peers.cc:597] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:10:59.471597   500 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:10:59.485904 25359 consensus_peers.cc:597] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:10:59.472406   504 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:59.494947 25359 consensus_peers.cc:597] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.495278 25285 consensus_peers.cc:597] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:10:59.496008   579 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.027s
W20260709 13:10:59.496119   638 log.cc:927] Time spent T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
I20260709 13:10:59.489023   500 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:59.534209 25359 consensus_peers.cc:597] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:10:59.527146 32380 consensus_queue.cc:1048] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.013s
I20260709 13:10:59.550041   369 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:59.551151   500 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:59.553172 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:59.553817 25937 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 16.
I20260709 13:10:59.554857 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:10:59.555641 25388 raft_consensus.cc:2393] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 18: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:59.559211   504 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:59.564079 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:59.564692 25937 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 16 FOLLOWER]: Advancing to term 18
I20260709 13:10:59.566884 25209 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59; no voters: a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:10:59.568367   369 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:10:59.569254   482 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:10:59.571007   482 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:59.571393   482 raft_consensus.cc:3060] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:10:59.572664 25909 leader_election.cc:336] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:10:59.575518 25937 raft_consensus.cc:2468] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 18.
I20260709 13:10:59.581086   482 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:59.583874 25358 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:10:59.592001 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:10:59.610424 25319 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 23.
I20260709 13:10:59.598301   579 raft_consensus.cc:2804] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:10:59.613561 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
W20260709 13:10:59.607903 25359 consensus_peers.cc:597] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.615600 25359 consensus_peers.cc:597] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.616377 25359 consensus_peers.cc:597] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:10:59.617085 25359 leader_election.cc:336] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:10:59.630636   494 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:10:59.631222   494 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:59.620561   369 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:10:59.669737   369 raft_consensus.cc:697] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 23 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:10:59.686033   369 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:10:59.692355 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 74943.5us
I20260709 13:10:59.717739   494 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:59.724071   482 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Requested vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:59.704383   579 raft_consensus.cc:697] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:10:59.787452 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:10:59.788786 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:59.789412 25316 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 23.
I20260709 13:10:59.798112 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: 
I20260709 13:10:59.800995 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:10:59.802129 25939 raft_consensus.cc:2393] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 46de6919f85341c6a9b574dff112e35a in current term 18: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:10:59.805784 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:59.801944   423 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:10:59.799882   482 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:10:59.835879   423 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:10:59.844691   482 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:10:59.908913   482 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:10:59.851717   579 consensus_queue.cc:237] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:59.861291   423 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:10:59.917771   639 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:10:59.918342   639 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:59.918589   494 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:10:59.919299   494 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:10:59.920349   639 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:59.931921 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:59.933473 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:59.955869 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:10:59.958307   494 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:10:59.967423 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:10:59.992143 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:10:59.992470 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 150095us
W20260709 13:10:59.994138 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 48580.7us
I20260709 13:10:59.998962 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.016593   639 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:00.047237   639 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:00.010815   492 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:00.049453   492 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:00.052815 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.010135   423 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.004225   659 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:00.055823   659 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.056334   423 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 election: Requested vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:00.058130   659 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:00.061026 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.062300 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:11:00.062880 25313 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:11:00.065430 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:11:00.001632   482 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:11:00.003746 32696 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:11:00.076069 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.083612 32696 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.087762 32696 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:00.078217   423 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:00.097545   423 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.099903   423 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:00.077555   659 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:00.117874 25313 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 24.
I20260709 13:11:00.122583   492 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:00.123225   492 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:11:00.132001 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.132769 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.116110   482 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:00.147694   500 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:00.152074   500 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:00.154206   500 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:00.149072   423 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:00.155263   423 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.159834   423 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:00.162843   482 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.172122   482 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:00.174930   639 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:00.146695   492 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.197110   492 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:00.202947 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.203860 25319 raft_consensus.cc:1240] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Rejecting Update request from peer f646dfd532974dceadfe00005d2bbf59 for earlier term 23. Current term is 24. Ops: []
I20260709 13:11:00.207245 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.211450 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.212939 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.214213 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.216370 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.220548 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.221593 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.222157 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.222369 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.220553 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.226629   642 consensus_queue.cc:1059] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.008s
I20260709 13:11:00.227530   618 raft_consensus.cc:3055] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:11:00.206144   639 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:00.227901   618 raft_consensus.cc:740] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:11:00.228232   639 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:00.228788   618 consensus_queue.cc:260] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.230084   618 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:11:00.230490   639 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:00.241873   423 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:00.242638   673 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:00.242529   423 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.243219   673 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:00.245314   482 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:00.245918   482 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.247902   482 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:00.248378   482 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:11:00.258745 32696 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:00.265429   492 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:00.265861 32696 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.266965   423 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:00.268779 32696 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:00.266084   492 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.270987   492 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:11:00.273389 25136 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39596 (request call id 289) took 11858 ms (client timeout 15000 ms). Trace:
I20260709 13:11:00.273581 25209 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.277722   659 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:11:00.278373   659 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.279002   492 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:11:00.279121   673 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:00.279583   492 raft_consensus.cc:697] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 24 LEADER]: Becoming Leader. State: Replica: 46de6919f85341c6a9b574dff112e35a, State: Running, Role: LEADER
I20260709 13:11:00.280339   659 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:00.280421   492 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.276299 25387 consensus_queue.cc:237] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.277783 32696 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:00.284485   659 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:00.284435 32696 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:00.285012   659 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:00.281018   673 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:00.285840 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.286556 32696 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:00.286953   659 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:00.287132 32696 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:11:00.285864   673 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:00.288772 32696 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:00.289420 32696 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.290040   673 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:00.248840   482 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.286978 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.295960 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.297335 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.299237 32696 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:11:00.273864 25136 rpcz_store.cc:269] 0709 13:10:48.415169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:48.422394 (+  7225us) service_pool.cc:224] Handling call
0709 13:11:00.273331 (+11850937us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:10:59.452130 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:10:59.452755 (+   625us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:10:59.452793 (+    38us) write_op.cc:435] Acquired schema lock
0709 13:10:59.452816 (+    23us) tablet.cc:662] Decoding operations
0709 13:10:59.454065 (+  1249us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:10:59.454153 (+    88us) write_op.cc:641] Partition lock acquired for write op
0709 13:10:59.454187 (+    34us) tablet.cc:685] Acquiring locks for 13 operations
0709 13:10:59.455358 (+  1171us) tablet.cc:701] Row locks acquired
0709 13:10:59.455380 (+    22us) write_op.cc:260] PREPARE: finished
0709 13:10:59.455529 (+   149us) write_op.cc:270] Start()
0709 13:10:59.455651 (+   122us) write_op.cc:276] Timestamp: P: 1783602659455480 usec, L: 0
0709 13:10:59.455676 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 13:10:59.456423 (+   747us) log.cc:844] Serialized 5246 byte log entry
0709 13:10:59.548110 (+ 91687us) op_driver.cc:464] REPLICATION: finished
0709 13:10:59.677896 (+129786us) write_op.cc:301] APPLY: starting
0709 13:10:59.678025 (+   129us) tablet.cc:1367] starting BulkCheckPresence
0709 13:10:59.911470 (+233445us) tablet.cc:1370] finished BulkCheckPresence
0709 13:10:59.911605 (+   135us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:11:00.051797 (+140192us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:11:00.052052 (+   255us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=26,key_file_lookups=26,delta_file_lookups=26,mrs_lookups=0
0709 13:11:00.052096 (+    44us) write_op.cc:312] APPLY: finished
0709 13:11:00.052394 (+   298us) log.cc:844] Serialized 121 byte log entry
0709 13:11:00.077939 (+ 25545us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:11:00.088407 (+ 10468us) write_op.cc:454] Released schema lock
0709 13:11:00.093237 (+  4830us) write_op.cc:341] FINISH: Updating metrics
0709 13:11:00.093696 (+   459us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:11:00.093731 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1517,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":112162,"apply.run_cpu_time_us":20291,"apply.run_wall_time_us":421308,"cfile_cache_hit":54,"cfile_cache_hit_bytes":58415,"num_ops":13,"prepare.queue_time_us":109738,"prepare.run_cpu_time_us":4943,"prepare.run_wall_time_us":26396,"replication_time_us":92027,"thread_start_us":118178,"threads_started":2}]]}
I20260709 13:11:00.302538   482 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:00.314589 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.315961 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.317068 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.318176 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.317214 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.322089 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.327615 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.327901 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.328346 25939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.328888 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.320050 32696 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:11:00.330262 25945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.330816 25944 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.332594 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.332921 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.333209 25940 raft_consensus.cc:1275] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:11:00.300639   673 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:11:00.299659   672 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:11:00.336423 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.335531   659 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:00.341907 32696 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:00.342491 32696 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:00.344506 32696 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:00.345391   677 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:00.341987   659 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.345995   677 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:00.347270   579 consensus_queue.cc:1048] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:11:00.347626   659 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:00.348096   675 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:00.349251   675 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.351965   675 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:00.348300   672 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:00.353034   672 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.353219 25143 catalog_manager.cc:5697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 13 to 15, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 15 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:11:00.355439   672 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:00.344967 32696 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:00.356478 32696 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:00.344933 25209 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.360538 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.361589   677 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:00.362689 32696 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:00.363232   668 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:11:00.364495   668 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:00.366176 25209 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.368314 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.369300   672 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:11:00.369385 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.370045   673 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:00.370249 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.370762 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.370594   673 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:00.372313 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.372707   673 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:00.372927 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.374056 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.376724 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.371686 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.379200   675 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:00.378556 32696 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:00.379890   675 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:00.378556 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.380131 32696 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:00.383603 32696 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:00.386818 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:11:00.387799 25359 consensus_peers.cc:597] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:00.383641   675 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:00.389627 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.391921 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.398389 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.399492 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.405063   579 raft_consensus.cc:2955] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Committing config change with OpId 18.14: config changed from index 8 to 14, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:11:00.411739 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.413475 25209 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.414354   673 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:11:00.415035   673 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:11:00.415299   672 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:00.415889   672 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.418115   672 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:00.419101 25937 raft_consensus.cc:1275] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:11:00.422742 25937 raft_consensus.cc:2955] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Committing config change with OpId 18.14: config changed from index 8 to 14, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:11:00.423887 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.425504 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.428247 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.428464 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.428810 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.432103 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.435791 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.442080 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.442947 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.443817 25212 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.448109 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.433066   672 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:00.455153 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.455760 25212 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.453655   659 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:11:00.445302   674 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:00.445478   492 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:00.462261   492 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:00.464460   492 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:00.466564 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:00.473313 25212 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:00.435277   673 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:00.477308   672 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:00.490778   672 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:00.492218   673 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:00.494482   673 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:00.496273 25131 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 37bfef8566b143cf94971f26f417678c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:11:00.436743   675 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:00.500393 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.501454 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.501559 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.504256   675 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:00.504688 25212 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:00.506350 25212 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:00.507439 25212 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.508589 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.508843   674 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:11:00.509563 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.509753   674 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:11:00.510516 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.512224 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.508641   659 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:00.509845 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.513255   673 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:11:00.513957 25212 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:00.514633 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.513540 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.498979   492 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:11:00.520184 25359 consensus_peers.cc:597] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:00.523324   675 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:00.524066 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:00.540521 25212 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:00.537051   659 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:11:00.552253 25212 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.550679   673 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:11:00.558653 25212 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.559015   492 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:11:00.559844 25212 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:00.560611 32696 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:11:00.561853 25212 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:00.562726   672 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:11:00.563453   672 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:00.565986 25212 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:00.566881   672 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:11:00.568034 25212 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:00.568742   672 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:00.570899 25212 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:00.571657   672 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:00.581310 32380 consensus_queue.cc:1048] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.002s
I20260709 13:11:00.593881 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.595187 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.595894 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.596230 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.597510 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.597528 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.597986 25212 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.603041   659 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:11:00.605428 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:00.603978 25212 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:00.609294 25212 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.611171 25212 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:00.615480 25212 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.621292   675 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:11:00.610287   659 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:11:00.615247   673 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:11:00.628724 25212 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:00.630528   673 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:11:00.645727 25212 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:00.617599   492 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:00.662072 25212 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:00.664040   492 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:11:00.665422 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:00.669806 25212 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:00.671154 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:00.670665   659 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:00.679565 32696 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:11:00.664801   673 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:11:00.678457   672 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:00.689931 25212 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:00.694875   672 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:11:00.763096 25144 catalog_manager.cc:5697] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 23 to 25, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 25 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:11:00.765022 25144 catalog_manager.cc:5697] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 14 to 19. New cstate: current_term: 19 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:11:00.794430 25144 catalog_manager.cc:5697] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:11:00.795856 25144 catalog_manager.cc:5697] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:11:00.798321 25144 catalog_manager.cc:5697] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 18 to 20, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 20 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:11:00.812434 25144 catalog_manager.cc:5697] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 20 to 24, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 24 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:11:00.814647 25144 catalog_manager.cc:5697] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:11:00.817685 25144 catalog_manager.cc:5697] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:11:00.815919 25143 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39582 (request call id 311) took 26522 ms (client timeout 15000 ms). Trace:
I20260709 13:11:00.819509 25144 catalog_manager.cc:5697] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 18 to 19, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 19 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:11:00.822268 25937 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 17. (term mismatch)
I20260709 13:11:00.822652 25937 pending_rounds.cc:85] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: Aborting all ops after (but not including) 16
I20260709 13:11:00.822048 25144 catalog_manager.cc:5697] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 22 to 25. New cstate: current_term: 25 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:11:00.822852 25937 pending_rounds.cc:107] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: Aborting uncommitted NO_OP operation due to leader change: 23.17
I20260709 13:11:00.823141 25937 raft_consensus.cc:2889] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:11:00.823529 25144 catalog_manager.cc:5697] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:11:00.824653   673 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 13:11:00.825019 25144 catalog_manager.cc:5697] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index -1 to 13, VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) added, no longer has a pending config: VOTER 46de6919f85341c6a9b574dff112e35a (127.23.140.65), VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66), VOTER a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67), NON_VOTER f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 18 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:11:00.826903 25144 catalog_manager.cc:5697] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 14 to 17, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 17 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:11:00.818843 25143 rpcz_store.cc:269] 0709 13:10:34.311375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:34.319315 (+  7940us) service_pool.cc:224] Handling call
0709 13:10:34.767532 (+448217us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b5400470ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1c090 00007f2976cbaf3a 00007f297ca711cc 00007f297cb6f37e 00007f297b0302a4 00007f297b030232 00007f297b030161 00007f297b0300dc 00007f297b02f392 00007f2974fed48c 00007f2974fec8c6
0709 13:11:00.815852 (+26048320us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:11:00.504772 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:11:00.505366 (+   594us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:11:00.505408 (+    42us) write_op.cc:435] Acquired schema lock
0709 13:11:00.505429 (+    21us) tablet.cc:662] Decoding operations
0709 13:11:00.505783 (+   354us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:11:00.505858 (+    75us) write_op.cc:641] Partition lock acquired for write op
0709 13:11:00.505893 (+    35us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:11:00.506077 (+   184us) tablet.cc:701] Row locks acquired
0709 13:11:00.506096 (+    19us) write_op.cc:260] PREPARE: finished
0709 13:11:00.506248 (+   152us) write_op.cc:270] Start()
0709 13:11:00.506364 (+   116us) write_op.cc:276] Timestamp: P: 1783602660506198 usec, L: 0
0709 13:11:00.506391 (+    27us) op_driver.cc:348] REPLICATION: starting
0709 13:11:00.507094 (+   703us) log.cc:844] Serialized 541 byte log entry
0709 13:11:00.606331 (+ 99237us) op_driver.cc:464] REPLICATION: finished
0709 13:11:00.713424 (+107093us) write_op.cc:301] APPLY: starting
0709 13:11:00.718873 (+  5449us) tablet.cc:1367] starting BulkCheckPresence
0709 13:11:00.724076 (+  5203us) tablet.cc:1370] finished BulkCheckPresence
0709 13:11:00.724190 (+   114us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:11:00.725041 (+   851us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:11:00.725274 (+   233us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=2,mrs_lookups=0
0709 13:11:00.725333 (+    59us) write_op.cc:312] APPLY: finished
0709 13:11:00.726756 (+  1423us) log.cc:844] Serialized 25 byte log entry
0709 13:11:00.727602 (+   846us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:11:00.727703 (+   101us) write_op.cc:454] Released schema lock
0709 13:11:00.727748 (+    45us) write_op.cc:341] FINISH: Updating metrics
0709 13:11:00.744600 (+ 16852us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:11:00.744634 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":17176,"reported_tablets":49,"spinlock_wait_cycles":333086080,"child_traces":[["op",{"apply.queue_time_us":95385,"apply.run_cpu_time_us":3583,"apply.run_wall_time_us":36379,"cfile_cache_hit":6,"cfile_cache_hit_bytes":8303,"num_ops":1,"prepare.queue_time_us":147000,"prepare.run_cpu_time_us":2711,"prepare.run_wall_time_us":14409,"replication_time_us":83470,"thread_start_us":190209,"threads_started":2}]]}
I20260709 13:11:00.849164 25313 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Refusing update from remote peer 46de6919f85341c6a9b574dff112e35a: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 13:11:00.828181 25144 catalog_manager.cc:5697] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 17 to 19, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 19 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:11:00.854785 25144 catalog_manager.cc:5697] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 17 to 18, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 18 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:11:00.857002 25144 catalog_manager.cc:5697] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 18 to 22. New cstate: current_term: 22 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:11:00.867317 25144 catalog_manager.cc:5697] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 18 to 21. New cstate: current_term: 21 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:11:00.866746 32696 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.006s
W20260709 13:11:01.055716 25359 consensus_peers.cc:597] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:01.106142 24113 thread.cc:527] Waited for 1000ms trying to join with heartbeat (tid 25271)
I20260709 13:11:01.134058 25144 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53264 (request call id 212) took 1987 ms. Trace:
I20260709 13:11:01.144862 25144 rpcz_store.cc:276] 0709 13:10:59.146489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:59.146889 (+   400us) service_pool.cc:224] Handling call
0709 13:11:01.133960 (+1987071us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:11:00.902567 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:11:00.903183 (+   616us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:11:00.903226 (+    43us) write_op.cc:435] Acquired schema lock
0709 13:11:00.903246 (+    20us) tablet.cc:662] Decoding operations
0709 13:11:00.904822 (+  1576us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:11:00.904903 (+    81us) write_op.cc:641] Partition lock acquired for write op
0709 13:11:00.904937 (+    34us) tablet.cc:685] Acquiring locks for 17 operations
0709 13:11:00.906633 (+  1696us) tablet.cc:701] Row locks acquired
0709 13:11:00.906661 (+    28us) write_op.cc:260] PREPARE: finished
0709 13:11:00.906823 (+   162us) write_op.cc:270] Start()
0709 13:11:00.906966 (+   143us) write_op.cc:276] Timestamp: P: 1783602660906769 usec, L: 0
0709 13:11:00.906992 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 13:11:00.919228 (+ 12236us) log.cc:844] Serialized 6787 byte log entry
0709 13:11:00.923899 (+  4671us) op_driver.cc:464] REPLICATION: finished
0709 13:11:00.924400 (+   501us) write_op.cc:301] APPLY: starting
0709 13:11:00.924487 (+    87us) tablet.cc:1367] starting BulkCheckPresence
0709 13:11:00.929527 (+  5040us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b5400470ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cade28 00007f297a6e8581 00007f297a6e7d58 00007f297a6ec9f1 00007f297a6ec9a9 00007f297a6ec939 00007f297a6ec901 00007f297a6ebb2d 000000000038f354 000000000038f199
0709 13:11:00.934233 (+  4706us) tablet.cc:1370] finished BulkCheckPresence
0709 13:11:00.934263 (+    30us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:11:01.012043 (+ 77780us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:11:01.012534 (+   491us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=34,key_file_lookups=34,delta_file_lookups=34,mrs_lookups=0
0709 13:11:01.012610 (+    76us) write_op.cc:312] APPLY: finished
0709 13:11:01.075984 (+ 63374us) log.cc:844] Serialized 156 byte log entry
0709 13:11:01.076196 (+   212us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:11:01.076383 (+   187us) write_op.cc:454] Released schema lock
0709 13:11:01.076456 (+    73us) write_op.cc:341] FINISH: Updating metrics
0709 13:11:01.076883 (+   427us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:11:01.076911 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":24649,"reported_tablets":21,"child_traces":[["op",{"apply.queue_time_us":260,"apply.run_cpu_time_us":23338,"apply.run_wall_time_us":152409,"cfile_cache_hit":68,"cfile_cache_hit_bytes":70992,"num_ops":17,"prepare.queue_time_us":453,"prepare.run_cpu_time_us":7805,"prepare.run_wall_time_us":27128,"replication_time_us":16742,"spinlock_wait_cycles":26309504}]]}
I20260709 13:11:01.310418 25138 catalog_manager.cc:5697] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 14 to 15, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 15 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:01.348210 25138 catalog_manager.cc:5697] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 10 to 15, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 15 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
W20260709 13:11:01.373081 25909 consensus_peers.cc:597] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:01.399494 25144 catalog_manager.cc:5697] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 22 to 25, leader changed from f646dfd532974dceadfe00005d2bbf59 (127.23.140.68) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 25 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:11:01.406426 25144 catalog_manager.cc:5697] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:11:01.420419 25359 consensus_peers.cc:597] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:01.428066 25359 consensus_peers.cc:597] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:01.461735 25144 catalog_manager.cc:5697] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 22 to 24, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 24 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
I20260709 13:11:01.463671 25144 catalog_manager.cc:5697] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 reported cstate change: config changed from index 8 to 14, VOTER 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) evicted. New cstate: current_term: 18 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } } }
W20260709 13:11:01.485980 25285 consensus_peers.cc:597] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:01.496646 25285 consensus_peers.cc:597] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:01.517948 25909 consensus_peers.cc:597] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:01.540961 25285 consensus_peers.cc:597] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:01.569514 25909 consensus_peers.cc:597] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:01.590540   543 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:01.591140   543 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:01.594463   543 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:01.595579 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:11:01.601574 25285 consensus_peers.cc:597] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:01.607025 25285 leader_election.cc:336] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:11:01.607506 25285 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:11:01.606288 25359 consensus_peers.cc:597] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:01.608647 25359 consensus_peers.cc:597] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:01.609692   543 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:11:01.622615 25909 consensus_peers.cc:597] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:01.622740 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:11:01.630427 25285 consensus_peers.cc:597] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:01.655690 25310 tablet_service.cc:1558] Processing DeleteTablet for tablet 37bfef8566b143cf94971f26f417678c with delete_type TABLET_DATA_TOMBSTONED (TS 2329e428ebbf484fa3ea9452133df567 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:41870
W20260709 13:11:01.686164 25359 consensus_peers.cc:597] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:01.687592   695 tablet_replica.cc:333] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:01.688786   695 raft_consensus.cc:2243] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Raft consensus shutting down.
W20260709 13:11:01.698275 25909 consensus_peers.cc:597] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:01.701835   695 raft_consensus.cc:2272] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:01.703976 25359 consensus_peers.cc:597] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:01.705284   695 ts_tablet_manager.cc:1916] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 13:11:01.725629 25285 consensus_peers.cc:597] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:01.726444 25359 consensus_peers.cc:597] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:01.734956   695 ts_tablet_manager.cc:1929] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 18.13
I20260709 13:11:01.738867   673 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:11:01.742022 25138 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39572 (request call id 293) took 47382 ms (client timeout 15000 ms). Trace:
I20260709 13:11:01.741891   673 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:11:01.742697 25138 rpcz_store.cc:269] 0709 13:10:14.360015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:14.416548 (+ 56533us) service_pool.cc:224] Handling call
0709 13:11:01.741931 (+47325383us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:11:01.363764 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:11:01.364410 (+   646us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:11:01.364445 (+    35us) write_op.cc:435] Acquired schema lock
0709 13:11:01.364461 (+    16us) tablet.cc:662] Decoding operations
0709 13:11:01.364850 (+   389us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:11:01.364903 (+    53us) write_op.cc:641] Partition lock acquired for write op
0709 13:11:01.364929 (+    26us) tablet.cc:685] Acquiring locks for 2 operations
0709 13:11:01.365301 (+   372us) tablet.cc:701] Row locks acquired
0709 13:11:01.365316 (+    15us) write_op.cc:260] PREPARE: finished
0709 13:11:01.365466 (+   150us) write_op.cc:270] Start()
0709 13:11:01.365597 (+   131us) write_op.cc:276] Timestamp: P: 1783602661365419 usec, L: 0
0709 13:11:01.365623 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 13:11:01.366248 (+   625us) log.cc:844] Serialized 929 byte log entry
0709 13:11:01.368398 (+  2150us) op_driver.cc:464] REPLICATION: finished
0709 13:11:01.369039 (+   641us) write_op.cc:301] APPLY: starting
0709 13:11:01.369150 (+   111us) tablet.cc:1367] starting BulkCheckPresence
0709 13:11:01.370599 (+  1449us) tablet.cc:1370] finished BulkCheckPresence
0709 13:11:01.370693 (+    94us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:11:01.372229 (+  1536us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:11:01.374451 (+  2222us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=4,mrs_lookups=0
0709 13:11:01.374501 (+    50us) write_op.cc:312] APPLY: finished
0709 13:11:01.374705 (+   204us) log.cc:844] Serialized 33 byte log entry
0709 13:11:01.375486 (+   781us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:11:01.375576 (+    90us) write_op.cc:454] Released schema lock
0709 13:11:01.375620 (+    44us) write_op.cc:341] FINISH: Updating metrics
0709 13:11:01.375997 (+   377us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:11:01.376024 (+    27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":539455,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":436,"apply.run_cpu_time_us":4323,"apply.run_wall_time_us":6907,"cfile_cache_hit":8,"cfile_cache_hit_bytes":8352,"num_ops":2,"prepare.queue_time_us":135,"prepare.run_cpu_time_us":2750,"prepare.run_wall_time_us":2758,"replication_time_us":2638}]]}
I20260709 13:11:01.746948 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:01.747704 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53b1fb6477d486e98bff9549cc487df" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:01.771648 25209 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:01.744285   673 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:01.748448   695 log.cc:1199] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567: Deleting WAL directory at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0/minicluster-data/ts-1-root/wals/37bfef8566b143cf94971f26f417678c
I20260709 13:11:01.772950   672 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:11:01.768102 25285 consensus_peers.cc:597] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:01.774250 25359 consensus_peers.cc:597] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:01.756215 25137 catalog_manager.cc:5697] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 15 to 16, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 16 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:11:01.775895 25128 catalog_manager.cc:5028] TS 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): tablet 37bfef8566b143cf94971f26f417678c (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:11:01.807499 25378 tablet_service.cc:1558] Processing DeleteTablet for tablet eda233ace2a24b379fef320a5f610c7a with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 4 (current committed config index is 10)) from {username='slave'} at 127.0.0.1:44978
I20260709 13:11:01.795836 25137 catalog_manager.cc:5697] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 15 to 16, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 16 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
W20260709 13:11:01.807768 25140 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39596 (request call id 287) took 57115 ms (client timeout 15000 ms). Trace:
W20260709 13:11:01.832854 25140 rpcz_store.cc:269] 0709 13:10:04.692306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:04.692563 (+   257us) service_pool.cc:224] Handling call
0709 13:11:01.807569 (+57115006us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":117457,"reported_tablets":48}
W20260709 13:11:01.834158 25285 consensus_peers.cc:597] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:01.853511 25285 consensus_peers.cc:597] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:01.897302 25285 consensus_peers.cc:597] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:01.902185 25285 consensus_peers.cc:597] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:01.896854 25131 catalog_manager.cc:5028] TS a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): tablet eda233ace2a24b379fef320a5f610c7a (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
W20260709 13:11:01.914819 25359 consensus_peers.cc:597] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:01.938206   673 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:01.938872   673 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:01.943152 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:01.943724 25313 raft_consensus.cc:1803] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: voting while tombstoned based on last-logged opid 18.13
I20260709 13:11:01.944108 25313 raft_consensus.cc:2468] T 37bfef8566b143cf94971f26f417678c P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46de6919f85341c6a9b574dff112e35a in term 18.
W20260709 13:11:01.957749 25285 consensus_peers.cc:597] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:01.961556 25137 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39596 (request call id 288) took 39458 ms (client timeout 15000 ms). Trace:
I20260709 13:11:01.961470 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:01.963958   673 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:11:01.962144 25137 rpcz_store.cc:269] 0709 13:10:22.594274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:22.717972 (+123698us) service_pool.cc:224] Handling call
0709 13:11:01.961473 (+39243501us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:11:01.873168 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:11:01.873740 (+   572us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:11:01.873787 (+    47us) write_op.cc:435] Acquired schema lock
0709 13:11:01.873813 (+    26us) tablet.cc:662] Decoding operations
0709 13:11:01.874392 (+   579us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:11:01.874803 (+   411us) write_op.cc:641] Partition lock acquired for write op
0709 13:11:01.874833 (+    30us) tablet.cc:685] Acquiring locks for 2 operations
0709 13:11:01.875147 (+   314us) tablet.cc:701] Row locks acquired
0709 13:11:01.875166 (+    19us) write_op.cc:260] PREPARE: finished
0709 13:11:01.875303 (+   137us) write_op.cc:270] Start()
0709 13:11:01.875449 (+   146us) write_op.cc:276] Timestamp: P: 1783602661875260 usec, L: 0
0709 13:11:01.875472 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 13:11:01.876074 (+   602us) log.cc:844] Serialized 929 byte log entry
0709 13:11:01.878077 (+  2003us) op_driver.cc:464] REPLICATION: finished
0709 13:11:01.878543 (+   466us) write_op.cc:301] APPLY: starting
0709 13:11:01.878622 (+    79us) tablet.cc:1367] starting BulkCheckPresence
0709 13:11:01.879957 (+  1335us) tablet.cc:1370] finished BulkCheckPresence
0709 13:11:01.880047 (+    90us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:11:01.881425 (+  1378us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:11:01.881582 (+   157us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=4,mrs_lookups=0
0709 13:11:01.881626 (+    44us) write_op.cc:312] APPLY: finished
0709 13:11:01.881827 (+   201us) log.cc:844] Serialized 33 byte log entry
0709 13:11:01.881979 (+   152us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:11:01.882064 (+    85us) write_op.cc:454] Released schema lock
0709 13:11:01.882107 (+    43us) write_op.cc:341] FINISH: Updating metrics
0709 13:11:01.882471 (+   364us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:11:01.882496 (+    25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":114589,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":247,"apply.run_cpu_time_us":3887,"apply.run_wall_time_us":3887,"cfile_cache_hit":8,"cfile_cache_hit_bytes":8352,"num_ops":2,"prepare.queue_time_us":15452,"prepare.run_cpu_time_us":2801,"prepare.run_wall_time_us":3219,"replication_time_us":2480}]]}
I20260709 13:11:01.967412 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37bfef8566b143cf94971f26f417678c" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:01.970103 25209 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
W20260709 13:11:01.973357 25359 consensus_peers.cc:597] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:01.975934   672 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:11:01.984992   672 raft_consensus.cc:493] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:11:01.985569   672 raft_consensus.cc:515] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:01.992560   672 leader_election.cc:290] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:01.995062 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:11:01.996405 25359 consensus_peers.cc:597] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:01.997685 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b9d971fbc94215a65a2247f2e2acb1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:11:01.992702 25285 consensus_peers.cc:597] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:01.999109 25209 leader_election.cc:304] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.001480   673 raft_consensus.cc:2749] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:11:02.003829 25359 consensus_peers.cc:597] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:02.023958   672 raft_consensus.cc:493] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:11:02.024641   672 raft_consensus.cc:515] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:02.027040   672 leader_election.cc:290] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:02.029379 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:02.030664 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf0501919f944794bbab0ffae6a334e1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.032100 25209 leader_election.cc:304] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.033227   673 raft_consensus.cc:2749] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:11:02.050812 25141 catalog_manager.cc:5697] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:11:02.053938 25141 catalog_manager.cc:5697] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 18 to 19, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 19 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:11:02.055722 25141 catalog_manager.cc:5697] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:11:02.061977 25141 catalog_manager.cc:5697] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 13 to 17, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 17 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
W20260709 13:11:02.067512 25359 consensus_peers.cc:597] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:02.107164 24113 thread.cc:527] Waited for 2000ms trying to join with heartbeat (tid 25271)
I20260709 13:11:02.108157 25141 catalog_manager.cc:5697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 21 to 23, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 23 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:02.157567 25141 catalog_manager.cc:5697] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 14 to 15, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 15 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:02.170136   672 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:02.171803   672 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:11:02.180032 25142 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39572 (request call id 292) took 70265 ms (client timeout 15000 ms). Trace:
W20260709 13:11:02.180783 25142 rpcz_store.cc:269] 0709 13:09:51.914349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:09:51.914664 (+   315us) service_pool.cc:224] Handling call
0709 13:09:52.234892 (+320228us) spinlock_profiling.cc:243] Waited 4.81 ms on lock 0x7b5400470ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1c090 00007f2976cbaf3a 00007f297ca711cc 00007f297cb6f37e 00007f297b0302a4 00007f297b030232 00007f297b030161 00007f297b0300dc 00007f297b02f392 00007f2974fed48c 00007f2974fec8c6
0709 13:11:02.179691 (+69944799us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":111450,"reported_tablets":48,"spinlock_wait_cycles":11068160}
I20260709 13:11:02.157567   697 raft_consensus.cc:493] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:02.208387   673 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:02.209281   701 raft_consensus.cc:493] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:02.209460   673 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:02.211686   700 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:02.212716   673 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:02.215023   672 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:02.191184   697 raft_consensus.cc:515] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:02.215663 25141 catalog_manager.cc:5697] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 17 to 21. New cstate: current_term: 21 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:11:02.210351   701 raft_consensus.cc:515] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:02.212572   700 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:02.232630   697 leader_election.cc:290] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:02.240067   697 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
W20260709 13:11:02.236397 25359 consensus_peers.cc:597] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:02.226835   672 raft_consensus.cc:493] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:02.249006   700 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:02.257648 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:02.260690 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:02.273963 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dc23779efbf4518a6ada8bde8243a11" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.243017   697 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:02.271157   673 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:02.298672   699 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:02.299606   699 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:02.301146   701 leader_election.cc:290] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:02.301164   697 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:02.302021   672 raft_consensus.cc:515] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:02.302409   697 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:02.301877 25141 catalog_manager.cc:5697] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:11:02.303596   697 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:02.315356   672 leader_election.cc:290] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:02.316152   672 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:02.308499 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.317019   672 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:02.318656 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:02.319913   672 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:02.301916   701 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:02.335500   701 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:02.339390 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ead2e92f0614db79c9249687c4246f9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.341493   708 raft_consensus.cc:493] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:02.339416 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cfa8937a5964f5790737950fc3b294e" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.344183 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:02.342463   708 raft_consensus.cc:515] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:02.353715 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17584a19690d4010851d65378cfcbafa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.357411   673 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:02.359835   701 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:02.360123   673 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:02.363860   708 leader_election.cc:290] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:02.365386   699 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:02.306435   697 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:02.367774 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:02.304734 25141 catalog_manager.cc:5697] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a reported cstate change: term changed from 16 to 20. New cstate: current_term: 20 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:11:02.375036 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:11:02.370328 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 52836.9us
I20260709 13:11:02.381765 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.382213 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.383548   697 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:02.384768 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.372918   699 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:02.393173   699 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:02.396306   699 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:02.399042 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.401748 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.418861 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7555c0d2bb294af19f602573931ceb90" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.419283 25209 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.387833   697 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:02.422454 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.423964 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "991bcf9b91a14242837dfb4e18891e91" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.437427   700 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:11:02.422287   697 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:02.445801   697 raft_consensus.cc:493] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:02.446549   697 raft_consensus.cc:515] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:11:02.450938 25285 consensus_peers.cc:597] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:02.451298   672 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:02.451699 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:02.452049   672 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:02.452973   700 raft_consensus.cc:493] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:11:02.453790   700 raft_consensus.cc:515] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:02.454401   672 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:02.454927   710 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:02.455612   710 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:02.456672 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.457229 25319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.457933   710 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:02.463477 25209 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.471889 25209 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.456176   700 leader_election.cc:290] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:02.448902   697 leader_election.cc:290] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:02.479671   710 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:11:02.480569 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:02.483302 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
W20260709 13:11:02.491138 25359 consensus_peers.cc:597] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:02.484946 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eda233ace2a24b379fef320a5f610c7a" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.485764 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f209a6539f54de28906492f142efb6b" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.483342 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.493044   710 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:02.517658   710 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:02.485598   705 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:11:02.520311   710 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:11:02.516637 25359 consensus_peers.cc:597] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:02.541316   700 raft_consensus.cc:493] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:11:02.542165   700 raft_consensus.cc:515] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:02.544354   700 leader_election.cc:290] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:11:02.557775 25359 consensus_peers.cc:597] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:02.561671   700 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:02.563287 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:02.563664   700 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:02.568603   700 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:02.568547 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:02.581857 25212 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.582800   700 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:11:02.587568 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 78249.1us
I20260709 13:11:02.592406 25212 leader_election.cc:304] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:02.593493   713 raft_consensus.cc:2749] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:11:02.596040 25212 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.596801   713 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:11:02.605304   713 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:02.606005   713 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:02.609059   713 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:02.610529   713 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:02.611536   713 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:02.612284   700 raft_consensus.cc:493] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:02.612928   700 raft_consensus.cc:515] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:02.615193   700 leader_election.cc:290] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:02.617735 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8652aecc6d4869bb4cad9e0749f3ba" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.618311 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb389d32141944d4b8f26a17471d50fc" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.619349 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ec1834a96f447f8fe74448e8480342" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.619809 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d19448306e4052a13c08a582f4ccfa" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.620818 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e2fe177353b4d6c9de4ccf872ca06ca" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.621270 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd2d31837dd470c8bd91f719ffeb461" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.621546 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ff9a7b00f104dd294125870b06a7df4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.622305 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292e92613a634963a5de1f016eb951ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.623225 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3025c38ad5a4bdfa61cfca91ba3c409" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.625509 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1884b7159da148dbb1c19c424d667792" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.630774   700 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:02.631589   700 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:02.592074 25209 leader_election.cc:304] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.633646   697 raft_consensus.cc:2749] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:11:02.634883 25209 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.635645   697 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:02.633912   700 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:02.639135   713 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:02.643105 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:02.644433 25937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:02.645084   697 raft_consensus.cc:493] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f646dfd532974dceadfe00005d2bbf59)
I20260709 13:11:02.645166 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fa7e9050d44475381c97025c2b11711" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.645853   697 raft_consensus.cc:515] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:02.647120 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.648216   697 leader_election.cc:290] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:02.662264 25390 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1bed174e340476591fddc8217b8f644" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.663822 25394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.664100 25388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.665390 25392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.662313 25391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63c3ef75ef234051b7ae1ea395400f54" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.662616 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69fee8cad2c84e1492403c23786622a1" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.663123 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.663378 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.666426   697 raft_consensus.cc:493] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:02.671026   697 raft_consensus.cc:515] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:02.674500   697 leader_election.cc:290] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:02.677479 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dac81b348b746fd8ecd3937028554ea" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.683385 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "993e3b05a73a4403b04c3b55a5b48abe" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:02.687278 25316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.689812 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b62f4877af9742298b1e149789ca3090" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:02.694379   713 raft_consensus.cc:493] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:02.695128   713 raft_consensus.cc:515] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:02.696381 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5481170c5c74798bdfd146068f8b539" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:02.697413   713 leader_election.cc:290] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:02.697805   697 raft_consensus.cc:493] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:02.697965   713 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:02.698410   697 raft_consensus.cc:515] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:02.698597   713 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:02.700625   697 leader_election.cc:290] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:02.701004   713 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:02.716893 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.729740   713 raft_consensus.cc:493] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:02.730429   713 raft_consensus.cc:515] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:02.734205   713 leader_election.cc:290] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:02.736953 25212 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:02.741680 25212 leader_election.cc:304] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:02.742910 25212 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:02.743646   710 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:11:02.744014 25212 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.744370   710 raft_consensus.cc:2749] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:11:02.744982   705 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:11:02.745143   710 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:11:02.746623 25212 leader_election.cc:304] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:02.747439   710 raft_consensus.cc:2749] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:02.747818 25212 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:02.748600   710 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:11:02.750046 25209 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.755856 25212 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.765545 25209 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.767010 25209 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.767212   697 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:02.781873   710 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:11:02.782845   710 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:11:02.783020   700 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:11:02.784283 25212 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:02.785871   672 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:11:02.787005 25212 leader_election.cc:304] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:02.787886   672 raft_consensus.cc:2749] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:02.791893 25212 leader_election.cc:304] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
W20260709 13:11:02.792313 25359 consensus_peers.cc:597] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:02.797323   672 raft_consensus.cc:2749] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:02.798189 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.799401 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.800192 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f616cb6823f424a847dceaa5900dfc0" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.802768 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.803541 25212 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:02.804015 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f37bfaeb12446c1a84af172a6f3dde9" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.805347 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.805698 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6afb077d630246d6a315a828b4dcc108" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.812191 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c23d7a710c04baf86ab9ba1a2ddc9cf" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.817334   672 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:02.820587 25212 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.822371 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adf2f61a8d304d4eb9c30a77d7de17e4" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.824402   672 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:11:02.825426   672 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:02.826004   672 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:02.828228   672 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:02.830565 25141 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39572 (request call id 295) took 9777 ms. Trace:
I20260709 13:11:02.838099 25141 rpcz_store.cc:276] 0709 13:10:53.053662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:53.054583 (+   921us) service_pool.cc:224] Handling call
0709 13:10:53.209466 (+154883us) spinlock_profiling.cc:243] Waited 25.4 ms on lock 0x7b5400470ed4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1c090 00007f2976cbaf3a 00007f297ca711cc 00007f297cb6f37e 00007f297b0302a4 00007f297b030232 00007f297b030161 00007f297b0300dc 00007f297b02f392 00007f2974fed48c 00007f2974fec8c6
0709 13:11:02.830479 (+9621013us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:11:02.532898 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:11:02.545908 (+ 13010us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:11:02.545970 (+    62us) write_op.cc:435] Acquired schema lock
0709 13:11:02.546012 (+    42us) tablet.cc:662] Decoding operations
0709 13:11:02.547327 (+  1315us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:11:02.547446 (+   119us) write_op.cc:641] Partition lock acquired for write op
0709 13:11:02.547485 (+    39us) tablet.cc:685] Acquiring locks for 9 operations
0709 13:11:02.548342 (+   857us) tablet.cc:701] Row locks acquired
0709 13:11:02.548392 (+    50us) write_op.cc:260] PREPARE: finished
0709 13:11:02.548598 (+   206us) write_op.cc:270] Start()
0709 13:11:02.548757 (+   159us) write_op.cc:276] Timestamp: P: 1783602662548519 usec, L: 0
0709 13:11:02.548776 (+    19us) op_driver.cc:348] REPLICATION: starting
0709 13:11:02.550040 (+  1264us) log.cc:844] Serialized 3646 byte log entry
0709 13:11:02.566966 (+ 16926us) op_driver.cc:464] REPLICATION: finished
0709 13:11:02.590637 (+ 23671us) write_op.cc:301] APPLY: starting
0709 13:11:02.590751 (+   114us) tablet.cc:1367] starting BulkCheckPresence
0709 13:11:02.637254 (+ 46503us) tablet.cc:1370] finished BulkCheckPresence
0709 13:11:02.637394 (+   140us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:11:02.684137 (+ 46743us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:11:02.684469 (+   332us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=18,key_file_lookups=18,delta_file_lookups=18,mrs_lookups=0
0709 13:11:02.684519 (+    50us) write_op.cc:312] APPLY: finished
0709 13:11:02.684910 (+   391us) log.cc:844] Serialized 89 byte log entry
0709 13:11:02.685151 (+   241us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:11:02.685282 (+   131us) write_op.cc:454] Released schema lock
0709 13:11:02.685350 (+    68us) write_op.cc:341] FINISH: Updating metrics
0709 13:11:02.693803 (+  8453us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:11:02.693851 (+    48us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":56392,"reported_tablets":48,"spinlock_wait_cycles":58476288,"child_traces":[["op",{"apply.queue_time_us":23329,"apply.run_cpu_time_us":14936,"apply.run_wall_time_us":102180,"cfile_cache_hit":38,"cfile_cache_hit_bytes":41711,"num_ops":9,"prepare.queue_time_us":85877,"prepare.run_cpu_time_us":5327,"prepare.run_wall_time_us":18994,"replication_time_us":17978,"spinlock_wait_cycles":40832,"thread_start_us":63260,"threads_started":2}]]}
I20260709 13:11:02.831763 25212 leader_election.cc:304] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.843432   672 raft_consensus.cc:2749] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:11:02.843895 25212 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.846659 25135 catalog_manager.cc:5697] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 20 to 21, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 21 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:02.851060 25135 catalog_manager.cc:5697] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 17 to 21, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 21 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:11:02.862937   713 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:11:02.865298 25212 leader_election.cc:304] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.872807 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.880306 25212 leader_election.cc:304] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:02.882272 25212 leader_election.cc:304] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:02.889629 25212 leader_election.cc:304] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:02.886286   697 raft_consensus.cc:2749] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:11:02.893198 25212 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:02.887951   673 raft_consensus.cc:2749] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:11:02.894526   705 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:11:02.893793   713 raft_consensus.cc:2749] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:11:02.893021   700 raft_consensus.cc:2749] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:11:02.905426 25212 leader_election.cc:304] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:02.911685 25212 leader_election.cc:304] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:02.914999   700 raft_consensus.cc:2749] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:02.911728   673 raft_consensus.cc:2749] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:02.915583   672 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:02.916957 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbeee5054bca4ae28d33636ae50ca11f" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.921362 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:11:02.915791   700 raft_consensus.cc:493] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:02.925607 24113 tablet_replica.cc:333] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:02.925288   700 raft_consensus.cc:515] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:02.927470 24113 raft_consensus.cc:2243] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:02.928467 24113 raft_consensus.cc:2272] T 94b9d971fbc94215a65a2247f2e2acb1 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:02.923126   672 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:02.930876   672 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:02.932015 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:02.934377   700 leader_election.cc:290] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:02.936684 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:02.945385 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56914a99b2d8417592e1f08979cfadf6" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:02.945574 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5cbaabf54304de089abde51df6dc2be" candidate_uuid: "46de6919f85341c6a9b574dff112e35a" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:11:02.953751 25359 consensus_peers.cc:597] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:02.958606 24113 tablet_replica.cc:333] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:02.944984 25212 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:02.959633   700 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:02.959836 24113 raft_consensus.cc:2243] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:02.960564 24113 raft_consensus.cc:2272] T 63c3ef75ef234051b7ae1ea395400f54 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:02.962385 25212 leader_election.cc:304] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:02.963150   700 raft_consensus.cc:2749] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:11:02.964468 25212 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46de6919f85341c6a9b574dff112e35a; no voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:02.965240   700 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:11:02.983955 24113 tablet_replica.cc:333] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:02.988401 24113 raft_consensus.cc:2243] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:02.989063 24113 raft_consensus.cc:2272] T 7555c0d2bb294af19f602573931ceb90 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.059602 24113 tablet_replica.cc:333] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.060964 24113 raft_consensus.cc:2243] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:11:03.062188 24113 raft_consensus.cc:2272] T d4d43163cfdd4a4094e92fe68bc9259a P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.103511 24113 tablet_replica.cc:333] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.106340 24113 raft_consensus.cc:2243] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:11:03.107692 24113 raft_consensus.cc:2272] T b2cfa29915b34839a60321576f9a67d6 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.127015 25136 catalog_manager.cc:5697] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 14 to 19. New cstate: current_term: 19 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:11:03.129580 25136 catalog_manager.cc:5697] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 16 to 22, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 22 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:11:03.132089 25136 catalog_manager.cc:5697] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 15 to 17, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 17 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:03.145274 24113 tablet_replica.cc:333] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.146510 24113 raft_consensus.cc:2243] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.147178 24113 raft_consensus.cc:2272] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.150692 25310 tablet_service.cc:1558] Processing DeleteTablet for tablet 37bfef8566b143cf94971f26f417678c with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 8 (current committed config index is 14)) from {username='slave'} at 127.0.0.1:41870
I20260709 13:11:03.143150 25136 catalog_manager.cc:5697] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 15 to 21. New cstate: current_term: 21 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:03.158577 25135 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39582 (request call id 312) took 8149 ms. Trace:
I20260709 13:11:03.160398 25135 rpcz_store.cc:276] 0709 13:10:55.008939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:10:55.009213 (+   274us) service_pool.cc:224] Handling call
0709 13:11:03.158488 (+8149275us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:11:02.990888 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:11:02.991523 (+   635us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:11:02.991593 (+    70us) write_op.cc:435] Acquired schema lock
0709 13:11:02.991611 (+    18us) tablet.cc:662] Decoding operations
0709 13:11:02.992026 (+   415us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:11:02.992075 (+    49us) write_op.cc:641] Partition lock acquired for write op
0709 13:11:02.992110 (+    35us) tablet.cc:685] Acquiring locks for 2 operations
0709 13:11:02.992400 (+   290us) tablet.cc:701] Row locks acquired
0709 13:11:02.992416 (+    16us) write_op.cc:260] PREPARE: finished
0709 13:11:02.992546 (+   130us) write_op.cc:270] Start()
0709 13:11:02.992654 (+   108us) write_op.cc:276] Timestamp: P: 1783602662992501 usec, L: 0
0709 13:11:02.992674 (+    20us) op_driver.cc:348] REPLICATION: starting
0709 13:11:02.993331 (+   657us) log.cc:844] Serialized 929 byte log entry
0709 13:11:02.995113 (+  1782us) op_driver.cc:464] REPLICATION: finished
0709 13:11:02.995522 (+   409us) write_op.cc:301] APPLY: starting
0709 13:11:02.995601 (+    79us) tablet.cc:1367] starting BulkCheckPresence
0709 13:11:02.996877 (+  1276us) tablet.cc:1370] finished BulkCheckPresence
0709 13:11:02.996975 (+    98us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:11:02.998480 (+  1505us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:11:02.998630 (+   150us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=4,mrs_lookups=0
0709 13:11:02.998669 (+    39us) write_op.cc:312] APPLY: finished
0709 13:11:02.998865 (+   196us) log.cc:844] Serialized 33 byte log entry
0709 13:11:02.999027 (+   162us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:11:02.999112 (+    85us) write_op.cc:454] Released schema lock
0709 13:11:02.999151 (+    39us) write_op.cc:341] FINISH: Updating metrics
0709 13:11:02.999480 (+   329us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:11:02.999508 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":49,"spinlock_wait_cycles":28032,"child_traces":[["op",{"apply.queue_time_us":220,"apply.run_cpu_time_us":3589,"apply.run_wall_time_us":3894,"cfile_cache_hit":8,"cfile_cache_hit_bytes":8352,"num_ops":2,"prepare.queue_time_us":152,"prepare.run_cpu_time_us":2727,"prepare.run_wall_time_us":2727,"replication_time_us":2308}]]}
I20260709 13:11:03.165633 24113 tablet_replica.cc:333] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.184224 25128 catalog_manager.cc:5028] TS 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): tablet 37bfef8566b143cf94971f26f417678c (table test-workload [id=aff9eddd1d8649479ed0f20a8545e3e3]) successfully deleted
I20260709 13:11:03.187414 24113 raft_consensus.cc:2243] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.188170 24113 raft_consensus.cc:2272] T 7f37bfaeb12446c1a84af172a6f3dde9 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.181746 25136 catalog_manager.cc:5697] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 20 to 21, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 21 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:03.190917 25136 catalog_manager.cc:5697] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 21 to 22, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 22 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:11:03.205161 24113 tablet_replica.cc:333] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.207134 24113 raft_consensus.cc:2243] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.208154 24113 raft_consensus.cc:2272] T 1884b7159da148dbb1c19c424d667792 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.211853 25136 catalog_manager.cc:5697] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:03.213971 25136 catalog_manager.cc:5697] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 16 to 17, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 17 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:03.216521 25136 catalog_manager.cc:5697] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 19 to 23, leader changed from a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 23 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:11:03.220961 24113 tablet_replica.cc:333] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.228396 24113 raft_consensus.cc:2243] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:11:03.230531 25136 catalog_manager.cc:5697] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 21 to 22, leader changed from 2329e428ebbf484fa3ea9452133df567 (127.23.140.66) to 46de6919f85341c6a9b574dff112e35a (127.23.140.65). New cstate: current_term: 22 leader_uuid: "46de6919f85341c6a9b574dff112e35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } }
I20260709 13:11:03.232167 24113 raft_consensus.cc:2272] T 72001570ee2c41cb88f0e9eac74992c7 P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.239337 24113 tablet_replica.cc:333] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.244876 24113 raft_consensus.cc:2243] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.296927 24113 raft_consensus.cc:2272] T 5ead2e92f0614db79c9249687c4246f9 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.302829 24113 tablet_replica.cc:333] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.316171 24113 raft_consensus.cc:2243] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.326015 24113 raft_consensus.cc:2272] T 37bfef8566b143cf94971f26f417678c P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.339331 24113 tablet_replica.cc:333] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.355415 24113 raft_consensus.cc:2243] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.361881 24113 raft_consensus.cc:2272] T b6d19448306e4052a13c08a582f4ccfa P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.375591 24113 tablet_replica.cc:333] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.379048 24113 raft_consensus.cc:2243] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.389582 24113 raft_consensus.cc:2272] T 0e2fe177353b4d6c9de4ccf872ca06ca P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.401202 25136 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39596 (request call id 290) took 2879 ms. Trace:
I20260709 13:11:03.411358 24113 tablet_replica.cc:333] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.412559 24113 raft_consensus.cc:2243] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.415938 24113 raft_consensus.cc:2272] T c5481170c5c74798bdfd146068f8b539 P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.425251 24113 tablet_replica.cc:333] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.404917 25136 rpcz_store.cc:276] 0709 13:11:00.521594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:11:00.521730 (+   136us) service_pool.cc:224] Handling call
0709 13:11:03.401103 (+2879373us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:11:03.245455 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:11:03.245990 (+   535us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:11:03.246025 (+    35us) write_op.cc:435] Acquired schema lock
0709 13:11:03.246041 (+    16us) tablet.cc:662] Decoding operations
0709 13:11:03.247008 (+   967us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:11:03.247075 (+    67us) write_op.cc:641] Partition lock acquired for write op
0709 13:11:03.247106 (+    31us) tablet.cc:685] Acquiring locks for 10 operations
0709 13:11:03.247941 (+   835us) tablet.cc:701] Row locks acquired
0709 13:11:03.247962 (+    21us) write_op.cc:260] PREPARE: finished
0709 13:11:03.248109 (+   147us) write_op.cc:270] Start()
0709 13:11:03.248219 (+   110us) write_op.cc:276] Timestamp: P: 1783602663248061 usec, L: 0
0709 13:11:03.248244 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 13:11:03.248855 (+   611us) log.cc:844] Serialized 4034 byte log entry
0709 13:11:03.253445 (+  4590us) op_driver.cc:464] REPLICATION: finished
0709 13:11:03.256026 (+  2581us) write_op.cc:301] APPLY: starting
0709 13:11:03.256114 (+    88us) tablet.cc:1367] starting BulkCheckPresence
0709 13:11:03.264433 (+  8319us) tablet.cc:1370] finished BulkCheckPresence
0709 13:11:03.264524 (+    91us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:11:03.299508 (+ 34984us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:11:03.299715 (+   207us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=20,key_file_lookups=20,delta_file_lookups=20,mrs_lookups=0
0709 13:11:03.299762 (+    47us) write_op.cc:312] APPLY: finished
0709 13:11:03.300043 (+   281us) log.cc:844] Serialized 97 byte log entry
0709 13:11:03.306997 (+  6954us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:11:03.307128 (+   131us) write_op.cc:454] Released schema lock
0709 13:11:03.307194 (+    66us) write_op.cc:341] FINISH: Updating metrics
0709 13:11:03.314812 (+  7618us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:11:03.314840 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":78792,"reported_tablets":44,"child_traces":[["op",{"apply.queue_time_us":2371,"apply.run_cpu_time_us":12605,"apply.run_wall_time_us":58712,"cfile_cache_hit":40,"cfile_cache_hit_bytes":41760,"num_ops":10,"prepare.queue_time_us":9069,"prepare.run_cpu_time_us":3729,"prepare.run_wall_time_us":26913,"replication_time_us":5061}]]}
I20260709 13:11:03.447285 24113 raft_consensus.cc:2243] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:11:03.462076 24113 raft_consensus.cc:2272] T 17ef4786898b4ad696baed95b4209788 P 46de6919f85341c6a9b574dff112e35a [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.475873 24113 tablet_replica.cc:333] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.477149 24113 raft_consensus.cc:2243] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.484086 24113 raft_consensus.cc:2272] T 3cfa8937a5964f5790737950fc3b294e P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:03.488816 25359 consensus_peers.cc:597] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:03.492022 24113 tablet_replica.cc:333] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.506333 24113 raft_consensus.cc:2243] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.513996 24113 raft_consensus.cc:2272] T 8fa7e9050d44475381c97025c2b11711 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.532804 24113 tablet_replica.cc:333] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.533861 24113 raft_consensus.cc:2243] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.537596 24113 raft_consensus.cc:2272] T 8f209a6539f54de28906492f142efb6b P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.552095 24113 tablet_replica.cc:333] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.552879 24113 raft_consensus.cc:2243] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.569754 24113 raft_consensus.cc:2272] T 993e3b05a73a4403b04c3b55a5b48abe P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.578100 24113 tablet_replica.cc:333] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.579000 24113 raft_consensus.cc:2243] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.579766 24113 raft_consensus.cc:2272] T cb389d32141944d4b8f26a17471d50fc P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.582901 24113 tablet_replica.cc:333] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.583740 24113 raft_consensus.cc:2243] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.584801 24113 raft_consensus.cc:2272] T b62f4877af9742298b1e149789ca3090 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.618633 24113 tablet_replica.cc:333] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.619802 24113 raft_consensus.cc:2243] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.624678 24113 raft_consensus.cc:2272] T d1bed174e340476591fddc8217b8f644 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.628504 24113 tablet_replica.cc:333] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.629356 24113 raft_consensus.cc:2243] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:11:03.630463 24113 raft_consensus.cc:2272] T bacfad6698774ab8b26566bcecbcb615 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.647495 24113 tablet_replica.cc:333] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.648352 24113 raft_consensus.cc:2243] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:11:03.649502 24113 raft_consensus.cc:2272] T b7464c8ac67847019342ea6c88895f77 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.652501 24113 tablet_replica.cc:333] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.653398 24113 raft_consensus.cc:2243] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.676977 24113 raft_consensus.cc:2272] T a53b1fb6477d486e98bff9549cc487df P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.680410 24113 tablet_replica.cc:333] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.681293 24113 raft_consensus.cc:2243] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.681983 24113 raft_consensus.cc:2272] T bbeee5054bca4ae28d33636ae50ca11f P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.695043 24113 tablet_replica.cc:333] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.703233 24113 raft_consensus.cc:2243] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.704025 24113 raft_consensus.cc:2272] T 69fee8cad2c84e1492403c23786622a1 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.711338 24113 tablet_replica.cc:333] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.712378 24113 raft_consensus.cc:2243] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.712980 24113 raft_consensus.cc:2272] T 991bcf9b91a14242837dfb4e18891e91 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.721665 24113 tablet_replica.cc:333] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.724618 24113 raft_consensus.cc:2243] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.725354 24113 raft_consensus.cc:2272] T 56914a99b2d8417592e1f08979cfadf6 P 46de6919f85341c6a9b574dff112e35a [term 15 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.735352 24113 tablet_replica.cc:333] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.736272 24113 raft_consensus.cc:2243] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.738139 24113 raft_consensus.cc:2272] T 2dc23779efbf4518a6ada8bde8243a11 P 46de6919f85341c6a9b574dff112e35a [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.743432 24113 tablet_replica.cc:333] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.745853 24113 raft_consensus.cc:2243] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.746742 24113 raft_consensus.cc:2272] T e3025c38ad5a4bdfa61cfca91ba3c409 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.766973 24113 tablet_replica.cc:333] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.768143 24113 raft_consensus.cc:2243] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.768975 24113 raft_consensus.cc:2272] T f75b60146e7d4b9b9954b4fb2c414679 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.778275 24113 tablet_replica.cc:333] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.782760 24113 raft_consensus.cc:2243] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.783604 24113 raft_consensus.cc:2272] T 6f616cb6823f424a847dceaa5900dfc0 P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.794960 24113 tablet_replica.cc:333] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.795912 24113 raft_consensus.cc:2243] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.799717 24113 raft_consensus.cc:2272] T 6afb077d630246d6a315a828b4dcc108 P 46de6919f85341c6a9b574dff112e35a [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.805248 24113 tablet_replica.cc:333] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
W20260709 13:11:03.806993 25359 consensus_peers.cc:597] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:03.806283 24113 raft_consensus.cc:2243] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.828749 24113 raft_consensus.cc:2272] T eda233ace2a24b379fef320a5f610c7a P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:03.847297 25285 consensus_peers.cc:597] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:03.888492 25359 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 381 similar messages]
I20260709 13:11:03.895515 24113 tablet_replica.cc:333] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.899451 24113 raft_consensus.cc:2243] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.900285 24113 raft_consensus.cc:2272] T cf0501919f944794bbab0ffae6a334e1 P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.932129 24113 tablet_replica.cc:333] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
W20260709 13:11:03.932639 25285 consensus_peers.cc:597] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:03.939282 24113 raft_consensus.cc:2243] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 23 LEADER]: Raft consensus shutting down.
W20260709 13:11:03.954589 25285 consensus_peers.cc:597] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:03.957566 24113 raft_consensus.cc:2272] T 2f9c4bd911cc450aa35a16a5d10bf16c P 46de6919f85341c6a9b574dff112e35a [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:03.970847 24113 tablet_replica.cc:333] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:03.972108 24113 raft_consensus.cc:2243] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:03.972865 24113 raft_consensus.cc:2272] T 6ff9a7b00f104dd294125870b06a7df4 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:03.988018 25359 consensus_peers.cc:597] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:03.994036 25909 consensus_peers.cc:597] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:03.997587 24113 tablet_replica.cc:333] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:04.013655 24113 raft_consensus.cc:2243] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:04.015738 24113 raft_consensus.cc:2272] T adf2f61a8d304d4eb9c30a77d7de17e4 P 46de6919f85341c6a9b574dff112e35a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:04.021981 24113 tablet_replica.cc:333] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:04.028570 24113 raft_consensus.cc:2243] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:04.029687 24113 raft_consensus.cc:2272] T 292e92613a634963a5de1f016eb951ea P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:04.036600 24113 tablet_replica.cc:333] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:04.037590 24113 raft_consensus.cc:2243] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:11:04.038812 24113 raft_consensus.cc:2272] T 3bd9cfccf308499991b366150c2d46ea P 46de6919f85341c6a9b574dff112e35a [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:04.052513 24113 tablet_replica.cc:333] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:04.054540 24113 raft_consensus.cc:2243] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:04.061803 24113 raft_consensus.cc:2272] T 9dac81b348b746fd8ecd3937028554ea P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:04.074142 25909 consensus_peers.cc:597] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:04.078037 24113 tablet_replica.cc:333] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:04.079058 24113 raft_consensus.cc:2243] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:04.080156 24113 raft_consensus.cc:2272] T e5cbaabf54304de089abde51df6dc2be P 46de6919f85341c6a9b574dff112e35a [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:04.084425 24113 tablet_replica.cc:333] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:04.093888 24113 raft_consensus.cc:2243] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:04.094806 24113 raft_consensus.cc:2272] T acd2d31837dd470c8bd91f719ffeb461 P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:04.118414 25285 consensus_peers.cc:597] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:04.136264 25285 consensus_peers.cc:597] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:04.141319 24113 tablet_replica.cc:333] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:04.144814 24113 raft_consensus.cc:2243] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Raft consensus shutting down.
W20260709 13:11:04.145246 25909 consensus_peers.cc:597] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:04.145949 24113 raft_consensus.cc:2272] T 1b8652aecc6d4869bb4cad9e0749f3ba P 46de6919f85341c6a9b574dff112e35a [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:04.150350 24113 tablet_replica.cc:333] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:04.160380 24113 raft_consensus.cc:2243] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:04.161116 24113 raft_consensus.cc:2272] T 58ec1834a96f447f8fe74448e8480342 P 46de6919f85341c6a9b574dff112e35a [term 16 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:04.167622 25359 consensus_peers.cc:597] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:04.176744 24113 tablet_replica.cc:333] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
I20260709 13:11:04.183842   706 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:04.184516   706 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:04.186753   706 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:04.192919 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
W20260709 13:11:04.197570 25909 leader_election.cc:336] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:11:04.204984 25359 consensus_peers.cc:597] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:04.206849 25908 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:04.208228   706 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:11:04.218845 25285 consensus_peers.cc:597] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:04.227695 24113 raft_consensus.cc:2243] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:04.230540 24113 raft_consensus.cc:2272] T 17584a19690d4010851d65378cfcbafa P 46de6919f85341c6a9b574dff112e35a [term 19 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:04.263500 25285 consensus_peers.cc:597] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:04.300724 25909 consensus_peers.cc:597] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:04.308306   669 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:04.309140   669 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:04.337962   669 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:11:04.321877 25285 consensus_peers.cc:597] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:04.333611 25909 consensus_peers.cc:597] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:04.343607 25285 leader_election.cc:336] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:11:04.365963 24113 tablet_replica.cc:333] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a: stopping tablet replica
W20260709 13:11:04.391747 25285 consensus_peers.cc:597] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:04.392802 25285 consensus_peers.cc:597] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:04.395350 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:04.411033 24113 raft_consensus.cc:2243] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:11:04.443102 24113 raft_consensus.cc:2272] T e81523f4f4344a04a1e0b03c8de99290 P 46de6919f85341c6a9b574dff112e35a [term 24 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:04.466833 25285 consensus_peers.cc:597] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:04.482693 25359 consensus_peers.cc:597] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:04.520119   706 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:04.520764   706 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:04.525568 25386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
W20260709 13:11:04.527073 25285 consensus_peers.cc:597] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:04.530573 25909 leader_election.cc:336] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:11:04.531087 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:04.532028   711 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:11:04.533406   706 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:04.544183   712 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:04.544896   712 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:04.545928   706 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:04.546635   706 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:04.548465   712 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:04.549194   706 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:04.550863 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:04.551506 25940 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 18.
I20260709 13:11:04.553254 25284 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
W20260709 13:11:04.556777 25909 leader_election.cc:336] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:11:04.562487   712 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:11:04.563369   712 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:04.563905   712 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:11:04.581957 25287 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:04.583026   669 raft_consensus.cc:2749] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:11:04.606971   660 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:04.607734   660 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:04.610669   660 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:04.620546 25285 leader_election.cc:336] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:11:04.650285   720 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:04.650988   720 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:04.653615   720 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:04.667196 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 85453.4us
W20260709 13:11:04.682050 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 46693us
I20260709 13:11:04.686025   722 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:04.686759   722 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:04.689446   722 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:11:04.703857 25359 consensus_peers.cc:597] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:04.753227 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 49438.5us
W20260709 13:11:04.754045 25359 consensus_peers.cc:597] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:04.754904 25359 consensus_peers.cc:597] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:04.756011 25359 consensus_peers.cc:597] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:04.707222 25285 consensus_peers.cc:597] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:04.731122   720 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:04.759330   720 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:04.762553   720 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:04.763232 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 56078.1us
W20260709 13:11:04.771404 25285 leader_election.cc:336] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:11:04.740577 25313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:04.744355 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:04.806083 25318 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 23.
I20260709 13:11:04.745303 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:04.751658   723 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:04.808100   723 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:04.809298 25313 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 20.
I20260709 13:11:04.814085 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:04.814810 25940 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 20.
I20260709 13:11:04.816478 25284 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:11:04.738166   712 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:04.820300 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d43163cfdd4a4094e92fe68bc9259a" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:11:04.820906 25940 raft_consensus.cc:3060] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:11:04.829783   722 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:11:04.830363   722 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:04.830814   722 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:11:04.845248   712 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:04.846086   723 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:11:04.846783 25285 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 75409.4us
I20260709 13:11:04.869367 25321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:04.870368 25321 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 22.
I20260709 13:11:04.883239 25940 raft_consensus.cc:2468] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 19.
I20260709 13:11:04.884977 25284 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:11:04.885967   712 raft_consensus.cc:2804] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:11:04.886610   712 raft_consensus.cc:697] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:11:04.887668   712 consensus_queue.cc:237] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:11:04.946633 25285 consensus_peers.cc:597] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:04.966698   722 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:04.967260 25358 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:11:04.968304   660 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:11:04.968791   660 raft_consensus.cc:493] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:04.969866 25908 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:04.972332   706 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:11:04.972767   706 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:04.973268   706 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:11:04.981976 25285 leader_election.cc:336] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:11:04.982916 25285 leader_election.cc:336] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:11:04.986826   660 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:11:04.991577   712 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:04.992180   712 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:04.994244   712 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:11:04.995895 25285 leader_election.cc:336] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:11:04.996783 25135 catalog_manager.cc:5697] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 18 to 19, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 19 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:05.009559 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:11:05.010572   722 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:05.017218 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:11:05.018306   715 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:11:05.018819   715 raft_consensus.cc:493] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:05.019308   715 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:11:05.028491 25285 leader_election.cc:336] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:11:05.033010   706 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:05.074085 25940 raft_consensus.cc:2393] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 21: Already voted for candidate f646dfd532974dceadfe00005d2bbf59 in this term.
I20260709 13:11:05.078758   706 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:05.081393   726 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:05.081997   726 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:05.084528   726 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:05.087338   660 raft_consensus.cc:515] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:05.095075 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:05.095607 25938 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 24.
I20260709 13:11:05.094607 25284 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
W20260709 13:11:05.094095 25909 leader_election.cc:336] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:11:05.101363   715 raft_consensus.cc:515] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:05.104693   669 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:11:05.098414 25358 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:11:05.106521   726 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:11:05.107050   726 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:05.107514   726 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:11:05.113531   660 leader_election.cc:290] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 24 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:05.123250   715 leader_election.cc:290] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:05.122777   726 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:11:05.126295 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 49980.9us
I20260709 13:11:05.129542 25312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:11:05.130199 25312 raft_consensus.cc:2393] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f646dfd532974dceadfe00005d2bbf59 in current term 21: Already voted for candidate 2329e428ebbf484fa3ea9452133df567 in this term.
I20260709 13:11:05.137372 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:11:05.139165 25318 raft_consensus.cc:3060] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:11:05.139137 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72001570ee2c41cb88f0e9eac74992c7" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:11:05.142484 25315 raft_consensus.cc:3060] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:11:05.172508 25908 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:05.172681 25318 raft_consensus.cc:2468] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 23.
I20260709 13:11:05.174921 25315 raft_consensus.cc:2468] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 24.
I20260709 13:11:05.186276   660 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:05.186882   728 raft_consensus.cc:2749] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:11:05.186934   660 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:05.189229   660 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:05.190975 25359 consensus_peers.cc:597] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:05.191916 25359 consensus_peers.cc:597] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:05.193274 25359 consensus_peers.cc:597] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:05.193842 25359 leader_election.cc:336] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:11:05.198959 25389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bacfad6698774ab8b26566bcecbcb615" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:05.199753 25389 raft_consensus.cc:2393] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2329e428ebbf484fa3ea9452133df567 in current term 23: Already voted for candidate a2a77ecf2c0941618b5bc79d4c04db61 in this term.
I20260709 13:11:05.207583   726 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 25 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:05.208326 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:05.209012 25387 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 22.
I20260709 13:11:05.217430 25287 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:05.223126   729 raft_consensus.cc:2749] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:11:05.227334 25287 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:05.228169   722 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:11:05.228857   722 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:05.229296   722 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:11:05.229954 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17ef4786898b4ad696baed95b4209788" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59"
I20260709 13:11:05.230552 25938 raft_consensus.cc:3060] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:11:05.229859 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:05.237839 25358 leader_election.cc:304] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
I20260709 13:11:05.242197 25938 raft_consensus.cc:2468] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 25.
I20260709 13:11:05.245837   733 raft_consensus.cc:2804] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:11:05.246289   669 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:05.247094   733 raft_consensus.cc:697] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:11:05.247102   669 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:05.248302   733 consensus_queue.cc:237] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:05.249382   669 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:05.246490 25358 leader_election.cc:304] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:11:05.252568 25285 leader_election.cc:336] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:11:05.256567   726 raft_consensus.cc:2804] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:11:05.276360   726 raft_consensus.cc:697] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:11:05.277689   726 consensus_queue.cc:237] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:05.269865   722 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:05.285785   722 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:11:05.292327 25285 leader_election.cc:336] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:11:05.311867 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:05.313024 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7464c8ac67847019342ea6c88895f77" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:11:05.313668 25395 raft_consensus.cc:3060] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:11:05.324443   722 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:05.325047   722 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:05.329685 25395 raft_consensus.cc:2468] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 23.
W20260709 13:11:05.330634 25285 leader_election.cc:336] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:11:05.331483 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 74174.7us
I20260709 13:11:05.331984   722 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:11:05.322273 25359 leader_election.cc:336] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:11:05.339246 25359 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:11:05.344779 25285 consensus_peers.cc:597] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:05.350188 25135 catalog_manager.cc:5697] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 22 to 23, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 23 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
W20260709 13:11:05.361747 25359 consensus_peers.cc:597] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:05.362743 25359 leader_election.cc:336] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:11:05.363782 25359 consensus_peers.cc:597] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:05.364352 25359 consensus_peers.cc:597] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:05.366169   739 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:11:05.363305 25358 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:11:05.370555   734 raft_consensus.cc:2804] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:11:05.371109   734 raft_consensus.cc:697] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 25 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:11:05.372035   734 consensus_queue.cc:237] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:05.399734 25287 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:05.401744   722 raft_consensus.cc:2749] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:11:05.400957 25287 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:05.404404   722 raft_consensus.cc:2804] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:11:05.405012   722 raft_consensus.cc:697] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 23 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:11:05.406004   722 consensus_queue.cc:237] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:11:05.413241 25359 consensus_peers.cc:597] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:05.419481 25395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:05.420143 25395 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 23.
W20260709 13:11:05.420006 25359 leader_election.cc:336] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:11:05.421602 25359 leader_election.cc:336] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:11:05.422331 25359 leader_election.cc:336] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:11:05.422863 25359 leader_election.cc:336] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:11:05.423079 25359 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:11:05.436622 25359 leader_election.cc:336] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:11:05.438983 25359 consensus_peers.cc:597] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:05.441229 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:05.442143   726 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:11:05.440847   739 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:05.445397   669 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:11:05.456401   669 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:05.456255   739 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:05.456853   669 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:11:05.441854 25909 connection.cc:573] client connection to 127.23.140.65:45313 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260709 13:11:05.458668   739 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:05.513084 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:05.513880 25315 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 22.
I20260709 13:11:05.522769 25135 catalog_manager.cc:5697] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 22 to 23, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 23 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:05.531172 25938 raft_consensus.cc:1275] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:11:05.532805   729 consensus_queue.cc:1048] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:11:05.531209 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 
W20260709 13:11:05.529265 25359 connection.cc:769] client connection to 127.23.140.65:45313 send error: Network error: sendmsg error: Broken pipe (error 32)
I20260709 13:11:05.557034   734 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:11:05.557682   734 raft_consensus.cc:493] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:05.558179   734 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:11:05.622701 25359 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:45313, user_credentials={real_user=slave}} blocked reactor thread for 76437.2us
I20260709 13:11:05.637507   734 raft_consensus.cc:515] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:05.636967 25141 catalog_manager.cc:5697] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 23 to 24, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 24 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:05.639662   734 leader_election.cc:290] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:05.639593   669 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:05.639901 25141 catalog_manager.cc:5697] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 24 to 25, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 25 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:05.641829   669 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:05.654338   722 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:05.654935   722 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:05.657023   722 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:11:05.685631 25285 leader_election.cc:336] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:05.686290 25285 leader_election.cc:336] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:05.737530 25359 leader_election.cc:336] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:05.738545 25359 leader_election.cc:336] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:05.739390 25359 consensus_peers.cc:597] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:05.740319 25943 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:05.765944 25284 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567; no voters: 46de6919f85341c6a9b574dff112e35a, f646dfd532974dceadfe00005d2bbf59
I20260709 13:11:05.766592 25315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd9cfccf308499991b366150c2d46ea" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:11:05.767398 25315 raft_consensus.cc:3060] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:11:05.770207 25321 raft_consensus.cc:1275] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 13:11:05.774808 25315 raft_consensus.cc:2468] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 23.
I20260709 13:11:05.775856 25321 raft_consensus.cc:1275] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 13:11:05.776077   669 raft_consensus.cc:2749] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:11:05.776491 25359 consensus_peers.cc:597] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:05.783375 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f9c4bd911cc450aa35a16a5d10bf16c" candidate_uuid: "2329e428ebbf484fa3ea9452133df567" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61"
I20260709 13:11:05.783990 25387 raft_consensus.cc:3060] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:11:05.791004 25387 raft_consensus.cc:2468] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329e428ebbf484fa3ea9452133df567 in term 24.
I20260709 13:11:05.800136 25358 leader_election.cc:304] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:05.801932   743 consensus_queue.cc:1048] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:11:05.808651   660 raft_consensus.cc:2804] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:11:05.809320   660 raft_consensus.cc:697] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 LEADER]: Becoming Leader. State: Replica: a2a77ecf2c0941618b5bc79d4c04db61, State: Running, Role: LEADER
I20260709 13:11:05.810353   660 consensus_queue.cc:237] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:11:05.817688 25359 consensus_peers.cc:597] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:05.818303   720 consensus_queue.cc:1048] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20260709 13:11:05.825593 25359 consensus_peers.cc:597] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:05.842626 25135 catalog_manager.cc:5697] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 reported cstate change: term changed from 22 to 23, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67). New cstate: current_term: 23 leader_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:05.848464 25287 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:05.857658   669 raft_consensus.cc:2804] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:11:05.858156   669 raft_consensus.cc:697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 24 LEADER]: Becoming Leader. State: Replica: 2329e428ebbf484fa3ea9452133df567, State: Running, Role: LEADER
I20260709 13:11:05.858973   669 consensus_queue.cc:237] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:05.865505   706 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:05.866170   706 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:05.868438   706 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:05.868628 25938 raft_consensus.cc:1275] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 25 index: 20. (index mismatch)
I20260709 13:11:05.873236 25314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:05.873255   720 consensus_queue.cc:1048] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 13:11:05.874112 25314 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 24.
W20260709 13:11:05.875509 25909 leader_election.cc:336] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:05.881398 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:05.882287   706 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:11:05.882690   706 raft_consensus.cc:493] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 46de6919f85341c6a9b574dff112e35a)
I20260709 13:11:05.883090   706 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:11:05.891506   706 raft_consensus.cc:515] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:11:05.897843 25909 leader_election.cc:336] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:05.920442   706 leader_election.cc:290] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 25 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:05.927557 25285 consensus_peers.cc:597] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:05.928215 25389 raft_consensus.cc:1275] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 13:11:05.922060 25135 catalog_manager.cc:5697] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 reported cstate change: term changed from 23 to 24, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to 2329e428ebbf484fa3ea9452133df567 (127.23.140.66). New cstate: current_term: 24 leader_uuid: "2329e428ebbf484fa3ea9452133df567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:05.941493 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e81523f4f4344a04a1e0b03c8de99290" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:11:05.942062 25318 raft_consensus.cc:3060] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:11:05.948750 25318 raft_consensus.cc:2468] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 25.
I20260709 13:11:05.973322   669 consensus_queue.cc:1048] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:11:05.983260 25908 leader_election.cc:304] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:05.984606   706 raft_consensus.cc:2804] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:11:05.985388   706 raft_consensus.cc:697] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 25 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:11:05.986340   706 consensus_queue.cc:237] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:11:06.062793 25359 consensus_peers.cc:597] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:06.075867 25141 catalog_manager.cc:5697] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 24 to 25, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 25 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:11:06.196630 25315 raft_consensus.cc:1275] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Refusing update from remote peer a2a77ecf2c0941618b5bc79d4c04db61: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 13:11:06.198135   743 consensus_queue.cc:1048] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260709 13:11:06.238487 25359 consensus_peers.cc:597] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:06.243252 25285 consensus_peers.cc:597] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:06.294807 25285 consensus_peers.cc:597] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:06.355854 25389 raft_consensus.cc:1275] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Refusing update from remote peer 2329e428ebbf484fa3ea9452133df567: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 13:11:06.357821   750 consensus_queue.cc:1048] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 13:11:06.448804 25909 consensus_peers.cc:597] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:06.452251 25909 consensus_peers.cc:597] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:06.479215 25285 consensus_peers.cc:597] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:06.499046 25909 consensus_peers.cc:597] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:06.500593 24113 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 25267)
W20260709 13:11:06.532021 25285 consensus_peers.cc:597] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:06.539197 25285 consensus_peers.cc:597] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:06.554345 25359 consensus_peers.cc:597] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:06.572667 25909 consensus_peers.cc:597] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:11:06.583974 25318 raft_consensus.cc:1275] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 25 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 13:11:06.585774   706 consensus_queue.cc:1048] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 13:11:06.605470 25285 consensus_peers.cc:597] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:06.619753 25359 consensus_peers.cc:597] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:06.766929 25285 consensus_peers.cc:597] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:11:06.883178   767 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:06.883720   767 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:06.900851   767 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:06.904188 25320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567" is_pre_election: true
I20260709 13:11:06.904939 25320 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 21.
W20260709 13:11:06.926658 25359 consensus_peers.cc:597] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:06.944562 25285 consensus_peers.cc:597] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:06.945283 25285 consensus_peers.cc:597] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:11:06.946485 25908 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 
I20260709 13:11:06.951485   706 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:11:06.951890   706 raft_consensus.cc:493] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:06.952234   706 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:11:06.965266 25909 consensus_peers.cc:597] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:06.965960 25909 consensus_peers.cc:597] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:06.966566 25909 leader_election.cc:336] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:06.977629 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 69299.4us
I20260709 13:11:07.004568   706 raft_consensus.cc:515] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:11:07.007014 25285 consensus_peers.cc:597] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.012676 25909 leader_election.cc:336] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:07.014403   706 leader_election.cc:290] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 election: Requested vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:07.041690 25318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2cfa29915b34839a60321576f9a67d6" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "2329e428ebbf484fa3ea9452133df567"
I20260709 13:11:07.042431 25318 raft_consensus.cc:3060] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:11:07.051473 25318 raft_consensus.cc:2468] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 22.
W20260709 13:11:07.070977 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 95587.2us
I20260709 13:11:07.072790 25908 leader_election.cc:304] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:07.076754   766 raft_consensus.cc:2804] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:11:07.077325   766 raft_consensus.cc:697] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 22 LEADER]: Becoming Leader. State: Replica: f646dfd532974dceadfe00005d2bbf59, State: Running, Role: LEADER
I20260709 13:11:07.078171   766 consensus_queue.cc:237] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:11:07.084753 25285 consensus_peers.cc:597] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.121869 25285 consensus_peers.cc:597] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.125702 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 147365us
W20260709 13:11:07.150357 25285 consensus_peers.cc:597] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:11:07.155814 25141 catalog_manager.cc:5697] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 reported cstate change: term changed from 20 to 22, leader changed from 46de6919f85341c6a9b574dff112e35a (127.23.140.65) to f646dfd532974dceadfe00005d2bbf59 (127.23.140.68). New cstate: current_term: 22 leader_uuid: "f646dfd532974dceadfe00005d2bbf59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:11:07.209745 25359 consensus_peers.cc:597] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.225878 25285 consensus_peers.cc:597] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.297175 25359 consensus_peers.cc:597] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.307140 25285 consensus_peers.cc:597] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.308023 25359 consensus_peers.cc:597] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.335656 25359 consensus_peers.cc:597] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.391588 25359 consensus_peers.cc:597] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.461253 25359 consensus_peers.cc:597] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.501103 24113 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 25267)
W20260709 13:11:07.504184 25359 consensus_peers.cc:597] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.504882 25359 consensus_peers.cc:597] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.512162 25359 consensus_peers.cc:597] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.592710 25909 consensus_peers.cc:597] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:07.668154 25318 raft_consensus.cc:1275] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Refusing update from remote peer f646dfd532974dceadfe00005d2bbf59: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:11:07.670260   778 consensus_queue.cc:1048] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260709 13:11:07.689621 25359 consensus_peers.cc:597] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.766276 25359 consensus_peers.cc:597] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.777472 25966 debug-util.cc:398] Leaking SignalData structure 0x7b080036c9c0 after lost signal to thread 24116
W20260709 13:11:07.778916 25966 debug-util.cc:398] Leaking SignalData structure 0x7b080036b860 after lost signal to thread 25178
W20260709 13:11:07.780609 25966 debug-util.cc:398] Leaking SignalData structure 0x7b08000ffa60 after lost signal to thread 25345
W20260709 13:11:07.781994 25966 debug-util.cc:398] Leaking SignalData structure 0x7b080034f9c0 after lost signal to thread 25419
W20260709 13:11:07.783147 25966 debug-util.cc:398] Leaking SignalData structure 0x7b0800322be0 after lost signal to thread 25969
W20260709 13:11:07.886085 25359 consensus_peers.cc:597] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.886729 25359 consensus_peers.cc:597] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.929071 25359 consensus_peers.cc:597] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:07.966590 25359 consensus_peers.cc:597] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:08.094944 25285 consensus_peers.cc:597] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:08.338330 25359 consensus_peers.cc:597] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:08.349654 25359 consensus_peers.cc:597] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:08.408335 25359 consensus_peers.cc:597] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:08.411556 25359 consensus_peers.cc:597] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:08.502089 24113 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 25267)
W20260709 13:11:08.505329 25359 consensus_peers.cc:597] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:08.603757 25285 consensus_peers.cc:597] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:08.782725 25285 consensus_peers.cc:597] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:08.818321 25285 consensus_peers.cc:597] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:08.825312 25285 consensus_peers.cc:597] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:08.889138 25909 consensus_peers.cc:597] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:08.911547 25909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111) [suppressed 446 similar messages]
W20260709 13:11:08.952526 25359 consensus_peers.cc:597] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:08.986712 25285 consensus_peers.cc:597] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.030228 25909 consensus_peers.cc:597] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.079555 25909 consensus_peers.cc:597] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.082726 25909 consensus_peers.cc:597] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.082726 25285 consensus_peers.cc:597] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.102406 25285 consensus_peers.cc:597] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.118278 25909 consensus_peers.cc:597] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:09.192780 25285 consensus_peers.cc:597] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.268146 25359 consensus_peers.cc:597] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.281032 25285 consensus_peers.cc:597] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.285789 25359 consensus_peers.cc:597] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.299856 25285 consensus_peers.cc:597] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.346151 25285 consensus_peers.cc:597] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.446502 25359 consensus_peers.cc:597] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.479683 25909 consensus_peers.cc:597] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.502543 24113 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 25267)
W20260709 13:11:09.511811 25285 consensus_peers.cc:597] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.513522 25359 consensus_peers.cc:597] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.537504 25285 consensus_peers.cc:597] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.666002 25285 consensus_peers.cc:597] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.670588 25285 consensus_peers.cc:597] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.676502 25285 consensus_peers.cc:597] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.770257 25359 consensus_peers.cc:597] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.845479 25359 consensus_peers.cc:597] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.933264 25359 consensus_peers.cc:597] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:09.980718 25359 consensus_peers.cc:597] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:10.083624 25909 consensus_peers.cc:597] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:10.090371 25359 consensus_peers.cc:597] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:10.145253 25359 consensus_peers.cc:597] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:10.180228 25359 consensus_peers.cc:597] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:10.230545 25359 consensus_peers.cc:597] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:10.240411 25359 consensus_peers.cc:597] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:10.243299 25359 consensus_peers.cc:597] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:10.293972 25359 consensus_peers.cc:597] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:10.336582 25359 consensus_peers.cc:597] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:10.502986 24113 thread.cc:527] Waited for 5000ms trying to join with diag-logger (tid 25267)
W20260709 13:11:10.510740 25285 consensus_peers.cc:597] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:10.661274 25359 consensus_peers.cc:597] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:10.679085 25359 consensus_peers.cc:597] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:10.696146 25359 consensus_peers.cc:597] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:10.708338 25359 consensus_peers.cc:597] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:10.910481 25359 consensus_peers.cc:597] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:10.968237 25359 consensus_peers.cc:597] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:11.133402 25285 consensus_peers.cc:597] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:11.183390 25359 consensus_peers.cc:597] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:11.185695 25359 consensus_peers.cc:597] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:11.242195 25285 consensus_peers.cc:597] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:11.294144 25909 consensus_peers.cc:597] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:11.306883 25285 consensus_peers.cc:597] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:11.446168 25285 consensus_peers.cc:597] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:11.485994 25285 consensus_peers.cc:597] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:11.503736 24113 thread.cc:527] Waited for 6000ms trying to join with diag-logger (tid 25267)
W20260709 13:11:11.529819 25909 consensus_peers.cc:597] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:11.545725 25909 consensus_peers.cc:597] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:11.552379 25285 consensus_peers.cc:597] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:11.588311 25285 consensus_peers.cc:597] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:11.640570 25285 consensus_peers.cc:597] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:11.641499 25909 consensus_peers.cc:597] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:11.651216 25909 consensus_peers.cc:597] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:11.654989 25285 consensus_peers.cc:597] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:11.678650 25909 consensus_peers.cc:597] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:11.821096 25285 consensus_peers.cc:597] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:11.825865 25359 consensus_peers.cc:597] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:11.864614 25359 consensus_peers.cc:597] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:11.953099 25285 consensus_peers.cc:597] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:11.955236 25285 consensus_peers.cc:597] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:11.965183 25285 consensus_peers.cc:597] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:12.047582 25359 consensus_peers.cc:597] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:12.086963 25359 consensus_peers.cc:597] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:12.087901 25285 consensus_peers.cc:597] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:12.108291 25285 consensus_peers.cc:597] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:12.129103 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 96942.3us
W20260709 13:11:12.152886 25359 consensus_peers.cc:597] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:12.335064 25285 consensus_peers.cc:597] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:12.462809 25359 consensus_peers.cc:597] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:12.473117 25359 consensus_peers.cc:597] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:12.477418 25359 consensus_peers.cc:597] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:12.504460 24113 thread.cc:527] Waited for 7000ms trying to join with diag-logger (tid 25267)
W20260709 13:11:12.578348 25359 consensus_peers.cc:597] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:12.610718 25359 consensus_peers.cc:597] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:12.644255 25359 consensus_peers.cc:597] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:12.658641 25285 consensus_peers.cc:597] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:12.681308 25909 consensus_peers.cc:597] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:12.724416 25359 consensus_peers.cc:597] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:12.792467 25359 consensus_peers.cc:597] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:12.797255 25359 consensus_peers.cc:597] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:12.919502 25359 consensus_peers.cc:597] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:12.991140 25359 consensus_peers.cc:597] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:13.006722 25359 consensus_peers.cc:597] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:13.100636 25359 consensus_peers.cc:597] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:13.391659 25285 consensus_peers.cc:597] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:13.438136 25359 consensus_peers.cc:597] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:13.457785 25359 consensus_peers.cc:597] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:13.459777 25359 consensus_peers.cc:597] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:13.504971 24113 thread.cc:527] Waited for 8000ms trying to join with diag-logger (tid 25267)
W20260709 13:11:13.555562 25359 consensus_peers.cc:597] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:13.614432 25359 consensus_peers.cc:597] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:13.627372 25285 consensus_peers.cc:597] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:13.630628 25359 consensus_peers.cc:597] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:13.815763 25909 consensus_peers.cc:597] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:13.872593 25284 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:39211, user_credentials={real_user=slave}} blocked reactor thread for 64028.2us
W20260709 13:11:13.902920 25909 consensus_peers.cc:597] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:13.909463 25285 consensus_peers.cc:597] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:13.937444 25285 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111) [suppressed 481 similar messages]
W20260709 13:11:13.941320 25285 consensus_peers.cc:597] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.002564 25909 consensus_peers.cc:597] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.009272 25909 consensus_peers.cc:597] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.023088 25285 consensus_peers.cc:597] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.028492 25909 consensus_peers.cc:597] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.111173 25285 consensus_peers.cc:597] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.117581 25285 consensus_peers.cc:597] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.150398 25285 consensus_peers.cc:597] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.167501 25285 consensus_peers.cc:597] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:14.220849 25359 consensus_peers.cc:597] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.254978 25285 consensus_peers.cc:597] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.346536 25285 consensus_peers.cc:597] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.431043 25285 consensus_peers.cc:597] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.441505 25359 consensus_peers.cc:597] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.470824 25285 consensus_peers.cc:597] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.473707 25285 consensus_peers.cc:597] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.505491 24113 thread.cc:527] Waited for 9000ms trying to join with diag-logger (tid 25267)
W20260709 13:11:14.538076 25285 consensus_peers.cc:597] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.538897 25909 consensus_peers.cc:597] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:14.550787 25359 consensus_peers.cc:597] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.615083 25359 consensus_peers.cc:597] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.832796 25285 consensus_peers.cc:597] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.835327 25359 consensus_peers.cc:597] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.889781 25359 consensus_peers.cc:597] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.894717 25359 consensus_peers.cc:597] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.960220 25359 consensus_peers.cc:597] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:14.963220 25359 consensus_peers.cc:597] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:15.051364 25359 consensus_peers.cc:597] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:15.122929 25285 consensus_peers.cc:597] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:15.153625 25359 consensus_peers.cc:597] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:15.181552 25359 consensus_peers.cc:597] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:15.209699 25359 consensus_peers.cc:597] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:15.239650 25359 consensus_peers.cc:597] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:15.321655 25359 consensus_peers.cc:597] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:15.373446 25909 consensus_peers.cc:597] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:11:15.505993 24113 thread.cc:527] Waited for 10000ms trying to join with diag-logger (tid 25267)
W20260709 13:11:15.511341 25359 consensus_peers.cc:597] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:15.579710 25359 consensus_peers.cc:597] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:15.645370 25359 consensus_peers.cc:597] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:15.823413 25359 consensus_peers.cc:597] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:15.827260 25359 consensus_peers.cc:597] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:15.830412 25285 consensus_peers.cc:597] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:15.846915 25359 consensus_peers.cc:597] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:16.042673 25359 consensus_peers.cc:597] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:16.094072 25359 consensus_peers.cc:597] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:16.177451 25285 consensus_peers.cc:597] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:16.197247 25909 consensus_peers.cc:597] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:16.254130 25909 consensus_peers.cc:597] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:16.260875 25285 consensus_peers.cc:597] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:16.331189 25909 consensus_peers.cc:597] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:16.331939 25285 consensus_peers.cc:597] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:16.382027 25359 consensus_peers.cc:597] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:16.437973 25285 consensus_peers.cc:597] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:16.506884 24113 thread.cc:527] Waited for 11000ms trying to join with diag-logger (tid 25267)
W20260709 13:11:16.518834 25909 consensus_peers.cc:597] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:16.573302 25285 consensus_peers.cc:597] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:16.616138 25285 consensus_peers.cc:597] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:16.618824 25285 consensus_peers.cc:597] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:16.628072 25909 consensus_peers.cc:597] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:16.667452 25359 consensus_peers.cc:597] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:16.770165 25285 consensus_peers.cc:597] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:16.780995 25285 consensus_peers.cc:597] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:16.865252 25359 consensus_peers.cc:597] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:16.866807 25285 consensus_peers.cc:597] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:16.884660 25285 consensus_peers.cc:597] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:16.894471 25909 consensus_peers.cc:597] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:16.942714 25285 consensus_peers.cc:597] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:17.049928 25359 consensus_peers.cc:597] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:17.224987 25285 consensus_peers.cc:597] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:17.226063 25285 consensus_peers.cc:597] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:17.273092 25359 consensus_peers.cc:597] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:17.276371 25359 consensus_peers.cc:597] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:17.310243 25359 consensus_peers.cc:597] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:17.372423 25359 consensus_peers.cc:597] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:17.507339 24113 thread.cc:527] Waited for 12000ms trying to join with diag-logger (tid 25267)
W20260709 13:11:17.510103 25359 consensus_peers.cc:597] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:17.535105 25285 consensus_peers.cc:597] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:17.571695 25285 consensus_peers.cc:597] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:17.647006 25359 consensus_peers.cc:597] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:17.737500 25359 consensus_peers.cc:597] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:17.753350 25909 consensus_peers.cc:597] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:11:17.765645 25359 consensus_peers.cc:597] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:17.798255 25359 consensus_peers.cc:597] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:17.860603 25359 consensus_peers.cc:597] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:17.861269 25359 consensus_peers.cc:597] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:17.889447 25359 consensus_peers.cc:597] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:17.895334 25359 consensus_peers.cc:597] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:17.904969 25359 consensus_peers.cc:597] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:18.077026 25416 debug-util.cc:398] Leaking SignalData structure 0x7b080020ea60 after lost signal to thread 24116
W20260709 13:11:18.078251 25416 debug-util.cc:398] Leaking SignalData structure 0x7b0800328260 after lost signal to thread 25178
W20260709 13:11:18.079567 25416 debug-util.cc:398] Leaking SignalData structure 0x7b080019e6c0 after lost signal to thread 25345
W20260709 13:11:18.080663 25416 debug-util.cc:398] Leaking SignalData structure 0x7b080014f940 after lost signal to thread 25419
W20260709 13:11:18.081838 25416 debug-util.cc:398] Leaking SignalData structure 0x7b08002596e0 after lost signal to thread 25969
W20260709 13:11:18.190982 25359 consensus_peers.cc:597] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:18.234516 25359 consensus_peers.cc:597] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:18.342098 25359 consensus_peers.cc:597] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:18.453384 25359 consensus_peers.cc:597] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:18.468080 25359 consensus_peers.cc:597] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:18.577206 25359 consensus_peers.cc:597] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:18.671171 25909 consensus_peers.cc:597] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:18.737993 25909 consensus_peers.cc:597] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:18.768476 25909 consensus_peers.cc:597] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:18.942576 25359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111) [suppressed 462 similar messages]
W20260709 13:11:18.971674 25909 consensus_peers.cc:597] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:19.054029 25359 consensus_peers.cc:597] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:19.169749 25285 consensus_peers.cc:597] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:19.181044 25359 consensus_peers.cc:597] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:19.203961 25909 consensus_peers.cc:597] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:19.242676 25285 consensus_peers.cc:597] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:19.312222 25287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 83981.3us
W20260709 13:11:19.391701 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 57620.3us
W20260709 13:11:19.411351 25285 consensus_peers.cc:597] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:19.437656 25359 consensus_peers.cc:597] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:19.466631 25285 consensus_peers.cc:597] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:19.536260 25909 consensus_peers.cc:597] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:19.537055 25285 consensus_peers.cc:597] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:19.725701 25285 consensus_peers.cc:597] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:19.738361 25359 consensus_peers.cc:597] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:19.746138 25285 consensus_peers.cc:597] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:19.804569 25359 consensus_peers.cc:597] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:19.818104 25359 consensus_peers.cc:597] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:19.859375 25285 consensus_peers.cc:597] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:19.869673 25359 consensus_peers.cc:597] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:19.895851 25359 consensus_peers.cc:597] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:19.959599 25359 consensus_peers.cc:597] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:19.962333 25359 consensus_peers.cc:597] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:19.992332 25285 consensus_peers.cc:597] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:19.995136 25359 consensus_peers.cc:597] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:20.056137 25285 consensus_peers.cc:597] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:20.117136 25285 consensus_peers.cc:597] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:20.122474 25285 consensus_peers.cc:597] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:20.171295 25359 consensus_peers.cc:597] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:20.189452 25359 consensus_peers.cc:597] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:20.195222 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 106967us
W20260709 13:11:20.195957 25285 consensus_peers.cc:597] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:20.204303 25285 consensus_peers.cc:597] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:20.208439 25359 consensus_peers.cc:597] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:20.257217 25285 consensus_peers.cc:597] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:20.269387 25359 consensus_peers.cc:597] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:20.363196 25359 consensus_peers.cc:597] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:11:20.365836 24113 tablet_server.cc:196] TabletServer@127.23.140.65:0 shutdown complete.
W20260709 13:11:20.374782 25909 consensus_peers.cc:597] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:11:20.382735 25285 consensus_peers.cc:597] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:20.447666 25359 consensus_peers.cc:597] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:20.509507 25285 consensus_peers.cc:597] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:20.619449 25359 consensus_peers.cc:597] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:20.738830 25359 consensus_peers.cc:597] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 13:11:20.760468 24113 tablet_server.cc:179] TabletServer@127.23.140.66:0 shutting down...
W20260709 13:11:20.803304 25358 consensus_peers.cc:597] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:20.810015 25358 consensus_peers.cc:597] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:20.822675 25359 consensus_peers.cc:597] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:20.833858 25358 consensus_peers.cc:597] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:20.836314 25359 consensus_peers.cc:597] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:20.835234 25358 consensus_peers.cc:597] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:20.848068 25908 consensus_peers.cc:597] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:20.871562 25358 consensus_peers.cc:597] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:20.894788 25358 consensus_peers.cc:597] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:20.895714 25908 consensus_peers.cc:597] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:20.899119 25359 consensus_peers.cc:597] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:11:20.967690 25908 consensus_peers.cc:597] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:20.975313 25359 consensus_peers.cc:597] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:20.980525 25358 consensus_peers.cc:597] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:21.002070 25358 consensus_peers.cc:597] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:21.002803 25358 consensus_peers.cc:597] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:21.009462 25358 consensus_peers.cc:597] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:21.010078 25358 consensus_peers.cc:597] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:21.030107 25358 consensus_peers.cc:597] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:21.031169 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260709 13:11:21.033514 25358 consensus_peers.cc:597] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:21.034904 24113 tablet_replica.cc:333] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.037181 24113 raft_consensus.cc:2243] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:11:21.042095 24113 raft_consensus.cc:2272] T d4d43163cfdd4a4094e92fe68bc9259a P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:21.040577 25358 consensus_peers.cc:597] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:21.050508 25358 consensus_peers.cc:597] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:21.055606 24113 tablet_replica.cc:333] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.056644 24113 raft_consensus.cc:2243] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.057317 24113 raft_consensus.cc:2272] T 94b9d971fbc94215a65a2247f2e2acb1 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.060201 24113 tablet_replica.cc:333] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.061036 24113 raft_consensus.cc:2243] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.061806 24113 raft_consensus.cc:2272] T d45c0f8b003848989966aa931c6a7c11 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.064692 24113 tablet_replica.cc:333] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.066530 24113 raft_consensus.cc:2243] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.067333 24113 raft_consensus.cc:2272] T b2cfa29915b34839a60321576f9a67d6 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:21.068929 25908 consensus_peers.cc:597] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:21.070364 24113 tablet_replica.cc:333] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.071383 24113 raft_consensus.cc:2243] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Raft consensus shutting down.
W20260709 13:11:21.065749 25358 consensus_peers.cc:597] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:21.072575 24113 raft_consensus.cc:2272] T 7555c0d2bb294af19f602573931ceb90 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.075501 24113 tablet_replica.cc:333] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.076357 24113 raft_consensus.cc:2243] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.076969 24113 raft_consensus.cc:2272] T e5cbaabf54304de089abde51df6dc2be P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.080044 24113 tablet_replica.cc:333] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.080907 24113 raft_consensus.cc:2243] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.082135 24113 raft_consensus.cc:2272] T 46ab4fd91d3f4766a7b97848f5244360 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:21.083555 25908 consensus_peers.cc:597] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:21.090353 24113 tablet_replica.cc:333] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.091379 24113 raft_consensus.cc:2243] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.092042 24113 raft_consensus.cc:2272] T 7f37bfaeb12446c1a84af172a6f3dde9 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.094906 24113 tablet_replica.cc:333] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.095813 24113 raft_consensus.cc:2243] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.096400 24113 raft_consensus.cc:2272] T 1884b7159da148dbb1c19c424d667792 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.099342 24113 tablet_replica.cc:333] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.100204 24113 raft_consensus.cc:2243] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.100791 24113 raft_consensus.cc:2272] T 72001570ee2c41cb88f0e9eac74992c7 P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.103619 24113 tablet_replica.cc:333] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.110900 24113 raft_consensus.cc:2243] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Raft consensus shutting down.
W20260709 13:11:21.124349 25908 consensus_peers.cc:597] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:21.115103 24113 raft_consensus.cc:2272] T 0e2fe177353b4d6c9de4ccf872ca06ca P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.130666 24113 tablet_replica.cc:333] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.132043 24113 raft_consensus.cc:2243] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.132872 24113 raft_consensus.cc:2272] T 98122796f52e40968d1d8ccd7b032b32 P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.136142 24113 tablet_replica.cc:333] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
W20260709 13:11:21.136197 25908 consensus_peers.cc:597] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:21.137043 24113 raft_consensus.cc:2243] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.138587 24113 raft_consensus.cc:2272] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:21.139397 25908 consensus_peers.cc:597] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:21.141635 24113 tablet_replica.cc:333] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.142536 24113 raft_consensus.cc:2243] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.143756 24113 raft_consensus.cc:2272] T f406a4d41bb9441bba0a83c1d6a56aa3 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.146813 24113 tablet_replica.cc:333] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.147648 24113 raft_consensus.cc:2243] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.148270 24113 raft_consensus.cc:2272] T 9b5c9f65605b4dfca1bbfe0f09629c6d P 2329e428ebbf484fa3ea9452133df567 [term 15 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.151134 24113 tablet_replica.cc:333] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.152091 24113 raft_consensus.cc:2243] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.152704 24113 raft_consensus.cc:2272] T cb389d32141944d4b8f26a17471d50fc P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.155740 24113 tablet_replica.cc:333] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.156534 24113 raft_consensus.cc:2243] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.157550 24113 raft_consensus.cc:2272] T b62f4877af9742298b1e149789ca3090 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.160471 24113 tablet_replica.cc:333] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.161352 24113 raft_consensus.cc:2243] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.162739 24113 raft_consensus.cc:2272] T 8fd399e2f6384e98ada57177a6010d14 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:21.168004 25908 consensus_peers.cc:597] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:21.172403 25358 consensus_peers.cc:597] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:21.172986 25358 consensus_peers.cc:597] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:21.176806 25908 consensus_peers.cc:597] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:21.177174 24113 tablet_replica.cc:333] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.178078 24113 raft_consensus.cc:2243] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:11:21.179229 24113 raft_consensus.cc:2272] T 8fa7e9050d44475381c97025c2b11711 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:21.184441 25358 consensus_peers.cc:597] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:21.184847 25909 consensus_peers.cc:597] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:11:21.187397 24113 tablet_replica.cc:333] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.188335 24113 raft_consensus.cc:2243] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.189222 24113 raft_consensus.cc:2272] T 3cfa8937a5964f5790737950fc3b294e P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.192051 24113 tablet_replica.cc:333] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.193074 24113 raft_consensus.cc:2243] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:11:21.194224 24113 raft_consensus.cc:2272] T 8f209a6539f54de28906492f142efb6b P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.203264 24113 tablet_replica.cc:333] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.204088 24113 raft_consensus.cc:2243] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.204823 24113 raft_consensus.cc:2272] T 4e29e9ab7eda4242964fa76e29c2a221 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.207482 24113 tablet_replica.cc:333] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.208632 24113 raft_consensus.cc:2243] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.209252 24113 raft_consensus.cc:2272] T 6f616cb6823f424a847dceaa5900dfc0 P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.211867 24113 tablet_replica.cc:333] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.212646 24113 raft_consensus.cc:2243] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:11:21.213873 24113 raft_consensus.cc:2272] T b7464c8ac67847019342ea6c88895f77 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.216825 24113 tablet_replica.cc:333] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.218008 24113 raft_consensus.cc:2243] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:11:21.219321 24113 raft_consensus.cc:2272] T a53b1fb6477d486e98bff9549cc487df P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.224387 24113 tablet_replica.cc:333] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.225888 24113 raft_consensus.cc:2243] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:11:21.232010 24113 raft_consensus.cc:2272] T bbeee5054bca4ae28d33636ae50ca11f P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.235039 24113 tablet_replica.cc:333] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.235919 24113 raft_consensus.cc:2243] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.236552 24113 raft_consensus.cc:2272] T 2730c370dd9a4a1ea5adfb543e3ba120 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.239639 24113 tablet_replica.cc:333] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.240408 24113 raft_consensus.cc:2243] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.241091 24113 raft_consensus.cc:2272] T 898e00cfe03c4fc68dc9d6e92a55e996 P 2329e428ebbf484fa3ea9452133df567 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.244059 24113 tablet_replica.cc:333] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.244858 24113 raft_consensus.cc:2243] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:11:21.246057 24113 raft_consensus.cc:2272] T 292e92613a634963a5de1f016eb951ea P 2329e428ebbf484fa3ea9452133df567 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.248708 24113 tablet_replica.cc:333] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.249485 24113 raft_consensus.cc:2243] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.250142 24113 raft_consensus.cc:2272] T adf2f61a8d304d4eb9c30a77d7de17e4 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.252810 24113 tablet_replica.cc:333] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.253593 24113 raft_consensus.cc:2243] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:11:21.259102 24113 raft_consensus.cc:2272] T 1b8652aecc6d4869bb4cad9e0749f3ba P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.262208 24113 tablet_replica.cc:333] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.263090 24113 raft_consensus.cc:2243] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 16 LEADER]: Raft consensus shutting down.
I20260709 13:11:21.264292 24113 raft_consensus.cc:2272] T 58ec1834a96f447f8fe74448e8480342 P 2329e428ebbf484fa3ea9452133df567 [term 16 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.267663 24113 tablet_replica.cc:333] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.268401 24113 raft_consensus.cc:2243] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Raft consensus shutting down.
W20260709 13:11:21.268620 25908 consensus_peers.cc:597] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:21.269274 24113 raft_consensus.cc:2272] T 3cff7b4cb7184e4c9a69372d5901e19a P 2329e428ebbf484fa3ea9452133df567 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.272307 24113 tablet_replica.cc:333] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.273151 24113 raft_consensus.cc:2243] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:11:21.274569 24113 raft_consensus.cc:2272] T d1bed174e340476591fddc8217b8f644 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.277540 24113 tablet_replica.cc:333] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.278287 24113 raft_consensus.cc:2243] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.278904 24113 raft_consensus.cc:2272] T bacfad6698774ab8b26566bcecbcb615 P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.281816 24113 tablet_replica.cc:333] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.282577 24113 raft_consensus.cc:2243] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:11:21.285719 24113 raft_consensus.cc:2272] T 991bcf9b91a14242837dfb4e18891e91 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.289229 24113 tablet_replica.cc:333] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.290565 24113 raft_consensus.cc:2243] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.291301 24113 raft_consensus.cc:2272] T e3025c38ad5a4bdfa61cfca91ba3c409 P 2329e428ebbf484fa3ea9452133df567 [term 21 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:21.304733 25359 consensus_peers.cc:597] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:11:21.314143 24113 tablet_replica.cc:333] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.315778 24113 raft_consensus.cc:2243] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:11:21.317073 24113 raft_consensus.cc:2272] T 2dc23779efbf4518a6ada8bde8243a11 P 2329e428ebbf484fa3ea9452133df567 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.319963 24113 tablet_replica.cc:333] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.321585 24113 raft_consensus.cc:2243] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.322222 24113 raft_consensus.cc:2272] T de96ce5d28144cf3894e100bfc4959fa P 2329e428ebbf484fa3ea9452133df567 [term 25 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:21.322568 25909 consensus_peers.cc:597] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:11:21.325084 24113 tablet_replica.cc:333] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.325933 24113 raft_consensus.cc:2243] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:11:21.327275 24113 raft_consensus.cc:2272] T f75b60146e7d4b9b9954b4fb2c414679 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.329873 24113 tablet_replica.cc:333] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.330727 24113 raft_consensus.cc:2243] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.331343 24113 raft_consensus.cc:2272] T e81523f4f4344a04a1e0b03c8de99290 P 2329e428ebbf484fa3ea9452133df567 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.334097 24113 tablet_replica.cc:333] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.335129 24113 raft_consensus.cc:2243] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.335716 24113 raft_consensus.cc:2272] T eda233ace2a24b379fef320a5f610c7a P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.338302 24113 tablet_replica.cc:333] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.339118 24113 raft_consensus.cc:2243] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 17 LEADER]: Raft consensus shutting down.
I20260709 13:11:21.342592 24113 raft_consensus.cc:2272] T 6afb077d630246d6a315a828b4dcc108 P 2329e428ebbf484fa3ea9452133df567 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.347102 24113 tablet_replica.cc:333] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
W20260709 13:11:21.347491 25909 consensus_peers.cc:597] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:11:21.347857 24113 raft_consensus.cc:2243] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.348624 24113 raft_consensus.cc:2272] T cf0501919f944794bbab0ffae6a334e1 P 2329e428ebbf484fa3ea9452133df567 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.351449 24113 tablet_replica.cc:333] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.352222 24113 raft_consensus.cc:2243] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:11:21.353428 24113 raft_consensus.cc:2272] T 2f9c4bd911cc450aa35a16a5d10bf16c P 2329e428ebbf484fa3ea9452133df567 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.356045 24113 tablet_replica.cc:333] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.356786 24113 raft_consensus.cc:2243] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.357736 24113 raft_consensus.cc:2272] T 3bd9cfccf308499991b366150c2d46ea P 2329e428ebbf484fa3ea9452133df567 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.360328 24113 tablet_replica.cc:333] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.361014 24113 raft_consensus.cc:2243] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:11:21.362180 24113 raft_consensus.cc:2272] T acd2d31837dd470c8bd91f719ffeb461 P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:21.364758 24113 tablet_replica.cc:333] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567: stopping tablet replica
I20260709 13:11:21.366058 24113 raft_consensus.cc:2243] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:21.366791 24113 raft_consensus.cc:2272] T 9dac81b348b746fd8ecd3937028554ea P 2329e428ebbf484fa3ea9452133df567 [term 19 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:21.378541 25358 consensus_peers.cc:597] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:21.528301 25909 consensus_peers.cc:597] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:11:21.612711 25359 consensus_peers.cc:597] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:21.623183 25909 consensus_peers.cc:597] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:11:21.784731 25359 consensus_peers.cc:597] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:11:22.009904 25359 consensus_peers.cc:597] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:11:22.069553   832 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:22.070163   832 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:22.073613   832 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:22.081933 25909 leader_election.cc:336] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:22.090917 25908 leader_election.cc:336] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:22.091540 25908 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:22.092412   832 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:11:22.134621 25909 consensus_peers.cc:597] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:22.139286 25359 consensus_peers.cc:597] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:11:22.168005 25359 consensus_peers.cc:597] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:11:22.303228   814 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:22.303793   814 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:11:22.317183 25909 leader_election.cc:336] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:22.319003   814 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:22.322578   839 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:22.323175   839 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:22.328667   839 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
W20260709 13:11:22.337606 25908 leader_election.cc:336] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:22.377673 25908 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:22.370905   832 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:22.381175   832 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:22.383450   832 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:22.368665 25358 leader_election.cc:336] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:22.375874   836 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:22.385154   836 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:22.387545   836 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:22.396143 25909 leader_election.cc:336] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:22.397106 25908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 59524.2us
I20260709 13:11:22.397622   832 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:22.400560 25940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:22.405858 25358 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, f646dfd532974dceadfe00005d2bbf59
W20260709 13:11:22.406769 25908 leader_election.cc:336] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:22.411278 25908 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:22.411657   836 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:11:22.412132   832 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:11:22.412613 25358 leader_election.cc:336] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
W20260709 13:11:22.409179 25359 consensus_peers.cc:597] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:11:22.416576 25359 leader_election.cc:336] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:22.424597 25359 leader_election.cc:336] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:22.425189 25359 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:22.426497   836 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:11:22.427546 25359 consensus_peers.cc:597] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:11:22.432219   839 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:22.432737   839 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:11:22.433930 25359 consensus_peers.cc:597] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:11:22.434757   839 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:22.435036   832 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:22.435580   832 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:22.437736   832 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:22.441836 25908 leader_election.cc:336] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
W20260709 13:11:22.443485 25909 leader_election.cc:336] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:22.443892 25909 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:22.445140   832 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:22.479658   832 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:22.481635   832 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:22.491371   828 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:22.497980   832 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
W20260709 13:11:22.494462 25358 leader_election.cc:336] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:22.496332   842 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:22.492352   821 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:22.499109   842 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:22.499877   821 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:22.501991   842 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:22.503222   821 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:22.503842 24113 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 25342)
W20260709 13:11:22.503887 25359 leader_election.cc:336] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:22.504237   832 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:22.498164   828 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:22.508618   832 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:22.511622 25359 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:11:22.512158 25909 leader_election.cc:336] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:22.515619   828 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:22.514635   832 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:22.521051 25908 leader_election.cc:336] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:22.521729 25908 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:22.523788   832 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:11:22.524264 25909 leader_election.cc:336] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:22.524878   821 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:11:22.532553 25908 leader_election.cc:336] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:22.533056 25908 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:22.534279   832 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:11:22.536619 25358 leader_election.cc:336] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
W20260709 13:11:22.537292 25358 leader_election.cc:336] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
W20260709 13:11:22.541644 25359 consensus_peers.cc:597] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:11:22.553822 25359 consensus_peers.cc:597] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:11:22.557874 25359 leader_election.cc:336] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:22.558346 25359 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:22.559198   828 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:11:22.560374 25359 leader_election.cc:336] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:22.560832 25359 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:22.561703   834 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:11:22.557240 25358 leader_election.cc:336] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
W20260709 13:11:22.564105 25359 leader_election.cc:336] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:22.572858 25359 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:22.576479   828 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:11:22.588639   832 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:22.589474   828 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:22.589615   832 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:22.590404   828 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:22.592509   828 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:22.594702   832 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:22.598989 25908 leader_election.cc:336] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
W20260709 13:11:22.601281 25909 leader_election.cc:336] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:22.601797 25909 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:22.602591   832 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:11:22.618784 25359 consensus_peers.cc:597] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:11:22.631877 25358 leader_election.cc:336] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
W20260709 13:11:22.648940 25359 leader_election.cc:336] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:22.649454 25359 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:22.650463   821 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:22.678507   842 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:22.679216   842 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:22.686343   842 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:22.692076 25358 leader_election.cc:336] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
W20260709 13:11:22.711566 25359 leader_election.cc:336] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:22.712625 25359 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:22.713485   842 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:11:22.744316 25359 consensus_peers.cc:597] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:11:22.757104   842 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:22.757869   842 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:11:22.762782 25359 consensus_peers.cc:597] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:11:22.766649 25359 leader_election.cc:336] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:22.766920   842 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:22.767419 25358 leader_election.cc:336] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:22.768179 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:22.771700   828 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:11:22.777805 25359 consensus_peers.cc:597] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:11:22.801110   830 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:22.802057   830 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:22.805871   830 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:22.808781 25387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "f646dfd532974dceadfe00005d2bbf59" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" is_pre_election: true
I20260709 13:11:22.809787 25387 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f646dfd532974dceadfe00005d2bbf59 in term 19.
W20260709 13:11:22.813953 25909 leader_election.cc:336] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:22.817718   828 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:22.818434   828 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:22.820430   828 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:22.825585 25359 leader_election.cc:336] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:22.832751 25358 leader_election.cc:336] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:22.833307 25358 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:22.834107   828 raft_consensus.cc:2749] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:11:22.859450 25175 debug-util.cc:398] Leaking SignalData structure 0x7b0800175480 after lost signal to thread 24116
W20260709 13:11:22.861248 25175 debug-util.cc:398] Leaking SignalData structure 0x7b08002a50c0 after lost signal to thread 25178
W20260709 13:11:22.863197 25175 debug-util.cc:398] Leaking SignalData structure 0x7b08002b7660 after lost signal to thread 25419
W20260709 13:11:22.865020 25175 debug-util.cc:398] Leaking SignalData structure 0x7b08002a0f60 after lost signal to thread 25969
W20260709 13:11:22.941609 25359 consensus_peers.cc:597] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:11:23.082468 25358 consensus_peers.cc:597] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.124464 25359 consensus_peers.cc:597] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:23.194355 25358 consensus_peers.cc:597] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.195039 25358 consensus_peers.cc:597] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.216521 25358 consensus_peers.cc:597] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.217720 25909 consensus_peers.cc:597] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:11:23.230302 25359 consensus_peers.cc:597] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:11:23.506699 24113 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 25342)
W20260709 13:11:23.767745 25909 consensus_peers.cc:597] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:11:23.833024 25358 consensus_peers.cc:597] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.834489 25358 consensus_peers.cc:597] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.847653 25908 leader_election.cc:336] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:23.850370 25908 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:23.857398   830 raft_consensus.cc:2749] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:11:23.860773 25908 consensus_peers.cc:597] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.876791 25358 consensus_peers.cc:597] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.877529 25358 consensus_peers.cc:597] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.917260 25359 consensus_peers.cc:597] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:11:23.929103 25358 consensus_peers.cc:597] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.933351 25909 consensus_peers.cc:597] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:11:23.932596   848 raft_consensus.cc:493] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:23.945041   848 raft_consensus.cc:515] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:23.947511   848 leader_election.cc:290] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:23.948256 25359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111) [suppressed 572 similar messages]
W20260709 13:11:23.949187 25359 consensus_peers.cc:597] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:11:23.949829 25359 consensus_peers.cc:597] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:11:23.951016 25359 consensus_peers.cc:597] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:23.951625 25359 consensus_peers.cc:597] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:23.931872 25358 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42315, user_credentials={real_user=slave}} blocked reactor thread for 53827us
W20260709 13:11:23.956872 25359 consensus_peers.cc:597] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:11:23.957792 25359 leader_election.cc:336] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:23.961517 25908 consensus_peers.cc:597] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.965235 25911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:34661, user_credentials={real_user=slave}} blocked reactor thread for 48100.8us
W20260709 13:11:23.959405 25358 consensus_peers.cc:597] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.969902 25358 consensus_peers.cc:597] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.982328 25358 consensus_peers.cc:597] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.987372 25358 consensus_peers.cc:597] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.989394 25358 consensus_peers.cc:597] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.989955 25358 consensus_peers.cc:597] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.993383 25358 consensus_peers.cc:597] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.991640 25908 consensus_peers.cc:597] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:23.996008 25358 consensus_peers.cc:597] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:24.006201 25358 consensus_peers.cc:597] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:24.006698 25358 consensus_peers.cc:597] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:24.007172 25358 consensus_peers.cc:597] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:24.007555 25358 leader_election.cc:336] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:24.007913 25358 leader_election.cc:304] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:24.009311   846 raft_consensus.cc:2749] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:11:24.031442 25908 consensus_peers.cc:597] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:24.051718 25908 consensus_peers.cc:597] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:24.074771   832 raft_consensus.cc:493] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:24.075625   832 raft_consensus.cc:515] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:24.079227   832 leader_election.cc:290] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:24.080825   834 raft_consensus.cc:493] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:24.081508   834 raft_consensus.cc:515] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:24.084051   834 leader_election.cc:290] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:24.085088   849 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:24.085814   849 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:24.088325   849 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:24.092250 25358 leader_election.cc:336] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
W20260709 13:11:24.093894 25359 leader_election.cc:336] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:24.094311 25359 leader_election.cc:304] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:24.095502   834 raft_consensus.cc:2749] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:11:24.090837 25908 consensus_peers.cc:597] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:24.103268 25909 leader_election.cc:336] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:24.103950 25909 leader_election.cc:336] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:24.111250 25908 leader_election.cc:336] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:24.111901 25908 leader_election.cc:304] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:24.112778   832 raft_consensus.cc:2749] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:11:24.114027 25908 leader_election.cc:336] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:24.114547 25908 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:24.115381   832 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:11:24.123980 25908 consensus_peers.cc:597] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:24.172451   834 raft_consensus.cc:493] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:24.173305   834 raft_consensus.cc:515] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:11:24.181828 25359 leader_election.cc:336] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:24.183359   834 leader_election.cc:290] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:24.187579 25358 leader_election.cc:336] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:24.187999 25358 leader_election.cc:304] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:24.188865   834 raft_consensus.cc:2749] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:11:24.194571 25909 consensus_peers.cc:597] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:11:24.195190 25909 consensus_peers.cc:597] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:11:24.200400 25908 consensus_peers.cc:597] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:11:24.236982   832 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:24.238615   832 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:24.243556   849 raft_consensus.cc:493] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:24.250674   849 raft_consensus.cc:515] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:11:24.247979 25359 consensus_peers.cc:597] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 13:11:24.252986   849 leader_election.cc:290] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:24.257117   832 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:24.280365 25909 leader_election.cc:336] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:24.290594 25909 leader_election.cc:336] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:24.296860 25908 leader_election.cc:336] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:24.297483 25908 leader_election.cc:304] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:24.320154   832 raft_consensus.cc:2749] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:11:24.321296 25908 leader_election.cc:336] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:24.325752 25908 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:24.326978   849 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:24.327704   849 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:24.330178   849 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
I20260709 13:11:24.326589   832 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:11:24.340726 25909 leader_election.cc:336] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:24.360630   842 raft_consensus.cc:493] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:24.361403   842 raft_consensus.cc:515] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:24.364065   842 leader_election.cc:290] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:24.404358 25909 consensus_peers.cc:597] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:11:24.419023 25908 consensus_peers.cc:597] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:24.421160 25359 leader_election.cc:336] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:24.426082   844 raft_consensus.cc:493] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:24.438150   844 raft_consensus.cc:515] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:24.440404   844 leader_election.cc:290] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:24.460637 25908 consensus_peers.cc:597] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:24.461295 25908 leader_election.cc:336] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:24.461748 25908 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:24.462528   832 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:24.473376   844 raft_consensus.cc:493] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:24.473982   844 raft_consensus.cc:515] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:24.476051   844 leader_election.cc:290] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:24.486797 25358 leader_election.cc:336] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:24.491760 25358 leader_election.cc:304] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:24.494098   828 raft_consensus.cc:2749] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:11:24.502646 25359 leader_election.cc:336] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:24.519923 25358 leader_election.cc:336] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:24.523448 25358 leader_election.cc:304] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:11:24.522893 25359 leader_election.cc:336] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:24.524603 25359 consensus_peers.cc:597] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:11:24.526973   828 raft_consensus.cc:2749] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:11:24.539191 25909 consensus_peers.cc:597] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:24.543126 25358 leader_election.cc:336] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:24.546172 25358 leader_election.cc:304] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:24.547099   828 raft_consensus.cc:2749] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:11:24.582427   828 raft_consensus.cc:493] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:24.582947   828 raft_consensus.cc:515] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:24.584954   828 leader_election.cc:290] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:24.594924 25359 leader_election.cc:336] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:24.595578 25359 consensus_peers.cc:597] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:11:24.611750 25358 leader_election.cc:336] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:24.612380 25358 leader_election.cc:304] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:24.612982   828 raft_consensus.cc:2749] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:11:24.640163   832 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:24.640784   832 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:24.643190   832 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
W20260709 13:11:24.648358 25908 leader_election.cc:336] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
W20260709 13:11:24.649595 25909 leader_election.cc:336] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:24.651283 25909 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:24.652009   832 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:11:24.729526   828 raft_consensus.cc:493] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:24.730149   828 raft_consensus.cc:515] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
W20260709 13:11:24.737156 25359 leader_election.cc:336] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:24.739712   828 leader_election.cc:290] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:24.743525 25358 leader_election.cc:336] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:24.743986 25358 leader_election.cc:304] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:24.744589   844 raft_consensus.cc:2749] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:11:24.755190   844 raft_consensus.cc:493] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:24.755622   844 raft_consensus.cc:515] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } }
I20260709 13:11:24.757637   844 leader_election.cc:290] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:24.764114 25359 leader_election.cc:336] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:24.788069 25359 consensus_peers.cc:597] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:11:24.792256 25359 consensus_peers.cc:597] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:11:24.804701 25908 consensus_peers.cc:597] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:11:24.819864 25359 consensus_peers.cc:597] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:11:24.822742   832 raft_consensus.cc:493] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:24.823288   832 raft_consensus.cc:515] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:24.825623   832 leader_election.cc:290] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:24.829929 25908 leader_election.cc:336] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
W20260709 13:11:24.830713 25909 leader_election.cc:336] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:24.831104 25909 leader_election.cc:304] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:24.831795   832 raft_consensus.cc:2749] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:11:24.865319 25416 debug-util.cc:398] Leaking SignalData structure 0x7b080037cee0 after lost signal to thread 24116
W20260709 13:11:24.866214 25416 debug-util.cc:398] Leaking SignalData structure 0x7b08003b6bc0 after lost signal to thread 25178
W20260709 13:11:24.867278 25416 debug-util.cc:398] Leaking SignalData structure 0x7b080034a620 after lost signal to thread 25419
W20260709 13:11:24.868479 25416 debug-util.cc:398] Leaking SignalData structure 0x7b08003cc600 after lost signal to thread 25969
W20260709 13:11:25.009095 25359 consensus_peers.cc:597] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:11:25.093190 24113 tablet_server.cc:196] TabletServer@127.23.140.66:0 shutdown complete.
I20260709 13:11:25.234437 24113 tablet_server.cc:179] TabletServer@127.23.140.67:0 shutting down...
W20260709 13:11:25.246003 25911 consensus_peers.cc:597] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 -> Peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Couldn't send request to peer a2a77ecf2c0941618b5bc79d4c04db61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:25.250835 25911 consensus_peers.cc:597] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 -> Peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Couldn't send request to peer a2a77ecf2c0941618b5bc79d4c04db61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:25.291353   828 raft_consensus.cc:493] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:25.291893   828 raft_consensus.cc:515] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:11:25.293522 25911 consensus_peers.cc:597] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 -> Peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Couldn't send request to peer a2a77ecf2c0941618b5bc79d4c04db61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:25.294396   828 leader_election.cc:290] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), f646dfd532974dceadfe00005d2bbf59 (127.23.140.68:39211)
I20260709 13:11:25.297961 25938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f75b60146e7d4b9b9954b4fb2c414679" candidate_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f646dfd532974dceadfe00005d2bbf59" is_pre_election: true
I20260709 13:11:25.298599 25938 raft_consensus.cc:2468] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a77ecf2c0941618b5bc79d4c04db61 in term 19.
W20260709 13:11:25.302649 25359 leader_election.cc:336] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:25.307425 25911 consensus_peers.cc:597] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 -> Peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Couldn't send request to peer a2a77ecf2c0941618b5bc79d4c04db61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:11:25.320485 25359 consensus_peers.cc:597] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:11:25.335976 25359 consensus_peers.cc:597] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:11:25.345105 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:11:25.347875 24113 tablet_replica.cc:333] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.348862 24113 raft_consensus.cc:2243] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:25.349762 24113 raft_consensus.cc:2272] T d45c0f8b003848989966aa931c6a7c11 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.352875 24113 tablet_replica.cc:333] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.353879 24113 raft_consensus.cc:2243] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.355162 24113 raft_consensus.cc:2272] T 63c3ef75ef234051b7ae1ea395400f54 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.358981 24113 tablet_replica.cc:333] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.359877 24113 raft_consensus.cc:2243] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.360984 24113 raft_consensus.cc:2272] T 9c23d7a710c04baf86ab9ba1a2ddc9cf P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.363801 24113 tablet_replica.cc:333] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.364712 24113 raft_consensus.cc:2243] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Raft consensus shutting down.
W20260709 13:11:25.365742 25911 consensus_peers.cc:597] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 -> Peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Couldn't send request to peer a2a77ecf2c0941618b5bc79d4c04db61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:25.365772 24113 raft_consensus.cc:2272] T 98122796f52e40968d1d8ccd7b032b32 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.369716 24113 tablet_replica.cc:333] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.370718 24113 raft_consensus.cc:2243] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.371829 24113 raft_consensus.cc:2272] T 46ab4fd91d3f4766a7b97848f5244360 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.374801 24113 tablet_replica.cc:333] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.375687 24113 raft_consensus.cc:2243] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 LEADER]: Raft consensus shutting down.
W20260709 13:11:25.384634 25359 consensus_peers.cc:597] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:11:25.384944 24113 raft_consensus.cc:2272] T 1884b7159da148dbb1c19c424d667792 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.387809 24113 tablet_replica.cc:333] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.388741 24113 raft_consensus.cc:2243] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.389866 24113 raft_consensus.cc:2272] T 72001570ee2c41cb88f0e9eac74992c7 P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:25.389984 25359 consensus_peers.cc:597] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:11:25.392792 24113 tablet_replica.cc:333] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.394528 24113 raft_consensus.cc:2243] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.396477 24113 raft_consensus.cc:2272] T 37bfef8566b143cf94971f26f417678c P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.399469 24113 tablet_replica.cc:333] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.400482 24113 raft_consensus.cc:2243] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.401898 24113 raft_consensus.cc:2272] T 5ead2e92f0614db79c9249687c4246f9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.404831 24113 tablet_replica.cc:333] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.405690 24113 raft_consensus.cc:2243] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.406749 24113 raft_consensus.cc:2272] T 7f37bfaeb12446c1a84af172a6f3dde9 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.409606 24113 tablet_replica.cc:333] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.410504 24113 raft_consensus.cc:2243] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.411815 24113 raft_consensus.cc:2272] T b6d19448306e4052a13c08a582f4ccfa P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.414521 24113 tablet_replica.cc:333] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.415365 24113 raft_consensus.cc:2243] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.416410 24113 raft_consensus.cc:2272] T 0e2fe177353b4d6c9de4ccf872ca06ca P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.419209 24113 tablet_replica.cc:333] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.421020 24113 raft_consensus.cc:2243] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:25.421960 24113 raft_consensus.cc:2272] T f406a4d41bb9441bba0a83c1d6a56aa3 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.424782 24113 tablet_replica.cc:333] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.425644 24113 raft_consensus.cc:2243] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:25.426606 24113 raft_consensus.cc:2272] T c5481170c5c74798bdfd146068f8b539 P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.429865 24113 tablet_replica.cc:333] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.430866 24113 raft_consensus.cc:2243] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:25.431919 24113 raft_consensus.cc:2272] T 9b5c9f65605b4dfca1bbfe0f09629c6d P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.435156 24113 tablet_replica.cc:333] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.436043 24113 raft_consensus.cc:2243] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.437156 24113 raft_consensus.cc:2272] T cb389d32141944d4b8f26a17471d50fc P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:25.438654 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:11:25.440021 24113 tablet_replica.cc:333] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.440920 24113 raft_consensus.cc:2243] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.442225 24113 raft_consensus.cc:2272] T b62f4877af9742298b1e149789ca3090 P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.444941 24113 tablet_replica.cc:333] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.445827 24113 raft_consensus.cc:2243] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.447088 24113 raft_consensus.cc:2272] T 993e3b05a73a4403b04c3b55a5b48abe P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:25.448450 25911 consensus_peers.cc:597] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 -> Peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Couldn't send request to peer a2a77ecf2c0941618b5bc79d4c04db61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:25.449973 24113 tablet_replica.cc:333] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.450847 24113 raft_consensus.cc:2243] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:25.451630 24113 raft_consensus.cc:2272] T 8fd399e2f6384e98ada57177a6010d14 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.454438 24113 tablet_replica.cc:333] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.455295 24113 raft_consensus.cc:2243] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.456508 24113 raft_consensus.cc:2272] T 3cfa8937a5964f5790737950fc3b294e P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.459565 24113 tablet_replica.cc:333] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.460376 24113 raft_consensus.cc:2243] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:25.460969 24113 raft_consensus.cc:2272] T 4e29e9ab7eda4242964fa76e29c2a221 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.463511 24113 tablet_replica.cc:333] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.464295 24113 raft_consensus.cc:2243] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.465389 24113 raft_consensus.cc:2272] T 6f616cb6823f424a847dceaa5900dfc0 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.468039 24113 tablet_replica.cc:333] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.468823 24113 raft_consensus.cc:2243] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:25.469461 24113 raft_consensus.cc:2272] T b7464c8ac67847019342ea6c88895f77 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.472285 24113 tablet_replica.cc:333] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.473047 24113 raft_consensus.cc:2243] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:25.473834 24113 raft_consensus.cc:2272] T bbeee5054bca4ae28d33636ae50ca11f P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.476450 24113 tablet_replica.cc:333] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.477250 24113 raft_consensus.cc:2243] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.478295 24113 raft_consensus.cc:2272] T 2730c370dd9a4a1ea5adfb543e3ba120 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:25.480662 25911 consensus_peers.cc:597] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 -> Peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Couldn't send request to peer a2a77ecf2c0941618b5bc79d4c04db61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:25.480997 24113 tablet_replica.cc:333] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.481971 24113 raft_consensus.cc:2243] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:25.482650 24113 raft_consensus.cc:2272] T 1b8652aecc6d4869bb4cad9e0749f3ba P a2a77ecf2c0941618b5bc79d4c04db61 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.485486 24113 tablet_replica.cc:333] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.486394 24113 raft_consensus.cc:2243] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:25.487188 24113 raft_consensus.cc:2272] T 58ec1834a96f447f8fe74448e8480342 P a2a77ecf2c0941618b5bc79d4c04db61 [term 16 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.490186 24113 tablet_replica.cc:333] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
W20260709 13:11:25.490705 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:11:25.491052 24113 raft_consensus.cc:2243] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.492133 24113 raft_consensus.cc:2272] T 3cff7b4cb7184e4c9a69372d5901e19a P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.494864 24113 tablet_replica.cc:333] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.495700 24113 raft_consensus.cc:2243] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:25.497356 24113 raft_consensus.cc:2272] T 898e00cfe03c4fc68dc9d6e92a55e996 P a2a77ecf2c0941618b5bc79d4c04db61 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.500289 24113 tablet_replica.cc:333] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.501300 24113 raft_consensus.cc:2243] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.502477 24113 raft_consensus.cc:2272] T 17ef4786898b4ad696baed95b4209788 P a2a77ecf2c0941618b5bc79d4c04db61 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.505215 24113 tablet_replica.cc:333] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.505972 24113 raft_consensus.cc:2243] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.506997 24113 raft_consensus.cc:2272] T 56914a99b2d8417592e1f08979cfadf6 P a2a77ecf2c0941618b5bc79d4c04db61 [term 15 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.509729 24113 tablet_replica.cc:333] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.510515 24113 raft_consensus.cc:2243] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.511540 24113 raft_consensus.cc:2272] T de96ce5d28144cf3894e100bfc4959fa P a2a77ecf2c0941618b5bc79d4c04db61 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.514526 24113 tablet_replica.cc:333] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.515286 24113 raft_consensus.cc:2243] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.516290 24113 raft_consensus.cc:2272] T e3025c38ad5a4bdfa61cfca91ba3c409 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.518921 24113 tablet_replica.cc:333] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.519753 24113 raft_consensus.cc:2243] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:25.520722 24113 raft_consensus.cc:2272] T d1bed174e340476591fddc8217b8f644 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.523495 24113 tablet_replica.cc:333] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.524276 24113 raft_consensus.cc:2243] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.525522 24113 raft_consensus.cc:2272] T bacfad6698774ab8b26566bcecbcb615 P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.528162 24113 tablet_replica.cc:333] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.529636 24113 raft_consensus.cc:2243] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:25.530431 24113 raft_consensus.cc:2272] T 69fee8cad2c84e1492403c23786622a1 P a2a77ecf2c0941618b5bc79d4c04db61 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.533212 24113 tablet_replica.cc:333] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.533972 24113 raft_consensus.cc:2243] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:25.534634 24113 raft_consensus.cc:2272] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:25.534703 25911 consensus_peers.cc:597] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 -> Peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Couldn't send request to peer a2a77ecf2c0941618b5bc79d4c04db61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:25.537457 24113 tablet_replica.cc:333] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.538281 24113 raft_consensus.cc:2243] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:25.539080 24113 raft_consensus.cc:2272] T 6afb077d630246d6a315a828b4dcc108 P a2a77ecf2c0941618b5bc79d4c04db61 [term 17 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:25.540518 25359 consensus_peers.cc:597] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:11:25.542160 24113 tablet_replica.cc:333] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
W20260709 13:11:25.542510 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:11:25.542981 24113 raft_consensus.cc:2243] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:25.543843 24113 raft_consensus.cc:2272] T 2f9c4bd911cc450aa35a16a5d10bf16c P a2a77ecf2c0941618b5bc79d4c04db61 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.546492 24113 tablet_replica.cc:333] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.547266 24113 raft_consensus.cc:2243] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.548583 24113 raft_consensus.cc:2272] T 6ff9a7b00f104dd294125870b06a7df4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.551246 24113 tablet_replica.cc:333] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.552021 24113 raft_consensus.cc:2243] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.553239 24113 raft_consensus.cc:2272] T 17584a19690d4010851d65378cfcbafa P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.556025 24113 tablet_replica.cc:333] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.556705 24113 raft_consensus.cc:2243] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 LEADER]: Raft consensus shutting down.
W20260709 13:11:25.557255 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:11:25.557670 24113 raft_consensus.cc:2272] T adf2f61a8d304d4eb9c30a77d7de17e4 P a2a77ecf2c0941618b5bc79d4c04db61 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.560346 24113 tablet_replica.cc:333] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.561033 24113 raft_consensus.cc:2243] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:25.561836 24113 raft_consensus.cc:2272] T 292e92613a634963a5de1f016eb951ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.564380 24113 tablet_replica.cc:333] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
W20260709 13:11:25.564757 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:11:25.565145 24113 raft_consensus.cc:2243] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.566900 24113 raft_consensus.cc:2272] T 3bd9cfccf308499991b366150c2d46ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.569906 24113 tablet_replica.cc:333] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.570621 24113 raft_consensus.cc:2243] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.571600 24113 raft_consensus.cc:2272] T e5cbaabf54304de089abde51df6dc2be P a2a77ecf2c0941618b5bc79d4c04db61 [term 20 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:25.574038 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:11:25.574299 24113 tablet_replica.cc:333] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.575048 24113 raft_consensus.cc:2243] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:25.576112 24113 raft_consensus.cc:2272] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:25.578953 24113 tablet_replica.cc:333] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61: stopping tablet replica
I20260709 13:11:25.579591 24113 raft_consensus.cc:2243] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:11:25.580336 24113 raft_consensus.cc:2272] T 9dac81b348b746fd8ecd3937028554ea P a2a77ecf2c0941618b5bc79d4c04db61 [term 19 FOLLOWER]: Raft consensus is shut down!
W20260709 13:11:25.626422 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:11:25.632766 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:11:25.654386 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:11:25.659636 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:11:25.690563 25361 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:11:25.722764 25359 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:11:25.779379 25358 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:11:25.788026 25360 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:11:25.790859   635 leader_election.cc:336] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Service unavailable: reactor is shutting down (error 108)
I20260709 13:11:25.791342   635 leader_election.cc:304] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61, f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:11:25.791952   635 raft_consensus.cc:2686] T f75b60146e7d4b9b9954b4fb2c414679 P a2a77ecf2c0941618b5bc79d4c04db61: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260709 13:11:25.807212 25909 consensus_peers.cc:597] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:11:25.807667   635 leader_election.cc:336] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Service unavailable: reactor is shutting down (error 108)
I20260709 13:11:25.808197   635 leader_election.cc:304] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a77ecf2c0941618b5bc79d4c04db61; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
W20260709 13:11:25.808864   635 raft_consensus.cc:2686] T acd2d31837dd470c8bd91f719ffeb461 P a2a77ecf2c0941618b5bc79d4c04db61: unable to run election callback: Service unavailable: The pool has been shut down.
W20260709 13:11:25.991534 25908 consensus_peers.cc:597] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:26.029053 25909 consensus_peers.cc:597] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 13:11:26.129482   854 raft_consensus.cc:493] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:26.130100   854 raft_consensus.cc:515] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:26.132508   854 leader_election.cc:290] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:26.137859 25909 leader_election.cc:336] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:26.138631 25908 leader_election.cc:336] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:26.139034 25908 leader_election.cc:304] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:26.139760   854 raft_consensus.cc:2749] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:26.215945   854 raft_consensus.cc:493] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:26.216425   854 raft_consensus.cc:515] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:26.218257   854 leader_election.cc:290] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:26.225623 25909 leader_election.cc:336] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:26.226186 25908 leader_election.cc:336] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:26.226584 25908 leader_election.cc:304] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:26.227172   854 raft_consensus.cc:2749] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:11:26.329670   854 raft_consensus.cc:493] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:26.330178   854 raft_consensus.cc:515] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:26.332072   854 leader_election.cc:290] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315)
W20260709 13:11:26.336881 25909 leader_election.cc:336] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:26.337425 25908 leader_election.cc:336] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:26.337844 25908 leader_election.cc:304] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:26.338426   854 raft_consensus.cc:2749] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:11:26.358078   852 raft_consensus.cc:493] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:26.358654   852 raft_consensus.cc:515] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:26.360694   852 leader_election.cc:290] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:11:26.365823 25911 leader_election.cc:336] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Network error: Client connection negotiation failed: client connection to 127.23.140.67:34661: connect: Connection refused (error 111)
W20260709 13:11:26.367123 25908 leader_election.cc:336] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:26.367501 25908 leader_election.cc:304] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:26.368085   852 raft_consensus.cc:2749] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:11:26.409446 25908 consensus_peers.cc:597] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:26.416613   852 raft_consensus.cc:493] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:26.417193   852 raft_consensus.cc:515] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:26.419283   852 leader_election.cc:290] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:11:26.424369 25911 leader_election.cc:336] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Network error: Client connection negotiation failed: client connection to 127.23.140.67:34661: connect: Connection refused (error 111)
W20260709 13:11:26.425580 25908 consensus_peers.cc:597] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:26.428032 25908 leader_election.cc:336] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:26.428380 25908 leader_election.cc:304] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:26.428960   852 raft_consensus.cc:2749] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:11:26.450225 25909 consensus_peers.cc:597] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 13:11:26.471632   852 raft_consensus.cc:493] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:26.472210   852 raft_consensus.cc:515] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:26.474972   852 leader_election.cc:290] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:11:26.478832 25909 leader_election.cc:336] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:26.479743 25911 leader_election.cc:336] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Network error: Client connection negotiation failed: client connection to 127.23.140.67:34661: connect: Connection refused (error 111)
I20260709 13:11:26.480094 25911 leader_election.cc:304] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:26.480686   852 raft_consensus.cc:2749] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:11:26.485323   852 raft_consensus.cc:493] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:26.485850   852 raft_consensus.cc:515] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:26.487404   852 leader_election.cc:290] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:11:26.493326 25909 consensus_peers.cc:597] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:11:26.495332 25911 leader_election.cc:336] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Network error: Client connection negotiation failed: client connection to 127.23.140.67:34661: connect: Connection refused (error 111)
W20260709 13:11:26.496585 25909 leader_election.cc:336] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:26.497020 25909 leader_election.cc:304] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:26.497779   854 raft_consensus.cc:2749] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:11:26.511890 25908 consensus_peers.cc:597] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:26.530560   852 raft_consensus.cc:493] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:26.531167   852 raft_consensus.cc:515] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:26.533085   852 leader_election.cc:290] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:11:26.537513 25909 leader_election.cc:336] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:26.537999 25911 leader_election.cc:336] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Network error: Client connection negotiation failed: client connection to 127.23.140.67:34661: connect: Connection refused (error 111)
I20260709 13:11:26.538434 25911 leader_election.cc:304] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:26.539017   852 raft_consensus.cc:2749] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:11:26.575923   852 raft_consensus.cc:493] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:26.576427   852 raft_consensus.cc:515] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:26.578336   852 leader_election.cc:290] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:11:26.582767 25908 leader_election.cc:336] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
W20260709 13:11:26.583209 25911 leader_election.cc:336] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Network error: Client connection negotiation failed: client connection to 127.23.140.67:34661: connect: Connection refused (error 111)
I20260709 13:11:26.583642 25911 leader_election.cc:304] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:26.584280   852 raft_consensus.cc:2749] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:11:26.604749   852 raft_consensus.cc:493] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:26.605243   852 raft_consensus.cc:515] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:11:26.611905 25909 leader_election.cc:336] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:26.612303   852 leader_election.cc:290] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:11:26.615804 25911 leader_election.cc:336] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Network error: Client connection negotiation failed: client connection to 127.23.140.67:34661: connect: Connection refused (error 111)
I20260709 13:11:26.616175 25911 leader_election.cc:304] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:26.616756   852 raft_consensus.cc:2749] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:11:26.684641 25909 consensus_peers.cc:597] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:11:26.690486 25908 consensus_peers.cc:597] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:26.708634 25908 consensus_peers.cc:597] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:26.731667 25908 consensus_peers.cc:597] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:26.772500   852 raft_consensus.cc:493] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:26.773064   852 raft_consensus.cc:515] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:26.775849   852 leader_election.cc:290] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:26.778580   852 raft_consensus.cc:493] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:26.779042   852 raft_consensus.cc:515] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:11:26.779937 25909 leader_election.cc:336] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:26.782459 25911 leader_election.cc:336] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Network error: Client connection negotiation failed: client connection to 127.23.140.67:34661: connect: Connection refused (error 111)
I20260709 13:11:26.782860 25911 leader_election.cc:304] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:26.783488   854 raft_consensus.cc:2749] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:11:26.785270 25908 leader_election.cc:336] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:26.785688   852 leader_election.cc:290] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:11:26.790426 25909 leader_election.cc:336] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:26.790860 25911 leader_election.cc:336] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Network error: Client connection negotiation failed: client connection to 127.23.140.67:34661: connect: Connection refused (error 111)
I20260709 13:11:26.791280 25911 leader_election.cc:304] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:26.791862   852 raft_consensus.cc:2749] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:11:26.807631   852 raft_consensus.cc:493] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:26.808135   852 raft_consensus.cc:515] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:26.812565   852 leader_election.cc:290] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:11:26.813947 25909 leader_election.cc:336] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:26.814692 25911 leader_election.cc:336] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Network error: Client connection negotiation failed: client connection to 127.23.140.67:34661: connect: Connection refused (error 111)
I20260709 13:11:26.815032 25911 leader_election.cc:304] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:26.815591   854 raft_consensus.cc:2749] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:11:26.819075   854 raft_consensus.cc:493] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:26.819080   852 raft_consensus.cc:493] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:26.819707   854 raft_consensus.cc:515] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:26.819828   852 raft_consensus.cc:515] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:26.821825   854 leader_election.cc:290] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:26.823733   852 leader_election.cc:290] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:11:26.827816 25911 leader_election.cc:336] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Network error: Client connection negotiation failed: client connection to 127.23.140.67:34661: connect: Connection refused (error 111)
W20260709 13:11:26.830996 25908 consensus_peers.cc:597] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:11:26.831718 25911 leader_election.cc:336] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Network error: Client connection negotiation failed: client connection to 127.23.140.67:34661: connect: Connection refused (error 111)
W20260709 13:11:26.832226 25909 leader_election.cc:336] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:26.832603 25909 leader_election.cc:304] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:26.833218   852 raft_consensus.cc:2749] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:11:26.833364 25909 leader_election.cc:336] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:26.833827 25909 leader_election.cc:304] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:26.834460   852 raft_consensus.cc:2749] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:11:26.881364   852 raft_consensus.cc:493] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:26.881909   852 raft_consensus.cc:515] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
W20260709 13:11:26.883651 25908 consensus_peers.cc:597] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:26.883981   852 leader_election.cc:290] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
I20260709 13:11:26.888538 24113 tablet_server.cc:196] TabletServer@127.23.140.67:0 shutdown complete.
W20260709 13:11:26.889819 25911 leader_election.cc:336] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Network error: Client connection negotiation failed: client connection to 127.23.140.67:34661: connect: Connection refused (error 111)
W20260709 13:11:26.891441 25908 leader_election.cc:336] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:26.891893 25908 leader_election.cc:304] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:26.892603   852 raft_consensus.cc:2749] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:11:26.895427   852 raft_consensus.cc:493] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:26.895948   852 raft_consensus.cc:515] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:26.898108   852 leader_election.cc:290] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:11:26.902963 25909 leader_election.cc:336] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
W20260709 13:11:26.904110 25911 leader_election.cc:336] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Network error: Client connection negotiation failed: client connection to 127.23.140.67:34661: connect: Connection refused (error 111)
I20260709 13:11:26.904556 25911 leader_election.cc:304] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 46de6919f85341c6a9b574dff112e35a, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:26.906229   854 raft_consensus.cc:2749] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:11:26.910457 25909 consensus_peers.cc:597] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 13:11:26.930408   854 raft_consensus.cc:493] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a2a77ecf2c0941618b5bc79d4c04db61)
I20260709 13:11:26.930994   854 raft_consensus.cc:515] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "a2a77ecf2c0941618b5bc79d4c04db61" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 34661 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:26.933332   854 leader_election.cc:290] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661)
W20260709 13:11:26.939049 25908 leader_election.cc:336] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
W20260709 13:11:26.940132 25911 leader_election.cc:336] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer a2a77ecf2c0941618b5bc79d4c04db61 (127.23.140.67:34661): Network error: Client connection negotiation failed: client connection to 127.23.140.67:34661: connect: Connection refused (error 111)
I20260709 13:11:26.940550 25911 leader_election.cc:304] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, a2a77ecf2c0941618b5bc79d4c04db61
I20260709 13:11:26.941327   854 raft_consensus.cc:2749] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:11:26.953759 25909 consensus_peers.cc:597] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 -> Peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Couldn't send request to peer 46de6919f85341c6a9b574dff112e35a. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:11:27.016899 24113 tablet_server.cc:179] TabletServer@127.23.140.68:0 shutting down...
W20260709 13:11:27.064415 25908 consensus_peers.cc:597] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 -> Peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Couldn't send request to peer 2329e428ebbf484fa3ea9452133df567. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:11:27.073688 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:11:27.074728 24113 tablet_replica.cc:333] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.075750 24113 raft_consensus.cc:2243] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:11:27.076990 24113 raft_consensus.cc:2272] T eda233ace2a24b379fef320a5f610c7a P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.079782 24113 tablet_replica.cc:333] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.080410 24113 raft_consensus.cc:2243] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.081524 24113 raft_consensus.cc:2272] T 37bfef8566b143cf94971f26f417678c P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.083758 24113 tablet_replica.cc:333] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.084381 24113 raft_consensus.cc:2243] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:11:27.085327 24113 raft_consensus.cc:2272] T 94b9d971fbc94215a65a2247f2e2acb1 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.087284 24113 tablet_replica.cc:333] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.087900 24113 raft_consensus.cc:2243] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.088438 24113 raft_consensus.cc:2272] T 8fa7e9050d44475381c97025c2b11711 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.090483 24113 tablet_replica.cc:333] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.091137 24113 raft_consensus.cc:2243] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.091737 24113 raft_consensus.cc:2272] T b62f4877af9742298b1e149789ca3090 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.094225 24113 tablet_replica.cc:333] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.094848 24113 raft_consensus.cc:2243] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:11:27.095973 24113 raft_consensus.cc:2272] T cf0501919f944794bbab0ffae6a334e1 P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.098007 24113 tablet_replica.cc:333] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.098599 24113 raft_consensus.cc:2243] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.099079 24113 raft_consensus.cc:2272] T 5ead2e92f0614db79c9249687c4246f9 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.101080 24113 tablet_replica.cc:333] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.101720 24113 raft_consensus.cc:2243] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:11:27.102540 24113 raft_consensus.cc:2272] T d45c0f8b003848989966aa931c6a7c11 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.104666 24113 tablet_replica.cc:333] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.105329 24113 raft_consensus.cc:2243] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.105842 24113 raft_consensus.cc:2272] T 7555c0d2bb294af19f602573931ceb90 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.107862 24113 tablet_replica.cc:333] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.108475 24113 raft_consensus.cc:2243] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:11:27.109421 24113 raft_consensus.cc:2272] T b2cfa29915b34839a60321576f9a67d6 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.111637 24113 tablet_replica.cc:333] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.112176 24113 raft_consensus.cc:2243] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.112687 24113 raft_consensus.cc:2272] T 6ff9a7b00f104dd294125870b06a7df4 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.114730 24113 tablet_replica.cc:333] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.115259 24113 raft_consensus.cc:2243] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.115878 24113 raft_consensus.cc:2272] T a53b1fb6477d486e98bff9549cc487df P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.117919 24113 tablet_replica.cc:333] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.118506 24113 raft_consensus.cc:2243] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.119005 24113 raft_consensus.cc:2272] T 17584a19690d4010851d65378cfcbafa P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.121083 24113 tablet_replica.cc:333] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.121685 24113 raft_consensus.cc:2243] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:11:27.122545 24113 raft_consensus.cc:2272] T f406a4d41bb9441bba0a83c1d6a56aa3 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.124434 24113 tablet_replica.cc:333] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.124960 24113 raft_consensus.cc:2243] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.125450 24113 raft_consensus.cc:2272] T 56914a99b2d8417592e1f08979cfadf6 P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.127549 24113 tablet_replica.cc:333] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.128149 24113 raft_consensus.cc:2243] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:11:27.129055 24113 raft_consensus.cc:2272] T e81523f4f4344a04a1e0b03c8de99290 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.131170 24113 tablet_replica.cc:333] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.131795 24113 raft_consensus.cc:2243] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:11:27.132721 24113 raft_consensus.cc:2272] T c5481170c5c74798bdfd146068f8b539 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.135046 24113 tablet_replica.cc:333] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.135686 24113 raft_consensus.cc:2243] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.136427 24113 raft_consensus.cc:2272] T 3cff7b4cb7184e4c9a69372d5901e19a P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.136597   854 raft_consensus.cc:493] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 2329e428ebbf484fa3ea9452133df567)
I20260709 13:11:27.137217   854 raft_consensus.cc:515] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2329e428ebbf484fa3ea9452133df567" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42315 } } peers { permanent_uuid: "46de6919f85341c6a9b574dff112e35a" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 45313 } } peers { permanent_uuid: "f646dfd532974dceadfe00005d2bbf59" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 39211 } attrs { promote: false } }
I20260709 13:11:27.139724 24113 tablet_replica.cc:333] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.139617   854 leader_election.cc:290] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315), 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313)
I20260709 13:11:27.140491 24113 raft_consensus.cc:2243] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.141106 24113 raft_consensus.cc:2272] T 2730c370dd9a4a1ea5adfb543e3ba120 P f646dfd532974dceadfe00005d2bbf59 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.143786 24113 tablet_replica.cc:333] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.144520 24113 raft_consensus.cc:2243] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Raft consensus shutting down.
W20260709 13:11:27.145030 25908 leader_election.cc:336] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:27.145606 24113 raft_consensus.cc:2272] T 63c3ef75ef234051b7ae1ea395400f54 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.148360 24113 tablet_replica.cc:333] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.149374 24113 raft_consensus.cc:2243] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.150138 24113 raft_consensus.cc:2272] T de96ce5d28144cf3894e100bfc4959fa P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.153712 24113 tablet_replica.cc:333] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.154438 24113 raft_consensus.cc:2243] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.155140 24113 raft_consensus.cc:2272] T 2dc23779efbf4518a6ada8bde8243a11 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.159600 24113 tablet_replica.cc:333] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.160377 24113 raft_consensus.cc:2243] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 21 LEADER]: Raft consensus shutting down.
W20260709 13:11:27.160382 25909 leader_election.cc:336] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 46de6919f85341c6a9b574dff112e35a (127.23.140.65:45313): Network error: Client connection negotiation failed: client connection to 127.23.140.65:45313: connect: Connection refused (error 111)
I20260709 13:11:27.161000 25909 leader_election.cc:304] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f646dfd532974dceadfe00005d2bbf59; no voters: 2329e428ebbf484fa3ea9452133df567, 46de6919f85341c6a9b574dff112e35a
I20260709 13:11:27.161657 24113 raft_consensus.cc:2272] T 4e29e9ab7eda4242964fa76e29c2a221 P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.162258   852 raft_consensus.cc:2749] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:11:27.164207 24113 tablet_replica.cc:333] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.164810 24113 raft_consensus.cc:2243] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:11:27.165673 24113 raft_consensus.cc:2272] T 8fd399e2f6384e98ada57177a6010d14 P f646dfd532974dceadfe00005d2bbf59 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.167709 24113 tablet_replica.cc:333] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.168211 24113 raft_consensus.cc:2243] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.168730 24113 raft_consensus.cc:2272] T 8f209a6539f54de28906492f142efb6b P f646dfd532974dceadfe00005d2bbf59 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.171140 24113 tablet_replica.cc:333] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.171754 24113 raft_consensus.cc:2243] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.172315 24113 raft_consensus.cc:2272] T b6d19448306e4052a13c08a582f4ccfa P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.174283 24113 tablet_replica.cc:333] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.174798 24113 raft_consensus.cc:2243] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 15 LEADER]: Raft consensus shutting down.
I20260709 13:11:27.175596 24113 raft_consensus.cc:2272] T 9b5c9f65605b4dfca1bbfe0f09629c6d P f646dfd532974dceadfe00005d2bbf59 [term 15 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.177592 24113 tablet_replica.cc:333] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.178102 24113 raft_consensus.cc:2243] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.178661 24113 raft_consensus.cc:2272] T d4d43163cfdd4a4094e92fe68bc9259a P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.180673 24113 tablet_replica.cc:333] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.181238 24113 raft_consensus.cc:2243] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.181710 24113 raft_consensus.cc:2272] T 46ab4fd91d3f4766a7b97848f5244360 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.183583 24113 tablet_replica.cc:333] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.184064 24113 raft_consensus.cc:2243] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:11:27.184888 24113 raft_consensus.cc:2272] T 69fee8cad2c84e1492403c23786622a1 P f646dfd532974dceadfe00005d2bbf59 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.186628 24113 tablet_replica.cc:333] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.187115 24113 raft_consensus.cc:2243] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:11:27.188022 24113 raft_consensus.cc:2272] T 898e00cfe03c4fc68dc9d6e92a55e996 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.189787 24113 tablet_replica.cc:333] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.190277 24113 raft_consensus.cc:2243] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.190795 24113 raft_consensus.cc:2272] T f75b60146e7d4b9b9954b4fb2c414679 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.192835 24113 tablet_replica.cc:333] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.193387 24113 raft_consensus.cc:2243] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.193873 24113 raft_consensus.cc:2272] T 993e3b05a73a4403b04c3b55a5b48abe P f646dfd532974dceadfe00005d2bbf59 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.195773 24113 tablet_replica.cc:333] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.196235 24113 raft_consensus.cc:2243] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.196712 24113 raft_consensus.cc:2272] T 17ef4786898b4ad696baed95b4209788 P f646dfd532974dceadfe00005d2bbf59 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.198557 24113 tablet_replica.cc:333] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.199038 24113 raft_consensus.cc:2243] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.199748 24113 raft_consensus.cc:2272] T 991bcf9b91a14242837dfb4e18891e91 P f646dfd532974dceadfe00005d2bbf59 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.201637 24113 tablet_replica.cc:333] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59: stopping tablet replica
I20260709 13:11:27.202059 24113 raft_consensus.cc:2243] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:11:27.202473 24113 raft_consensus.cc:2272] T 98122796f52e40968d1d8ccd7b032b32 P f646dfd532974dceadfe00005d2bbf59 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:27.555697 24113 tablet_server.cc:196] TabletServer@127.23.140.68:0 shutdown complete.
I20260709 13:11:27.621877 24113 master.cc:562] Master@127.23.140.126:33305 shutting down...
W20260709 13:11:27.665247 25192 auto_rebalancer.cc:873] Removing replace marker failed: Catalog manager is not initialized. State: Closing
W20260709 13:11:27.665575 25192 auto_rebalancer.cc:901] Could not move replica: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 13:11:27.665793 25192 auto_rebalancer.cc:330] scheduled replica move failed to complete: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 13:11:47.932350 25128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111) [suppressed 352 similar messages]
W20260709 13:11:47.936389 25128 catalog_manager.cc:4755] TS 2329e428ebbf484fa3ea9452133df567 (127.23.140.66:42315): ChangeConfig:REMOVE_PEER RPC failed for tablet f75b60146e7d4b9b9954b4fb2c414679: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42315: connect: Connection refused (error 111)
I20260709 13:11:48.917364 24113 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:11:48.918422 24113 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:11:48.918834 24113 tablet_replica.cc:333] T 00000000000000000000000000000000 P 4d591857b6ac45ad9cab82445b07ae03: stopping tablet replica
I20260709 13:11:49.142899 24113 master.cc:584] Master@127.23.140.126:33305 shutdown complete.
I20260709 13:11:49.194621 24113 test_util.cc:182] -----------------------------------------------
I20260709 13:11:49.194873 24113 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602412999163-24113-0
[  FAILED  ] LeaderRebalancerTest.AddTserver (271062 ms)
[ RUN      ] LeaderRebalancerTest.RestartTserver
I20260709 13:11:49.198500 24113 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.140.126:33289
I20260709 13:11:49.199695 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:11:49.205521   861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:11:49.205598   860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:11:49.206100   863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:11:49.208364 24113 server_base.cc:1061] running on GCE node
I20260709 13:11:49.211889 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:11:49.212107 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:11:49.212321 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602709212284 us; error 0 us; skew 500 ppm
I20260709 13:11:49.212867 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:11:49.215376 24113 webserver.cc:533] Webserver started at http://127.23.140.126:46747/ using document root <none> and password file <none>
I20260709 13:11:49.215874 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:11:49.216081 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:11:49.216372 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:11:49.217767 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/master-0-root/instance:
uuid: "154f6da223eb4abba85302feda9ead4f"
format_stamp: "Formatted at 2026-07-09 13:11:49 on dist-test-slave-k73n"
I20260709 13:11:49.222259 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 13:11:49.225652   868 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:11:49.226390 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 13:11:49.226809 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/master-0-root
uuid: "154f6da223eb4abba85302feda9ead4f"
format_stamp: "Formatted at 2026-07-09 13:11:49 on dist-test-slave-k73n"
I20260709 13:11:49.227229 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:11:49.238739 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:11:49.239969 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:11:49.278807 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.126:33289
I20260709 13:11:49.278911   919 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.126:33289 every 8 connection(s)
I20260709 13:11:49.283278   920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:49.294800   920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f: Bootstrap starting.
I20260709 13:11:49.299841   920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:49.304047   920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f: No bootstrap required, opened a new log
I20260709 13:11:49.306321   920 raft_consensus.cc:359] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154f6da223eb4abba85302feda9ead4f" member_type: VOTER }
I20260709 13:11:49.306747   920 raft_consensus.cc:385] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:49.306928   920 raft_consensus.cc:740] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 154f6da223eb4abba85302feda9ead4f, State: Initialized, Role: FOLLOWER
I20260709 13:11:49.307433   920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154f6da223eb4abba85302feda9ead4f" member_type: VOTER }
I20260709 13:11:49.307842   920 raft_consensus.cc:399] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:11:49.308045   920 raft_consensus.cc:493] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:11:49.308256   920 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:49.312958   920 raft_consensus.cc:515] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154f6da223eb4abba85302feda9ead4f" member_type: VOTER }
I20260709 13:11:49.313468   920 leader_election.cc:304] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 154f6da223eb4abba85302feda9ead4f; no voters: 
I20260709 13:11:49.314540   920 leader_election.cc:290] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:11:49.314895   923 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:49.316355   923 raft_consensus.cc:697] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [term 1 LEADER]: Becoming Leader. State: Replica: 154f6da223eb4abba85302feda9ead4f, State: Running, Role: LEADER
I20260709 13:11:49.317027   923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154f6da223eb4abba85302feda9ead4f" member_type: VOTER }
I20260709 13:11:49.317670   920 sys_catalog.cc:565] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:11:49.324026   925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 154f6da223eb4abba85302feda9ead4f. Latest consensus state: current_term: 1 leader_uuid: "154f6da223eb4abba85302feda9ead4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154f6da223eb4abba85302feda9ead4f" member_type: VOTER } }
I20260709 13:11:49.324622   924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "154f6da223eb4abba85302feda9ead4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154f6da223eb4abba85302feda9ead4f" member_type: VOTER } }
I20260709 13:11:49.324947   925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [sys.catalog]: This master's current role is: LEADER
I20260709 13:11:49.325212   924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [sys.catalog]: This master's current role is: LEADER
I20260709 13:11:49.326938   933 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:11:49.333312   933 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:11:49.334342 24113 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:11:49.342280   933 catalog_manager.cc:1383] Generated new cluster ID: 88764db8b2f24031a9a19b9d28cfa7dc
I20260709 13:11:49.342511   933 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:11:49.360741   933 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:11:49.362219   933 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:11:49.376793   933 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f: Generated new TSK 0
I20260709 13:11:49.377398   933 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:11:49.401556 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:11:49.408376   941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:11:49.409557   942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:11:49.411746   944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:11:49.411904 24113 server_base.cc:1061] running on GCE node
I20260709 13:11:49.414481 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:11:49.414685 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:11:49.414901 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602709414864 us; error 0 us; skew 500 ppm
I20260709 13:11:49.415493 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:11:49.417951 24113 webserver.cc:533] Webserver started at http://127.23.140.65:40743/ using document root <none> and password file <none>
I20260709 13:11:49.418435 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:11:49.418628 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:11:49.418975 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:11:49.420171 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-0-root/instance:
uuid: "294145900dd3496cb4a1d722613e1558"
format_stamp: "Formatted at 2026-07-09 13:11:49 on dist-test-slave-k73n"
I20260709 13:11:49.424839 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.004s
I20260709 13:11:49.428429   949 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:11:49.429370 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 13:11:49.429711 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-0-root
uuid: "294145900dd3496cb4a1d722613e1558"
format_stamp: "Formatted at 2026-07-09 13:11:49 on dist-test-slave-k73n"
I20260709 13:11:49.430034 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:11:49.462234 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:11:49.463505 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:11:49.465117 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:11:49.467573 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:11:49.467820 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:11:49.468079 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:11:49.468272 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:11:49.511170 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.65:43721
I20260709 13:11:49.511277  1011 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.65:43721 every 8 connection(s)
I20260709 13:11:49.516520 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:11:49.526665  1016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:11:49.529103  1017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:11:49.533823  1012 heartbeater.cc:344] Connected to a master server at 127.23.140.126:33289
I20260709 13:11:49.534255  1012 heartbeater.cc:461] Registering TS with master...
I20260709 13:11:49.535025  1012 heartbeater.cc:507] Master 127.23.140.126:33289 requested a full tablet report, sending...
W20260709 13:11:49.535356  1019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:11:49.536074 24113 server_base.cc:1061] running on GCE node
I20260709 13:11:49.537511   885 ts_manager.cc:194] Registered new tserver with Master: 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:49.539541 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:11:49.539875 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:11:49.540078 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602709540062 us; error 0 us; skew 500 ppm
I20260709 13:11:49.540587   885 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37858
I20260709 13:11:49.541189 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:11:49.545388 24113 webserver.cc:533] Webserver started at http://127.23.140.66:38067/ using document root <none> and password file <none>
I20260709 13:11:49.546275 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:11:49.546545 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:11:49.546900 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:11:49.548954 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-1-root/instance:
uuid: "9c45b6d917734835a8a37b0f4c789ae6"
format_stamp: "Formatted at 2026-07-09 13:11:49 on dist-test-slave-k73n"
I20260709 13:11:49.555910 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:11:49.560755  1024 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:11:49.561825 24113 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 13:11:49.562204 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-1-root
uuid: "9c45b6d917734835a8a37b0f4c789ae6"
format_stamp: "Formatted at 2026-07-09 13:11:49 on dist-test-slave-k73n"
I20260709 13:11:49.562520 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:11:49.586212 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:11:49.588102 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:11:49.590363 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:11:49.593883 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:11:49.594074 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:11:49.594254 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:11:49.594376 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:11:49.637687 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.66:44097
I20260709 13:11:49.637791  1086 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.66:44097 every 8 connection(s)
I20260709 13:11:49.642618 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:11:49.650375  1090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:11:49.652626  1091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:11:49.654760  1093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:11:49.654814 24113 server_base.cc:1061] running on GCE node
I20260709 13:11:49.656270 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 13:11:49.656437  1087 heartbeater.cc:344] Connected to a master server at 127.23.140.126:33289
W20260709 13:11:49.656594 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:11:49.656890 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602709656875 us; error 0 us; skew 500 ppm
I20260709 13:11:49.656970  1087 heartbeater.cc:461] Registering TS with master...
I20260709 13:11:49.657582 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:11:49.657910  1087 heartbeater.cc:507] Master 127.23.140.126:33289 requested a full tablet report, sending...
I20260709 13:11:49.660344   885 ts_manager.cc:194] Registered new tserver with Master: 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:49.660395 24113 webserver.cc:533] Webserver started at http://127.23.140.67:42687/ using document root <none> and password file <none>
I20260709 13:11:49.661013 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:11:49.661250 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:11:49.661530 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:11:49.661963   885 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37862
I20260709 13:11:49.662741 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-2-root/instance:
uuid: "f9d473477839468bbdf000a21d5f9485"
format_stamp: "Formatted at 2026-07-09 13:11:49 on dist-test-slave-k73n"
I20260709 13:11:49.667820 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 13:11:49.671473  1098 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:11:49.672323 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 13:11:49.672662 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-2-root
uuid: "f9d473477839468bbdf000a21d5f9485"
format_stamp: "Formatted at 2026-07-09 13:11:49 on dist-test-slave-k73n"
I20260709 13:11:49.672916 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:11:49.686321 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:11:49.687606 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:11:49.689210 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:11:49.691462 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:11:49.691697 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:11:49.691941 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:11:49.692122 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:11:49.734720 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.67:33869
I20260709 13:11:49.734839  1160 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.67:33869 every 8 connection(s)
I20260709 13:11:49.739763 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:11:49.749172  1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:11:49.750465  1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:11:49.753759  1161 heartbeater.cc:344] Connected to a master server at 127.23.140.126:33289
W20260709 13:11:49.754069  1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:11:49.754233 24113 server_base.cc:1061] running on GCE node
I20260709 13:11:49.754254  1161 heartbeater.cc:461] Registering TS with master...
I20260709 13:11:49.755354 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 13:11:49.755427  1161 heartbeater.cc:507] Master 127.23.140.126:33289 requested a full tablet report, sending...
W20260709 13:11:49.755605 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:11:49.755928 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602709755910 us; error 0 us; skew 500 ppm
I20260709 13:11:49.756716 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:11:49.757889   885 ts_manager.cc:194] Registered new tserver with Master: f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:49.759539 24113 webserver.cc:533] Webserver started at http://127.23.140.68:39417/ using document root <none> and password file <none>
I20260709 13:11:49.759792   885 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37874
I20260709 13:11:49.760215 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:11:49.760510 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:11:49.760830 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:11:49.762420 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-3-root/instance:
uuid: "862b41697c74421fbac82c10c245434d"
format_stamp: "Formatted at 2026-07-09 13:11:49 on dist-test-slave-k73n"
I20260709 13:11:49.767011 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 13:11:49.770267  1172 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:11:49.771051 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.001s
I20260709 13:11:49.771324 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-3-root
uuid: "862b41697c74421fbac82c10c245434d"
format_stamp: "Formatted at 2026-07-09 13:11:49 on dist-test-slave-k73n"
I20260709 13:11:49.771592 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:11:49.786811 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:11:49.788031 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:11:49.789532 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:11:49.791754 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:11:49.791976 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:11:49.792238 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:11:49.792397 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:11:49.836465 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.68:34477
I20260709 13:11:49.836580  1234 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.68:34477 every 8 connection(s)
I20260709 13:11:49.850274  1235 heartbeater.cc:344] Connected to a master server at 127.23.140.126:33289
I20260709 13:11:49.850741  1235 heartbeater.cc:461] Registering TS with master...
I20260709 13:11:49.851523  1235 heartbeater.cc:507] Master 127.23.140.126:33289 requested a full tablet report, sending...
I20260709 13:11:49.853440   885 ts_manager.cc:194] Registered new tserver with Master: 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:49.854439 24113 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.014649066s
I20260709 13:11:49.854842   885 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37882
I20260709 13:11:49.882247   885 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37892:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20260709 13:11:50.126266  1126 tablet_service.cc:1511] Processing CreateTablet for tablet 6de2d5263c9a4dfebfe53a9eb7bc0bba (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:11:50.127887  1126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6de2d5263c9a4dfebfe53a9eb7bc0bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.134177  1125 tablet_service.cc:1511] Processing CreateTablet for tablet 39dcc0888dc64bff9ce20b35e4bc8f89 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:11:50.135725  1125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39dcc0888dc64bff9ce20b35e4bc8f89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.137429   977 tablet_service.cc:1511] Processing CreateTablet for tablet 6de2d5263c9a4dfebfe53a9eb7bc0bba (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:11:50.139039   977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6de2d5263c9a4dfebfe53a9eb7bc0bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.142413   974 tablet_service.cc:1511] Processing CreateTablet for tablet 161b9f09b3fb47febfabacf867781986 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:11:50.142997   971 tablet_service.cc:1511] Processing CreateTablet for tablet 3b917eb6128a4341b3e3288130a65db1 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:11:50.143877   974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 161b9f09b3fb47febfabacf867781986. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.144634   971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b917eb6128a4341b3e3288130a65db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.146119   973 tablet_service.cc:1511] Processing CreateTablet for tablet c2312d768cbf4b3495d5b1574b0b6eb5 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:11:50.147472   973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2312d768cbf4b3495d5b1574b0b6eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.149395  1052 tablet_service.cc:1511] Processing CreateTablet for tablet 6de2d5263c9a4dfebfe53a9eb7bc0bba (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:11:50.150386   972 tablet_service.cc:1511] Processing CreateTablet for tablet 219d22690fa145adb5b04192c1fc16bd (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:11:50.150859  1052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6de2d5263c9a4dfebfe53a9eb7bc0bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.150540  1050 tablet_service.cc:1511] Processing CreateTablet for tablet 240eb8c24e0142c8a23a0afdd5dccda6 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:11:50.151814   972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 219d22690fa145adb5b04192c1fc16bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.151929  1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 240eb8c24e0142c8a23a0afdd5dccda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.153153  1051 tablet_service.cc:1511] Processing CreateTablet for tablet 8f1971394c7d40bfa460d909b6f47438 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:11:50.153978  1049 tablet_service.cc:1511] Processing CreateTablet for tablet 161b9f09b3fb47febfabacf867781986 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:11:50.154593   975 tablet_service.cc:1511] Processing CreateTablet for tablet 39dcc0888dc64bff9ce20b35e4bc8f89 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:11:50.156010   975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39dcc0888dc64bff9ce20b35e4bc8f89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.156451  1048 tablet_service.cc:1511] Processing CreateTablet for tablet c2312d768cbf4b3495d5b1574b0b6eb5 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:11:50.158349   976 tablet_service.cc:1511] Processing CreateTablet for tablet 8f1971394c7d40bfa460d909b6f47438 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:11:50.158874  1047 tablet_service.cc:1511] Processing CreateTablet for tablet f0afcfb9359a421b85fabc92b7733d9b (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:11:50.159763   976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1971394c7d40bfa460d909b6f47438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.163211  1124 tablet_service.cc:1511] Processing CreateTablet for tablet 240eb8c24e0142c8a23a0afdd5dccda6 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:11:50.164245  1123 tablet_service.cc:1511] Processing CreateTablet for tablet 161b9f09b3fb47febfabacf867781986 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:11:50.164729  1124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 240eb8c24e0142c8a23a0afdd5dccda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.164470  1046 tablet_service.cc:1511] Processing CreateTablet for tablet 3b917eb6128a4341b3e3288130a65db1 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:11:50.154620  1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1971394c7d40bfa460d909b6f47438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.166059  1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 161b9f09b3fb47febfabacf867781986. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.166385  1122 tablet_service.cc:1511] Processing CreateTablet for tablet 219d22690fa145adb5b04192c1fc16bd (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:11:50.168814  1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b917eb6128a4341b3e3288130a65db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.168594  1121 tablet_service.cc:1511] Processing CreateTablet for tablet f0afcfb9359a421b85fabc92b7733d9b (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:11:50.170491  1123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 161b9f09b3fb47febfabacf867781986. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.173385  1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0afcfb9359a421b85fabc92b7733d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.175817  1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2312d768cbf4b3495d5b1574b0b6eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.179049  1121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0afcfb9359a421b85fabc92b7733d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.190609  1122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 219d22690fa145adb5b04192c1fc16bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.192301  1120 tablet_service.cc:1511] Processing CreateTablet for tablet 3b917eb6128a4341b3e3288130a65db1 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:11:50.194669  1120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b917eb6128a4341b3e3288130a65db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.194391   970 tablet_service.cc:1511] Processing CreateTablet for tablet 35cd7c6206834f5dabfca17b22a0605a (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:11:50.195967   970 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35cd7c6206834f5dabfca17b22a0605a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.206382  1200 tablet_service.cc:1511] Processing CreateTablet for tablet 8f1971394c7d40bfa460d909b6f47438 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:11:50.207998  1200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1971394c7d40bfa460d909b6f47438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.209966   969 tablet_service.cc:1511] Processing CreateTablet for tablet 69355f8224b14b23be2cf00be247206c (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:11:50.210363  1199 tablet_service.cc:1511] Processing CreateTablet for tablet 39dcc0888dc64bff9ce20b35e4bc8f89 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:11:50.211560   969 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69355f8224b14b23be2cf00be247206c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.211688  1199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39dcc0888dc64bff9ce20b35e4bc8f89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.214499  1198 tablet_service.cc:1511] Processing CreateTablet for tablet 240eb8c24e0142c8a23a0afdd5dccda6 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:11:50.216050  1198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 240eb8c24e0142c8a23a0afdd5dccda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.219394  1197 tablet_service.cc:1511] Processing CreateTablet for tablet c2312d768cbf4b3495d5b1574b0b6eb5 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:11:50.219523  1119 tablet_service.cc:1511] Processing CreateTablet for tablet 69355f8224b14b23be2cf00be247206c (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:11:50.220853  1197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2312d768cbf4b3495d5b1574b0b6eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.220930  1119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69355f8224b14b23be2cf00be247206c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.222327  1196 tablet_service.cc:1511] Processing CreateTablet for tablet 219d22690fa145adb5b04192c1fc16bd (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:11:50.233637  1196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 219d22690fa145adb5b04192c1fc16bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.233227  1193 tablet_service.cc:1511] Processing CreateTablet for tablet 69355f8224b14b23be2cf00be247206c (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:11:50.224161  1195 tablet_service.cc:1511] Processing CreateTablet for tablet f0afcfb9359a421b85fabc92b7733d9b (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:11:50.234884  1193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69355f8224b14b23be2cf00be247206c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.226142  1194 tablet_service.cc:1511] Processing CreateTablet for tablet 35cd7c6206834f5dabfca17b22a0605a (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:11:50.236030  1194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35cd7c6206834f5dabfca17b22a0605a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.237334  1195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0afcfb9359a421b85fabc92b7733d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.283856  1118 tablet_service.cc:1511] Processing CreateTablet for tablet f95cbeb7360c4c61b442e75e35d93e73 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:11:50.285420  1118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f95cbeb7360c4c61b442e75e35d93e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.293437  1045 tablet_service.cc:1511] Processing CreateTablet for tablet 35cd7c6206834f5dabfca17b22a0605a (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:11:50.294950  1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35cd7c6206834f5dabfca17b22a0605a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.326297  1255 tablet_bootstrap.cc:492] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:50.326464  1254 tablet_bootstrap.cc:492] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
W20260709 13:11:50.331986   932 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:11:50.337064  1192 tablet_service.cc:1511] Processing CreateTablet for tablet f95cbeb7360c4c61b442e75e35d93e73 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:11:50.338533  1192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f95cbeb7360c4c61b442e75e35d93e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.347975  1191 tablet_service.cc:1511] Processing CreateTablet for tablet 11cd8a46e76f4bc685aa01885b27cc57 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:11:50.349485  1191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11cd8a46e76f4bc685aa01885b27cc57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.352219  1254 tablet_bootstrap.cc:654] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:50.358124  1122 tablet_service.cc:1511] Processing CreateTablet for tablet fa73a73b72b2499fa5219cbf455099b4 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:11:50.367353  1122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa73a73b72b2499fa5219cbf455099b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.365908   970 tablet_service.cc:1511] Processing CreateTablet for tablet fa73a73b72b2499fa5219cbf455099b4 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:11:50.377586  1044 tablet_service.cc:1511] Processing CreateTablet for tablet f95cbeb7360c4c61b442e75e35d93e73 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:11:50.379012  1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f95cbeb7360c4c61b442e75e35d93e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.386519  1256 tablet_bootstrap.cc:492] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:50.388401   970 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa73a73b72b2499fa5219cbf455099b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.413115  1045 tablet_service.cc:1511] Processing CreateTablet for tablet fa73a73b72b2499fa5219cbf455099b4 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:11:50.414561  1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa73a73b72b2499fa5219cbf455099b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.418164   976 tablet_service.cc:1511] Processing CreateTablet for tablet 11cd8a46e76f4bc685aa01885b27cc57 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:11:50.418852  1191 tablet_service.cc:1511] Processing CreateTablet for tablet 29dde48ef51c4666a28b97181ec2ee21 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:11:50.419679   976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11cd8a46e76f4bc685aa01885b27cc57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.420146  1191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29dde48ef51c4666a28b97181ec2ee21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.426216  1119 tablet_service.cc:1511] Processing CreateTablet for tablet 29dde48ef51c4666a28b97181ec2ee21 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:11:50.427744  1119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29dde48ef51c4666a28b97181ec2ee21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.426306  1255 tablet_bootstrap.cc:654] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:50.448073  1256 tablet_bootstrap.cc:654] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:50.459077  1257 tablet_bootstrap.cc:492] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:50.462739  1255 tablet_bootstrap.cc:492] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:50.463277  1255 ts_tablet_manager.cc:1403] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.137s	user 0.006s	sys 0.008s
I20260709 13:11:50.466907   969 tablet_service.cc:1511] Processing CreateTablet for tablet 29dde48ef51c4666a28b97181ec2ee21 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:11:50.468348   969 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29dde48ef51c4666a28b97181ec2ee21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.468010  1255 raft_consensus.cc:359] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:50.468796  1255 raft_consensus.cc:385] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:50.469154  1255 raft_consensus.cc:740] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:50.473261  1118 tablet_service.cc:1511] Processing CreateTablet for tablet a81738ce3431443d9e332eb931ec3c0b (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:11:50.474709  1118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a81738ce3431443d9e332eb931ec3c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.481408  1050 tablet_service.cc:1511] Processing CreateTablet for tablet 11cd8a46e76f4bc685aa01885b27cc57 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:11:50.481429  1255 consensus_queue.cc:260] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:50.483052  1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11cd8a46e76f4bc685aa01885b27cc57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.484190  1124 tablet_service.cc:1511] Processing CreateTablet for tablet 4c162b2e111b472b88cd037925284de7 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:11:50.485081  1195 tablet_service.cc:1511] Processing CreateTablet for tablet a81738ce3431443d9e332eb931ec3c0b (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:11:50.486110  1049 tablet_service.cc:1511] Processing CreateTablet for tablet 4c162b2e111b472b88cd037925284de7 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:11:50.486937  1195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a81738ce3431443d9e332eb931ec3c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.487399  1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c162b2e111b472b88cd037925284de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.488468  1047 tablet_service.cc:1511] Processing CreateTablet for tablet a81738ce3431443d9e332eb931ec3c0b (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:11:50.489379  1124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c162b2e111b472b88cd037925284de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.490259  1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a81738ce3431443d9e332eb931ec3c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.500430  1254 tablet_bootstrap.cc:492] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:50.501010  1254 ts_tablet_manager.cc:1403] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.175s	user 0.027s	sys 0.006s
I20260709 13:11:50.502223   975 tablet_service.cc:1511] Processing CreateTablet for tablet 4c162b2e111b472b88cd037925284de7 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:11:50.521023   975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c162b2e111b472b88cd037925284de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.509935  1257 tablet_bootstrap.cc:654] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:50.532984   973 tablet_service.cc:1511] Processing CreateTablet for tablet cae53ca608e74146a2f89675ddca72a6 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:11:50.534556   973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cae53ca608e74146a2f89675ddca72a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.541895  1012 heartbeater.cc:499] Master 127.23.140.126:33289 was elected leader, sending a full tablet report...
I20260709 13:11:50.542433  1193 tablet_service.cc:1511] Processing CreateTablet for tablet cae53ca608e74146a2f89675ddca72a6 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:11:50.543746  1193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cae53ca608e74146a2f89675ddca72a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.539376  1254 raft_consensus.cc:359] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:50.547399  1254 raft_consensus.cc:385] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:50.554456  1254 raft_consensus.cc:740] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:50.555519  1254 consensus_queue.cc:260] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:50.556205  1256 tablet_bootstrap.cc:492] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:50.557281  1256 ts_tablet_manager.cc:1403] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.171s	user 0.007s	sys 0.008s
I20260709 13:11:50.561833  1256 raft_consensus.cc:359] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:50.562547  1256 raft_consensus.cc:385] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:50.562847  1256 raft_consensus.cc:740] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:50.563701  1256 consensus_queue.cc:260] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:50.564122  1255 ts_tablet_manager.cc:1434] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.101s	user 0.028s	sys 0.028s
I20260709 13:11:50.543888  1052 tablet_service.cc:1511] Processing CreateTablet for tablet cae53ca608e74146a2f89675ddca72a6 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:11:50.566428  1255 tablet_bootstrap.cc:492] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:50.566563  1052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cae53ca608e74146a2f89675ddca72a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.567193  1200 tablet_service.cc:1511] Processing CreateTablet for tablet c216d860ee834d4b8f77c8c39f0dba94 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:11:50.568481  1200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c216d860ee834d4b8f77c8c39f0dba94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.575194  1118 tablet_service.cc:1511] Processing CreateTablet for tablet c216d860ee834d4b8f77c8c39f0dba94 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:11:50.577004  1118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c216d860ee834d4b8f77c8c39f0dba94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.592006  1044 tablet_service.cc:1511] Processing CreateTablet for tablet 120ba483d3f1448e967c504d55e6d5f3 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:11:50.596618   969 tablet_service.cc:1511] Processing CreateTablet for tablet c216d860ee834d4b8f77c8c39f0dba94 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:11:50.598421  1257 tablet_bootstrap.cc:492] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:50.598543   969 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c216d860ee834d4b8f77c8c39f0dba94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.598960  1257 ts_tablet_manager.cc:1403] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.140s	user 0.012s	sys 0.004s
I20260709 13:11:50.601835  1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 120ba483d3f1448e967c504d55e6d5f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.607735  1124 tablet_service.cc:1511] Processing CreateTablet for tablet 120ba483d3f1448e967c504d55e6d5f3 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:11:50.609158  1124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 120ba483d3f1448e967c504d55e6d5f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.610203  1193 tablet_service.cc:1511] Processing CreateTablet for tablet 120ba483d3f1448e967c504d55e6d5f3 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:11:50.612496  1260 raft_consensus.cc:493] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:50.613235  1260 raft_consensus.cc:515] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:50.614384  1193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 120ba483d3f1448e967c504d55e6d5f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.614200   976 tablet_service.cc:1511] Processing CreateTablet for tablet 33ceafb69dc74411898ea99b377d1032 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:11:50.615578   976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33ceafb69dc74411898ea99b377d1032. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.616724  1260 leader_election.cc:290] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:50.628661  1257 raft_consensus.cc:359] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:50.629626  1257 raft_consensus.cc:385] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:50.629967  1257 raft_consensus.cc:740] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:50.630774  1257 consensus_queue.cc:260] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:50.632205  1119 tablet_service.cc:1511] Processing CreateTablet for tablet 33ceafb69dc74411898ea99b377d1032 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:11:50.633723  1119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33ceafb69dc74411898ea99b377d1032. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.644567  1198 tablet_service.cc:1511] Processing CreateTablet for tablet 7526fd343f7d40528597242eb03386f8 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:11:50.655017  1049 tablet_service.cc:1511] Processing CreateTablet for tablet 33ceafb69dc74411898ea99b377d1032 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:11:50.655321   970 tablet_service.cc:1511] Processing CreateTablet for tablet 7526fd343f7d40528597242eb03386f8 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:11:50.656334  1198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7526fd343f7d40528597242eb03386f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.656741   970 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7526fd343f7d40528597242eb03386f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.661083  1045 tablet_service.cc:1511] Processing CreateTablet for tablet 7526fd343f7d40528597242eb03386f8 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:11:50.664968  1087 heartbeater.cc:499] Master 127.23.140.126:33289 was elected leader, sending a full tablet report...
I20260709 13:11:50.656842  1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33ceafb69dc74411898ea99b377d1032. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.694052  1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7526fd343f7d40528597242eb03386f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.680960  1255 tablet_bootstrap.cc:654] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:50.703075  1235 heartbeater.cc:499] Master 127.23.140.126:33289 was elected leader, sending a full tablet report...
I20260709 13:11:50.704082  1257 ts_tablet_manager.cc:1434] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.105s	user 0.014s	sys 0.006s
I20260709 13:11:50.704249  1195 tablet_service.cc:1511] Processing CreateTablet for tablet b89f56a91e004f408b36c36d7aee0076 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:11:50.705190  1257 tablet_bootstrap.cc:492] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:50.708063  1195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b89f56a91e004f408b36c36d7aee0076. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.717211   975 tablet_service.cc:1511] Processing CreateTablet for tablet b89f56a91e004f408b36c36d7aee0076 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:11:50.718865   975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b89f56a91e004f408b36c36d7aee0076. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.719431  1122 tablet_service.cc:1511] Processing CreateTablet for tablet b89f56a91e004f408b36c36d7aee0076 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:11:50.720750  1122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b89f56a91e004f408b36c36d7aee0076. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.725755  1256 ts_tablet_manager.cc:1434] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.167s	user 0.022s	sys 0.015s
I20260709 13:11:50.726897  1256 tablet_bootstrap.cc:492] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:50.739563  1161 heartbeater.cc:499] Master 127.23.140.126:33289 was elected leader, sending a full tablet report...
I20260709 13:11:50.751708  1257 tablet_bootstrap.cc:654] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:50.754611  1254 ts_tablet_manager.cc:1434] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.241s	user 0.025s	sys 0.040s
I20260709 13:11:50.755838  1254 tablet_bootstrap.cc:492] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
W20260709 13:11:50.767246  1013 tablet.cc:2406] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 13:11:50.772193  1052 tablet_service.cc:1511] Processing CreateTablet for tablet 05ca8b4123ce4ab3a2e36c3b422e8af5 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:11:50.773542  1256 tablet_bootstrap.cc:654] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:50.775835  1255 tablet_bootstrap.cc:492] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:50.778764  1255 ts_tablet_manager.cc:1403] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.213s	user 0.008s	sys 0.007s
I20260709 13:11:50.775055  1200 tablet_service.cc:1511] Processing CreateTablet for tablet 05ca8b4123ce4ab3a2e36c3b422e8af5 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:11:50.781908  1200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ca8b4123ce4ab3a2e36c3b422e8af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.783598  1044 tablet_service.cc:1511] Processing CreateTablet for tablet 160ad44adbce4b7f97094b248fd05415 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:11:50.786595  1191 tablet_service.cc:1511] Processing CreateTablet for tablet 91c2241524a24f7e911295c67f13bd26 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:11:50.797708  1191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91c2241524a24f7e911295c67f13bd26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.799368  1255 raft_consensus.cc:359] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:50.800264  1255 raft_consensus.cc:385] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:50.800597  1255 raft_consensus.cc:740] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:50.786815   976 tablet_service.cc:1511] Processing CreateTablet for tablet 91c2241524a24f7e911295c67f13bd26 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:11:50.787149  1120 tablet_service.cc:1511] Processing CreateTablet for tablet 160ad44adbce4b7f97094b248fd05415 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:11:50.801493  1255 consensus_queue.cc:260] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:50.802114  1120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 160ad44adbce4b7f97094b248fd05415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.802309   976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91c2241524a24f7e911295c67f13bd26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.806551  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b917eb6128a4341b3e3288130a65db1" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:50.808048  1257 tablet_bootstrap.cc:492] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
W20260709 13:11:50.808399   950 leader_election.cc:343] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:50.796221  1265 raft_consensus.cc:493] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:50.808652  1257 ts_tablet_manager.cc:1403] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.104s	user 0.008s	sys 0.008s
I20260709 13:11:50.809113  1265 raft_consensus.cc:515] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:50.796504  1052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ca8b4123ce4ab3a2e36c3b422e8af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.777570   975 tablet_service.cc:1511] Processing CreateTablet for tablet 160ad44adbce4b7f97094b248fd05415 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:11:50.812335  1257 raft_consensus.cc:359] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:50.813894  1257 raft_consensus.cc:385] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:50.813333  1265 leader_election.cc:290] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:50.823875  1049 tablet_service.cc:1511] Processing CreateTablet for tablet 91c2241524a24f7e911295c67f13bd26 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:11:50.818928   975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 160ad44adbce4b7f97094b248fd05415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.827741  1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91c2241524a24f7e911295c67f13bd26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.828840  1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 160ad44adbce4b7f97094b248fd05415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.814283  1257 raft_consensus.cc:740] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:50.832844  1257 consensus_queue.cc:260] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:50.835880  1257 ts_tablet_manager.cc:1434] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.027s	user 0.005s	sys 0.005s
I20260709 13:11:50.837654  1257 tablet_bootstrap.cc:492] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:50.832792  1256 tablet_bootstrap.cc:492] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:50.840458  1256 ts_tablet_manager.cc:1403] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.114s	user 0.014s	sys 0.001s
I20260709 13:11:50.843770  1256 raft_consensus.cc:359] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:50.849874  1256 raft_consensus.cc:385] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:50.851505  1256 raft_consensus.cc:740] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:50.853940  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b917eb6128a4341b3e3288130a65db1" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:50.856932  1256 consensus_queue.cc:260] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:50.859561  1256 ts_tablet_manager.cc:1434] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.019s	user 0.000s	sys 0.006s
W20260709 13:11:50.860065   950 leader_election.cc:343] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:50.860682   950 leader_election.cc:304] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:50.861577  1260 raft_consensus.cc:2749] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:50.861702  1263 raft_consensus.cc:493] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:50.862267  1263 raft_consensus.cc:515] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:50.863158  1257 tablet_bootstrap.cc:654] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:50.882642  1124 tablet_service.cc:1511] Processing CreateTablet for tablet 545027a75bf749969ab288105031caff (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:11:50.883231   970 tablet_service.cc:1511] Processing CreateTablet for tablet 545027a75bf749969ab288105031caff (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:11:50.866324  1263 leader_election.cc:290] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:50.890937  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11cd8a46e76f4bc685aa01885b27cc57" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:11:50.895552  1173 leader_election.cc:343] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:50.899130  1124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 545027a75bf749969ab288105031caff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.903733  1255 ts_tablet_manager.cc:1434] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.125s	user 0.008s	sys 0.000s
I20260709 13:11:50.905573   970 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 545027a75bf749969ab288105031caff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.908087  1122 tablet_service.cc:1511] Processing CreateTablet for tablet e8aead26aca7435abf7d4af1ab2321af (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:11:50.909425  1122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8aead26aca7435abf7d4af1ab2321af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.904898  1119 tablet_service.cc:1511] Processing CreateTablet for tablet 05ca8b4123ce4ab3a2e36c3b422e8af5 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:11:50.910961  1119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ca8b4123ce4ab3a2e36c3b422e8af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.913172   973 tablet_service.cc:1511] Processing CreateTablet for tablet 90e13f7e16774e68bca79e3f141ed6bc (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:11:50.913527  1045 tablet_service.cc:1511] Processing CreateTablet for tablet e8aead26aca7435abf7d4af1ab2321af (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:11:50.914844  1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8aead26aca7435abf7d4af1ab2321af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.914870   973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90e13f7e16774e68bca79e3f141ed6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.932699  1255 tablet_bootstrap.cc:492] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:50.934553  1050 tablet_service.cc:1511] Processing CreateTablet for tablet 90e13f7e16774e68bca79e3f141ed6bc (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:11:50.936017  1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90e13f7e16774e68bca79e3f141ed6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.939780  1255 tablet_bootstrap.cc:654] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:50.934962  1254 tablet_bootstrap.cc:654] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:50.885640  1256 tablet_bootstrap.cc:492] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:50.963023  1118 tablet_service.cc:1511] Processing CreateTablet for tablet 90e13f7e16774e68bca79e3f141ed6bc (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:11:50.964526  1118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90e13f7e16774e68bca79e3f141ed6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.972002  1257 tablet_bootstrap.cc:492] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:50.973081  1257 ts_tablet_manager.cc:1403] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.136s	user 0.018s	sys 0.020s
I20260709 13:11:50.975557  1120 tablet_service.cc:1511] Processing CreateTablet for tablet e038f78f3e9540399aeda1f38cbea3fd (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:11:50.975984  1047 tablet_service.cc:1511] Processing CreateTablet for tablet 6aa365e967a941129ed031939342b1d8 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:11:50.976927  1120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e038f78f3e9540399aeda1f38cbea3fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.977300  1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6aa365e967a941129ed031939342b1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.978591  1191 tablet_service.cc:1511] Processing CreateTablet for tablet 545027a75bf749969ab288105031caff (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:11:50.979835  1191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 545027a75bf749969ab288105031caff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.981292  1265 raft_consensus.cc:493] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:50.981797  1265 raft_consensus.cc:515] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:50.982488  1256 tablet_bootstrap.cc:654] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:50.982193  1198 tablet_service.cc:1511] Processing CreateTablet for tablet e8aead26aca7435abf7d4af1ab2321af (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:11:50.983515  1198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8aead26aca7435abf7d4af1ab2321af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:50.984309  1265 leader_election.cc:290] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:50.991333  1257 raft_consensus.cc:359] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:50.992084  1257 raft_consensus.cc:385] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:50.992409  1257 raft_consensus.cc:740] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:51.001307   976 tablet_service.cc:1511] Processing CreateTablet for tablet e038f78f3e9540399aeda1f38cbea3fd (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:11:51.002730   976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e038f78f3e9540399aeda1f38cbea3fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.013511  1195 tablet_service.cc:1511] Processing CreateTablet for tablet 6aa365e967a941129ed031939342b1d8 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:11:51.014281   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11cd8a46e76f4bc685aa01885b27cc57" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:51.014926  1195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6aa365e967a941129ed031939342b1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.018505  1193 tablet_service.cc:1511] Processing CreateTablet for tablet e038f78f3e9540399aeda1f38cbea3fd (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:11:51.019896  1193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e038f78f3e9540399aeda1f38cbea3fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.019712   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219d22690fa145adb5b04192c1fc16bd" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:51.024010  1175 leader_election.cc:343] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.023931   975 tablet_service.cc:1511] Processing CreateTablet for tablet 6aa365e967a941129ed031939342b1d8 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:11:51.024770  1175 leader_election.cc:304] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:51.025439   975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6aa365e967a941129ed031939342b1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.025692  1265 raft_consensus.cc:2749] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:51.026363  1255 tablet_bootstrap.cc:492] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:51.026870  1255 ts_tablet_manager.cc:1403] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.094s	user 0.011s	sys 0.002s
W20260709 13:11:51.027197  1175 leader_election.cc:343] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:50.999782  1257 consensus_queue.cc:260] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:51.032176  1121 tablet_service.cc:1511] Processing CreateTablet for tablet 5d3595711f1e49dfb415fb2f6720e66d (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:11:51.033545  1121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d3595711f1e49dfb415fb2f6720e66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.044315  1266 raft_consensus.cc:493] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:51.044893  1266 raft_consensus.cc:515] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:51.047637  1266 leader_election.cc:290] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:51.043000  1257 ts_tablet_manager.cc:1434] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.070s	user 0.007s	sys 0.002s
I20260709 13:11:51.057828  1199 tablet_service.cc:1511] Processing CreateTablet for tablet 5d3595711f1e49dfb415fb2f6720e66d (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:11:51.059322  1199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d3595711f1e49dfb415fb2f6720e66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.059013   970 tablet_service.cc:1511] Processing CreateTablet for tablet 5cf962b21fa14634af6e810ca29f1bc7 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:11:51.060356   970 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf962b21fa14634af6e810ca29f1bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.072196  1049 tablet_service.cc:1511] Processing CreateTablet for tablet 5d3595711f1e49dfb415fb2f6720e66d (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:11:51.073699  1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d3595711f1e49dfb415fb2f6720e66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.083845  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35cd7c6206834f5dabfca17b22a0605a" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:51.084725  1210 raft_consensus.cc:2468] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:51.084930  1254 tablet_bootstrap.cc:492] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:51.085611  1254 ts_tablet_manager.cc:1403] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.330s	user 0.012s	sys 0.008s
I20260709 13:11:51.086501  1025 leader_election.cc:304] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:51.087443  1263 raft_consensus.cc:2804] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:51.087958  1263 raft_consensus.cc:493] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:51.088404  1263 raft_consensus.cc:3060] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:51.098361  1255 raft_consensus.cc:359] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:51.099270  1255 raft_consensus.cc:385] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:51.099849  1257 tablet_bootstrap.cc:492] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:51.099565  1052 tablet_service.cc:1511] Processing CreateTablet for tablet 5cf962b21fa14634af6e810ca29f1bc7 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:11:51.103130  1120 tablet_service.cc:1511] Processing CreateTablet for tablet 5cf962b21fa14634af6e810ca29f1bc7 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:11:51.104735  1120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf962b21fa14634af6e810ca29f1bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.105412  1052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf962b21fa14634af6e810ca29f1bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.111665  1192 tablet_service.cc:1511] Processing CreateTablet for tablet f6f69ac309b6488b81b409d785a2d465 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:11:51.112916   975 tablet_service.cc:1511] Processing CreateTablet for tablet f6f69ac309b6488b81b409d785a2d465 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:11:51.114281   975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6f69ac309b6488b81b409d785a2d465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.114079  1044 tablet_service.cc:1511] Processing CreateTablet for tablet f6f69ac309b6488b81b409d785a2d465 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:11:51.118738   973 tablet_service.cc:1511] Processing CreateTablet for tablet 7f70fc428ff6447f8dca2e27ed05683a (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:11:51.119045  1197 tablet_service.cc:1511] Processing CreateTablet for tablet 7f70fc428ff6447f8dca2e27ed05683a (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:11:51.120008   973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f70fc428ff6447f8dca2e27ed05683a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.112962  1192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6f69ac309b6488b81b409d785a2d465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.121672  1197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f70fc428ff6447f8dca2e27ed05683a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.122140  1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6f69ac309b6488b81b409d785a2d465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.125697  1119 tablet_service.cc:1511] Processing CreateTablet for tablet 7f70fc428ff6447f8dca2e27ed05683a (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:11:51.127223  1119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f70fc428ff6447f8dca2e27ed05683a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.129681  1263 raft_consensus.cc:515] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:51.130209  1254 raft_consensus.cc:359] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:51.130965  1254 raft_consensus.cc:385] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:51.131294  1254 raft_consensus.cc:740] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:51.131946  1263 leader_election.cc:290] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:51.099617  1255 raft_consensus.cc:740] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:51.132085  1254 consensus_queue.cc:260] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:51.133159  1255 consensus_queue.cc:260] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:51.133486  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35cd7c6206834f5dabfca17b22a0605a" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:51.134123  1210 raft_consensus.cc:3060] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:51.134666  1254 ts_tablet_manager.cc:1434] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.049s	user 0.005s	sys 0.002s
I20260709 13:11:51.135716  1254 tablet_bootstrap.cc:492] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:51.144520  1210 raft_consensus.cc:2468] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:51.144632   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35cd7c6206834f5dabfca17b22a0605a" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:51.145437   987 raft_consensus.cc:2468] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:51.146169  1047 tablet_service.cc:1511] Processing CreateTablet for tablet d3fbed7be9014cf5984a19768b6269d4 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:11:51.146610   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35cd7c6206834f5dabfca17b22a0605a" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:51.147260   986 raft_consensus.cc:3060] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:51.147552  1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3fbed7be9014cf5984a19768b6269d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.155591   986 raft_consensus.cc:2468] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:51.156420  1255 ts_tablet_manager.cc:1434] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.129s	user 0.008s	sys 0.000s
I20260709 13:11:51.157222  1027 leader_election.cc:304] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:51.157546  1255 tablet_bootstrap.cc:492] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:51.166114  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219d22690fa145adb5b04192c1fc16bd" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:51.169405  1126 tablet_service.cc:1511] Processing CreateTablet for tablet d3fbed7be9014cf5984a19768b6269d4 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:11:51.170766  1126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3fbed7be9014cf5984a19768b6269d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.173065  1257 tablet_bootstrap.cc:654] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
W20260709 13:11:51.173496  1173 leader_election.cc:343] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.173990  1173 leader_election.cc:304] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:51.175262  1265 raft_consensus.cc:2749] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:51.187578  1254 tablet_bootstrap.cc:654] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:51.168244  1263 raft_consensus.cc:2804] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:51.203676  1199 tablet_service.cc:1511] Processing CreateTablet for tablet d3fbed7be9014cf5984a19768b6269d4 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:11:51.205101  1199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3fbed7be9014cf5984a19768b6269d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.207224  1256 tablet_bootstrap.cc:492] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:51.207787  1256 ts_tablet_manager.cc:1403] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.324s	user 0.013s	sys 0.005s
I20260709 13:11:51.211148  1256 raft_consensus.cc:359] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:51.211966  1256 raft_consensus.cc:385] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:51.212303  1256 raft_consensus.cc:740] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:51.214164  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:11:51.216010  1099 leader_election.cc:343] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.223685  1256 consensus_queue.cc:260] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:51.226410  1263 raft_consensus.cc:697] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:11:51.231324  1263 consensus_queue.cc:237] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:51.230069   975 tablet_service.cc:1511] Processing CreateTablet for tablet eb722a3043814db7a188e49c9861ceca (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:11:51.232851  1257 tablet_bootstrap.cc:492] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:51.232962   975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb722a3043814db7a188e49c9861ceca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.233490  1257 ts_tablet_manager.cc:1403] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.137s	user 0.009s	sys 0.008s
I20260709 13:11:51.233322  1050 tablet_service.cc:1511] Processing CreateTablet for tablet eb722a3043814db7a188e49c9861ceca (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:11:51.234625  1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb722a3043814db7a188e49c9861ceca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.236595  1257 raft_consensus.cc:359] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:51.237351  1257 raft_consensus.cc:385] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:51.228668  1255 tablet_bootstrap.cc:654] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:51.237637  1257 raft_consensus.cc:740] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:51.243340  1198 tablet_service.cc:1511] Processing CreateTablet for tablet 11adc51543f7421cb3140972d9f0a424 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:11:51.245546   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:51.246021  1198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11adc51543f7421cb3140972d9f0a424. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:11:51.247283  1101 leader_election.cc:343] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.247845  1101 leader_election.cc:304] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:51.248680  1266 raft_consensus.cc:2749] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:51.245724  1124 tablet_service.cc:1511] Processing CreateTablet for tablet eb722a3043814db7a188e49c9861ceca (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:11:51.255373  1257 consensus_queue.cc:260] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:51.257411  1124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb722a3043814db7a188e49c9861ceca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.258914  1257 ts_tablet_manager.cc:1434] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.001s
I20260709 13:11:51.264146   970 tablet_service.cc:1511] Processing CreateTablet for tablet 11adc51543f7421cb3140972d9f0a424 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:11:51.265528   970 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11adc51543f7421cb3140972d9f0a424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.267572  1257 tablet_bootstrap.cc:492] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:51.267853  1266 raft_consensus.cc:493] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:51.268399  1266 raft_consensus.cc:515] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:51.270522  1266 leader_election.cc:290] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:51.272426  1052 tablet_service.cc:1511] Processing CreateTablet for tablet 11adc51543f7421cb3140972d9f0a424 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:11:51.273819  1052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11adc51543f7421cb3140972d9f0a424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.274909  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "161b9f09b3fb47febfabacf867781986" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:11:51.276629  1099 leader_election.cc:343] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.273880   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "161b9f09b3fb47febfabacf867781986" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:51.280655   986 raft_consensus.cc:2468] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:51.282307  1101 leader_election.cc:304] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:51.283797  1266 raft_consensus.cc:2804] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:51.284277  1266 raft_consensus.cc:493] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:51.284689  1266 raft_consensus.cc:3060] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:51.294721  1254 tablet_bootstrap.cc:492] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:51.295306  1254 ts_tablet_manager.cc:1403] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.160s	user 0.015s	sys 0.000s
I20260709 13:11:51.300675  1199 tablet_service.cc:1511] Processing CreateTablet for tablet a0110653f11d472797f30dc22496cb12 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:11:51.303370   973 tablet_service.cc:1511] Processing CreateTablet for tablet a0110653f11d472797f30dc22496cb12 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:11:51.304744   973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0110653f11d472797f30dc22496cb12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.304694  1286 raft_consensus.cc:493] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:51.305639  1286 raft_consensus.cc:515] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:51.309500  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:51.310745  1199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0110653f11d472797f30dc22496cb12. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:11:51.311465  1025 leader_election.cc:343] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 862b41697c74421fbac82c10c245434d (127.23.140.68:34477): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.312178  1254 raft_consensus.cc:359] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:51.320184  1288 raft_consensus.cc:493] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:51.321101  1288 raft_consensus.cc:515] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:51.322681  1044 tablet_service.cc:1511] Processing CreateTablet for tablet 7780e3bac1204186aefaa6f574717fdb (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:11:51.320863  1256 ts_tablet_manager.cc:1434] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.113s	user 0.027s	sys 0.007s
I20260709 13:11:51.320148  1120 tablet_service.cc:1511] Processing CreateTablet for tablet a0110653f11d472797f30dc22496cb12 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:11:51.324359  1256 tablet_bootstrap.cc:492] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:51.331384  1121 tablet_service.cc:1511] Processing CreateTablet for tablet 7780e3bac1204186aefaa6f574717fdb (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:11:51.318746  1254 raft_consensus.cc:385] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 13:11:51.336416   932 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:11:51.336700  1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7780e3bac1204186aefaa6f574717fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.337631  1120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0110653f11d472797f30dc22496cb12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.338950  1121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7780e3bac1204186aefaa6f574717fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.342499  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "240eb8c24e0142c8a23a0afdd5dccda6" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
W20260709 13:11:51.344297  1025 leader_election.cc:343] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.333225  1254 raft_consensus.cc:740] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:51.346743  1254 consensus_queue.cc:260] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:51.350970  1257 tablet_bootstrap.cc:654] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:51.354736  1255 tablet_bootstrap.cc:492] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:51.355301  1255 ts_tablet_manager.cc:1403] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.198s	user 0.010s	sys 0.029s
I20260709 13:11:51.358141  1288 leader_election.cc:290] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:51.358821  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "240eb8c24e0142c8a23a0afdd5dccda6" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:51.364494   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:51.364959  1025 leader_election.cc:343] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 862b41697c74421fbac82c10c245434d (127.23.140.68:34477): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.365569  1025 leader_election.cc:304] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
W20260709 13:11:51.366119  1027 leader_election.cc:343] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.363095  1286 leader_election.cc:290] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:51.366593  1027 leader_election.cc:304] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:11:51.363237  1266 raft_consensus.cc:515] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:51.369038  1195 tablet_service.cc:1511] Processing CreateTablet for tablet 7780e3bac1204186aefaa6f574717fdb (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:11:51.368888  1255 raft_consensus.cc:359] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:51.369978  1255 raft_consensus.cc:385] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:51.370353  1255 raft_consensus.cc:740] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:51.370675  1195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7780e3bac1204186aefaa6f574717fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.372614  1265 raft_consensus.cc:493] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:51.373416  1265 raft_consensus.cc:515] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:51.385700  1286 raft_consensus.cc:2749] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:51.386090  1288 raft_consensus.cc:2749] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:51.399953  1255 consensus_queue.cc:260] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:51.402858   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "161b9f09b3fb47febfabacf867781986" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:51.403518   986 raft_consensus.cc:3060] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:51.405411  1266 leader_election.cc:290] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:51.406533  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "161b9f09b3fb47febfabacf867781986" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:51.407780  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
W20260709 13:11:51.409931  1099 leader_election.cc:343] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:11:51.410845  1173 leader_election.cc:343] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.411734  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:51.412360  1050 tablet_service.cc:1511] Processing CreateTablet for tablet 47f79cfcb8a04931bc19562b5d2c7754 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
W20260709 13:11:51.413159  1173 leader_election.cc:343] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.413704  1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47f79cfcb8a04931bc19562b5d2c7754. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.413642  1173 leader_election.cc:304] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:51.416824   882 catalog_manager.cc:5697] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:51.419610   986 raft_consensus.cc:2468] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:51.420914  1101 leader_election.cc:304] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:51.406670  1265 leader_election.cc:290] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:51.425479  1265 raft_consensus.cc:2749] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:51.426013  1254 ts_tablet_manager.cc:1434] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.130s	user 0.015s	sys 0.003s
I20260709 13:11:51.426308   973 tablet_service.cc:1511] Processing CreateTablet for tablet 47f79cfcb8a04931bc19562b5d2c7754 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:11:51.424924  1266 raft_consensus.cc:2804] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:51.427349  1254 tablet_bootstrap.cc:492] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:51.427671  1266 raft_consensus.cc:697] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:11:51.435402  1255 ts_tablet_manager.cc:1434] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.080s	user 0.011s	sys 0.000s
I20260709 13:11:51.436838  1124 tablet_service.cc:1511] Processing CreateTablet for tablet 47f79cfcb8a04931bc19562b5d2c7754 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:11:51.428579  1266 consensus_queue.cc:237] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:51.444255   970 tablet_service.cc:1511] Processing CreateTablet for tablet f50253bf21654ca6937a357eb070beda (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:11:51.445405   973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47f79cfcb8a04931bc19562b5d2c7754. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.446532  1198 tablet_service.cc:1511] Processing CreateTablet for tablet f50253bf21654ca6937a357eb070beda (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:11:51.447605  1254 tablet_bootstrap.cc:654] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:51.447858  1198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f50253bf21654ca6937a357eb070beda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.450134  1255 tablet_bootstrap.cc:492] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:51.450343   970 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f50253bf21654ca6937a357eb070beda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.451224  1124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47f79cfcb8a04931bc19562b5d2c7754. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:11:51.466879  1173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 53778.4us
I20260709 13:11:51.474565  1044 tablet_service.cc:1511] Processing CreateTablet for tablet f50253bf21654ca6937a357eb070beda (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:11:51.479142  1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f50253bf21654ca6937a357eb070beda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.479427  1121 tablet_service.cc:1511] Processing CreateTablet for tablet 9eec3e72036941a5b16c3a0157986a93 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:11:51.480778  1121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eec3e72036941a5b16c3a0157986a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.482043  1256 tablet_bootstrap.cc:654] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:51.483345  1255 tablet_bootstrap.cc:654] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:51.484069  1197 tablet_service.cc:1511] Processing CreateTablet for tablet 9eec3e72036941a5b16c3a0157986a93 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:11:51.485488  1197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eec3e72036941a5b16c3a0157986a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.488555  1257 tablet_bootstrap.cc:492] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:51.489162  1257 ts_tablet_manager.cc:1403] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.222s	user 0.013s	sys 0.017s
I20260709 13:11:51.501298   975 tablet_service.cc:1511] Processing CreateTablet for tablet 9eec3e72036941a5b16c3a0157986a93 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:11:51.502718   975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eec3e72036941a5b16c3a0157986a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.513303  1050 tablet_service.cc:1511] Processing CreateTablet for tablet 4ddfc34a205c43ad81653b5000cbf394 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:11:51.514667  1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ddfc34a205c43ad81653b5000cbf394. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.520473  1120 tablet_service.cc:1511] Processing CreateTablet for tablet 4ddfc34a205c43ad81653b5000cbf394 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:11:51.522018  1120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ddfc34a205c43ad81653b5000cbf394. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.522187   973 tablet_service.cc:1511] Processing CreateTablet for tablet 483b49446c4e4a23b780921d7fabfbdf (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:11:51.523525   973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 483b49446c4e4a23b780921d7fabfbdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.539172  1198 tablet_service.cc:1511] Processing CreateTablet for tablet 4ddfc34a205c43ad81653b5000cbf394 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:11:51.539690  1052 tablet_service.cc:1511] Processing CreateTablet for tablet 483b49446c4e4a23b780921d7fabfbdf (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:11:51.540623  1198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ddfc34a205c43ad81653b5000cbf394. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.542920  1119 tablet_service.cc:1511] Processing CreateTablet for tablet 483b49446c4e4a23b780921d7fabfbdf (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:11:51.540984  1052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 483b49446c4e4a23b780921d7fabfbdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.544232  1119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 483b49446c4e4a23b780921d7fabfbdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.546537  1257 raft_consensus.cc:359] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:51.547307  1257 raft_consensus.cc:385] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:51.547582  1257 raft_consensus.cc:740] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:51.548566   883 catalog_manager.cc:5697] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:51.551258  1199 tablet_service.cc:1511] Processing CreateTablet for tablet 12df324e933f422c9a8ff5d37eb7d4ef (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:11:51.552762  1199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12df324e933f422c9a8ff5d37eb7d4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.551625  1257 consensus_queue.cc:260] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:51.559361  1254 tablet_bootstrap.cc:492] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:51.560006  1254 ts_tablet_manager.cc:1403] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.133s	user 0.018s	sys 0.004s
I20260709 13:11:51.563257  1254 raft_consensus.cc:359] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:51.565536  1255 tablet_bootstrap.cc:492] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:51.565721  1256 tablet_bootstrap.cc:492] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:51.566387  1255 ts_tablet_manager.cc:1403] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.116s	user 0.008s	sys 0.010s
I20260709 13:11:51.566452  1256 ts_tablet_manager.cc:1403] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.242s	user 0.016s	sys 0.013s
I20260709 13:11:51.569965  1255 raft_consensus.cc:359] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:51.570731  1255 raft_consensus.cc:385] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:51.571476  1255 raft_consensus.cc:740] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:51.571107  1254 raft_consensus.cc:385] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:51.572322  1254 raft_consensus.cc:740] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:51.573000  1266 raft_consensus.cc:493] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:51.573561  1266 raft_consensus.cc:515] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:51.573868  1254 consensus_queue.cc:260] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:51.579049   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:51.580605  1101 leader_election.cc:343] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.572939  1255 consensus_queue.cc:260] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:51.576071  1266 leader_election.cc:290] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:51.584331  1255 ts_tablet_manager.cc:1434] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.000s
I20260709 13:11:51.585893  1254 ts_tablet_manager.cc:1434] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.026s	user 0.005s	sys 0.001s
I20260709 13:11:51.576103  1256 raft_consensus.cc:359] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:51.582094   975 tablet_service.cc:1511] Processing CreateTablet for tablet 12df324e933f422c9a8ff5d37eb7d4ef (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:11:51.590755   975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12df324e933f422c9a8ff5d37eb7d4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.593998  1260 raft_consensus.cc:493] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:51.594519  1260 raft_consensus.cc:515] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:51.596807  1260 leader_election.cc:290] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:51.597514  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
W20260709 13:11:51.599843  1099 leader_election.cc:343] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 862b41697c74421fbac82c10c245434d (127.23.140.68:34477): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.600435  1099 leader_election.cc:304] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:11:51.608255  1254 tablet_bootstrap.cc:492] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:51.607491  1121 tablet_service.cc:1511] Processing CreateTablet for tablet fc8c8635d8e54625af9c95ecbcd5653b (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:11:51.609663  1121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc8c8635d8e54625af9c95ecbcd5653b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.589835  1256 raft_consensus.cc:385] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:51.616677  1256 raft_consensus.cc:740] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:51.581476  1257 ts_tablet_manager.cc:1434] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.086s	user 0.007s	sys 0.000s
I20260709 13:11:51.620085  1257 tablet_bootstrap.cc:492] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:51.631314  1266 raft_consensus.cc:2749] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:51.633134   970 tablet_service.cc:1511] Processing CreateTablet for tablet fc8c8635d8e54625af9c95ecbcd5653b (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:11:51.634488   970 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc8c8635d8e54625af9c95ecbcd5653b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.636871  1255 tablet_bootstrap.cc:492] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:51.650059   976 tablet_service.cc:1511] Processing CreateTablet for tablet 0867d95d9806478ba50d324c565446f6 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:11:51.651367   976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0867d95d9806478ba50d324c565446f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.652824  1197 tablet_service.cc:1511] Processing CreateTablet for tablet fc8c8635d8e54625af9c95ecbcd5653b (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:11:51.654207  1197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc8c8635d8e54625af9c95ecbcd5653b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.663939  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:51.664721  1062 raft_consensus.cc:2468] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:51.665534  1254 tablet_bootstrap.cc:654] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:51.630861  1052 tablet_service.cc:1511] Processing CreateTablet for tablet 12df324e933f422c9a8ff5d37eb7d4ef (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:11:51.669359  1052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12df324e933f422c9a8ff5d37eb7d4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.672856  1050 tablet_service.cc:1511] Processing CreateTablet for tablet f267e3e7ffec446c9dfb1cfd18ace091 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:11:51.674062  1119 tablet_service.cc:1511] Processing CreateTablet for tablet f267e3e7ffec446c9dfb1cfd18ace091 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:11:51.674468  1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f267e3e7ffec446c9dfb1cfd18ace091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.675402  1119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f267e3e7ffec446c9dfb1cfd18ace091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.628997  1256 consensus_queue.cc:260] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:51.678932  1256 ts_tablet_manager.cc:1434] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.112s	user 0.006s	sys 0.000s
I20260709 13:11:51.679984  1256 tablet_bootstrap.cc:492] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:51.684352  1263 consensus_queue.cc:1048] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:51.685945  1199 tablet_service.cc:1511] Processing CreateTablet for tablet f267e3e7ffec446c9dfb1cfd18ace091 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:11:51.686730  1257 tablet_bootstrap.cc:654] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:51.687355  1199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f267e3e7ffec446c9dfb1cfd18ace091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.670876   950 leader_election.cc:304] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:51.707374  1260 raft_consensus.cc:2804] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:51.707927  1260 raft_consensus.cc:493] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:51.708338  1260 raft_consensus.cc:3060] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:51.710212  1044 tablet_service.cc:1511] Processing CreateTablet for tablet 0867d95d9806478ba50d324c565446f6 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:11:51.710415  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:51.711630  1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0867d95d9806478ba50d324c565446f6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:11:51.713647   950 leader_election.cc:343] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 862b41697c74421fbac82c10c245434d (127.23.140.68:34477): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.715538  1120 tablet_service.cc:1511] Processing CreateTablet for tablet 0867d95d9806478ba50d324c565446f6 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:11:51.719507  1197 tablet_service.cc:1511] Processing CreateTablet for tablet 9b75f9192e7246d2a560fd091fb68606 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:11:51.720825  1197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b75f9192e7246d2a560fd091fb68606. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.735234  1120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0867d95d9806478ba50d324c565446f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.754383   976 tablet_service.cc:1511] Processing CreateTablet for tablet 9b75f9192e7246d2a560fd091fb68606 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:11:51.755827   976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b75f9192e7246d2a560fd091fb68606. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.757823  1293 raft_consensus.cc:493] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:51.758359  1293 raft_consensus.cc:515] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:51.758447  1049 tablet_service.cc:1511] Processing CreateTablet for tablet 9b75f9192e7246d2a560fd091fb68606 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:11:51.771853  1293 leader_election.cc:290] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:51.774328  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f95cbeb7360c4c61b442e75e35d93e73" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:51.775581  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f95cbeb7360c4c61b442e75e35d93e73" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
W20260709 13:11:51.777418  1173 leader_election.cc:343] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.768096  1255 tablet_bootstrap.cc:654] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
W20260709 13:11:51.778152  1173 leader_election.cc:343] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.778601  1173 leader_election.cc:304] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:51.780256  1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b75f9192e7246d2a560fd091fb68606. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.785918  1266 raft_consensus.cc:493] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:51.786440  1266 raft_consensus.cc:515] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:51.788501  1266 leader_election.cc:290] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:51.788308  1260 raft_consensus.cc:515] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:51.785298  1293 raft_consensus.cc:2749] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:51.790196   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219d22690fa145adb5b04192c1fc16bd" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:51.793913   986 raft_consensus.cc:2468] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:51.795161  1101 leader_election.cc:304] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:51.797008  1208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219d22690fa145adb5b04192c1fc16bd" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:51.797716  1208 raft_consensus.cc:2468] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:51.798435   970 tablet_service.cc:1511] Processing CreateTablet for tablet 75b503b2e43f4996b5f494ab0e65afb0 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:11:51.799396  1266 raft_consensus.cc:2804] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:51.799638   970 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75b503b2e43f4996b5f494ab0e65afb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.799877  1266 raft_consensus.cc:493] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:51.799921  1047 tablet_service.cc:1511] Processing CreateTablet for tablet a260eb172eba4c6db7c0d0fab6a7f7d1 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:11:51.800416  1266 raft_consensus.cc:3060] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:51.801432  1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a260eb172eba4c6db7c0d0fab6a7f7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.802254  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:51.802742  1119 tablet_service.cc:1511] Processing CreateTablet for tablet a260eb172eba4c6db7c0d0fab6a7f7d1 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
W20260709 13:11:51.803555   950 leader_election.cc:343] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 862b41697c74421fbac82c10c245434d (127.23.140.68:34477): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.804131  1119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a260eb172eba4c6db7c0d0fab6a7f7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.799304  1199 tablet_service.cc:1511] Processing CreateTablet for tablet 75b503b2e43f4996b5f494ab0e65afb0 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:11:51.809239  1256 tablet_bootstrap.cc:654] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:51.809569  1199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75b503b2e43f4996b5f494ab0e65afb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.814159  1257 tablet_bootstrap.cc:492] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:51.814693  1257 ts_tablet_manager.cc:1403] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.195s	user 0.005s	sys 0.013s
I20260709 13:11:51.818120  1257 raft_consensus.cc:359] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:51.818941  1257 raft_consensus.cc:385] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:51.818176  1121 tablet_service.cc:1511] Processing CreateTablet for tablet 75b503b2e43f4996b5f494ab0e65afb0 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:11:51.820289  1121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75b503b2e43f4996b5f494ab0e65afb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.820557  1266 raft_consensus.cc:515] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:51.823485  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219d22690fa145adb5b04192c1fc16bd" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:51.824077  1210 raft_consensus.cc:3060] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:51.826800  1307 raft_consensus.cc:493] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:51.827502  1307 raft_consensus.cc:515] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:51.819267  1257 raft_consensus.cc:740] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:51.828896  1266 leader_election.cc:290] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:51.829720  1307 leader_election.cc:290] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:51.834046  1256 tablet_bootstrap.cc:492] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:51.836941   973 tablet_service.cc:1511] Processing CreateTablet for tablet 09039e63d8db4e06b365f43533071272 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:11:51.837826   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219d22690fa145adb5b04192c1fc16bd" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:51.842443   986 raft_consensus.cc:3060] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:51.842461  1045 tablet_service.cc:1511] Processing CreateTablet for tablet 09039e63d8db4e06b365f43533071272 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:11:51.843151  1210 raft_consensus.cc:2468] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:51.843921  1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09039e63d8db4e06b365f43533071272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.838539  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:51.841300   973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09039e63d8db4e06b365f43533071272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.841811  1208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219d22690fa145adb5b04192c1fc16bd" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:51.839258  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219d22690fa145adb5b04192c1fc16bd" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:51.829771  1257 consensus_queue.cc:260] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:51.850417  1257 ts_tablet_manager.cc:1434] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.035s	user 0.005s	sys 0.003s
I20260709 13:11:51.851485  1257 tablet_bootstrap.cc:492] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:51.852632  1197 tablet_service.cc:1511] Processing CreateTablet for tablet a260eb172eba4c6db7c0d0fab6a7f7d1 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:11:51.853407  1136 raft_consensus.cc:2393] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 in current term 1: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:11:51.853961  1197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a260eb172eba4c6db7c0d0fab6a7f7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.854367  1208 raft_consensus.cc:2393] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 in current term 1: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:11:51.857656  1062 raft_consensus.cc:3060] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:51.861968  1099 leader_election.cc:304] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:51.861961  1257 tablet_bootstrap.cc:654] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:51.863188   950 leader_election.cc:304] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:51.863963  1288 consensus_queue.cc:1048] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:51.866873  1288 raft_consensus.cc:493] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:51.867466  1288 raft_consensus.cc:515] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:51.868384  1254 tablet_bootstrap.cc:492] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:51.869215  1254 ts_tablet_manager.cc:1403] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.261s	user 0.024s	sys 0.001s
I20260709 13:11:51.872320  1124 tablet_service.cc:1511] Processing CreateTablet for tablet 09039e63d8db4e06b365f43533071272 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:11:51.873315  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2312d768cbf4b3495d5b1574b0b6eb5" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:51.873879  1124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09039e63d8db4e06b365f43533071272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.874027  1209 raft_consensus.cc:2468] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:51.875257  1025 leader_election.cc:304] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:51.876350  1286 raft_consensus.cc:2804] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:51.876828  1286 raft_consensus.cc:493] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:51.877259  1286 raft_consensus.cc:3060] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:51.886821   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2312d768cbf4b3495d5b1574b0b6eb5" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:51.888568  1027 leader_election.cc:343] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.889173  1288 leader_election.cc:290] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:51.895826  1286 raft_consensus.cc:515] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:51.830576  1260 leader_election.cc:290] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:51.899399  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2312d768cbf4b3495d5b1574b0b6eb5" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:51.900017  1209 raft_consensus.cc:3060] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:51.900004  1254 raft_consensus.cc:359] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:51.901041  1254 raft_consensus.cc:385] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:51.847007  1256 ts_tablet_manager.cc:1403] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.165s	user 0.011s	sys 0.008s
I20260709 13:11:51.906344  1198 tablet_service.cc:1511] Processing CreateTablet for tablet 21783150bb0b4141be797e841c3f46fa (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:11:51.907714  1198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21783150bb0b4141be797e841c3f46fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.906157   986 raft_consensus.cc:2468] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:51.902410  1254 raft_consensus.cc:740] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:51.908519  1307 raft_consensus.cc:2749] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:51.902698  1266 raft_consensus.cc:2804] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:51.919634   973 tablet_service.cc:1511] Processing CreateTablet for tablet 21783150bb0b4141be797e841c3f46fa (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:11:51.921700   973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21783150bb0b4141be797e841c3f46fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.908279  1256 raft_consensus.cc:359] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:51.923127  1256 raft_consensus.cc:385] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:51.918304  1266 raft_consensus.cc:697] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:11:51.926362  1062 raft_consensus.cc:2468] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:51.926628   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2312d768cbf4b3495d5b1574b0b6eb5" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:51.926445  1119 tablet_service.cc:1511] Processing CreateTablet for tablet 3e9125adfbbd414ca6b40457fc69ad84 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:11:51.928012  1119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e9125adfbbd414ca6b40457fc69ad84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.930451   950 leader_election.cc:304] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 862b41697c74421fbac82c10c245434d
I20260709 13:11:51.931445  1195 tablet_service.cc:1511] Processing CreateTablet for tablet 3e9125adfbbd414ca6b40457fc69ad84 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:11:51.932822  1195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e9125adfbbd414ca6b40457fc69ad84. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:11:51.938282  1027 leader_election.cc:343] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:51.939536  1313 raft_consensus.cc:2804] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:51.940534  1313 raft_consensus.cc:697] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:11:51.941996  1049 tablet_service.cc:1511] Processing CreateTablet for tablet 21783150bb0b4141be797e841c3f46fa (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:11:51.929522  1286 leader_election.cc:290] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:51.943508  1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21783150bb0b4141be797e841c3f46fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.943743  1266 consensus_queue.cc:237] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:51.948271   976 tablet_service.cc:1511] Processing CreateTablet for tablet 3e9125adfbbd414ca6b40457fc69ad84 (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:11:51.949585   976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e9125adfbbd414ca6b40457fc69ad84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.950773  1050 tablet_service.cc:1511] Processing CreateTablet for tablet 9bbe4fae89eb429b989a7e1d87ab7dcf (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:11:51.952108  1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbe4fae89eb429b989a7e1d87ab7dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.923573  1256 raft_consensus.cc:740] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:51.959060  1256 consensus_queue.cc:260] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:51.941712  1313 consensus_queue.cc:237] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:51.917379  1254 consensus_queue.cc:260] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:51.966042  1255 tablet_bootstrap.cc:492] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:51.966627  1255 ts_tablet_manager.cc:1403] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.335s	user 0.018s	sys 0.006s
I20260709 13:11:51.969345  1256 ts_tablet_manager.cc:1434] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.066s	user 0.007s	sys 0.001s
I20260709 13:11:51.970474  1256 tablet_bootstrap.cc:492] T 161b9f09b3fb47febfabacf867781986 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:51.978773  1120 tablet_service.cc:1511] Processing CreateTablet for tablet 9bbe4fae89eb429b989a7e1d87ab7dcf (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:11:51.982686  1266 consensus_queue.cc:1048] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:51.992347  1120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbe4fae89eb429b989a7e1d87ab7dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:51.992684  1193 tablet_service.cc:1511] Processing CreateTablet for tablet 9bbe4fae89eb429b989a7e1d87ab7dcf (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:11:51.994130  1193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbe4fae89eb429b989a7e1d87ab7dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:52.004573  1254 ts_tablet_manager.cc:1434] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.135s	user 0.004s	sys 0.003s
I20260709 13:11:52.005743  1254 tablet_bootstrap.cc:492] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:52.020309  1255 raft_consensus.cc:359] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
W20260709 13:11:52.021176  1099 consensus_peers.cc:597] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:52.021176  1255 raft_consensus.cc:385] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.029026  1255 raft_consensus.cc:740] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.030771  1255 consensus_queue.cc:260] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.032593  1255 ts_tablet_manager.cc:1434] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.066s	user 0.006s	sys 0.000s
I20260709 13:11:52.032904  1286 raft_consensus.cc:493] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.033428  1286 raft_consensus.cc:515] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.035588  1286 leader_election.cc:290] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:52.036700  1255 tablet_bootstrap.cc:492] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:52.037071   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b917eb6128a4341b3e3288130a65db1" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:52.044178   986 raft_consensus.cc:2468] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:52.045696  1027 leader_election.cc:304] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:52.047360  1121 tablet_service.cc:1511] Processing CreateTablet for tablet ff504aa514be43da8b1aac12ab550daf (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:11:52.048722  1121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff504aa514be43da8b1aac12ab550daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:52.050237  1197 tablet_service.cc:1511] Processing CreateTablet for tablet ff504aa514be43da8b1aac12ab550daf (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:11:52.051633  1197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff504aa514be43da8b1aac12ab550daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:52.052912   973 tablet_service.cc:1511] Processing CreateTablet for tablet ff504aa514be43da8b1aac12ab550daf (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:11:52.054095   973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff504aa514be43da8b1aac12ab550daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:52.038341  1209 raft_consensus.cc:2468] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:52.038298  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b917eb6128a4341b3e3288130a65db1" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:52.041033  1254 tablet_bootstrap.cc:654] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.055706  1286 raft_consensus.cc:2804] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 13:11:52.057413  1025 leader_election.cc:343] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:52.037789   970 tablet_service.cc:1511] Processing CreateTablet for tablet 8acf44a40c084675a4f076a7acbda99d (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:11:52.059000   970 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8acf44a40c084675a4f076a7acbda99d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:52.037882   976 tablet_service.cc:1511] Processing CreateTablet for tablet f31edd3b8d2546a2a98a8a8d3d038f8b (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:11:52.062047   976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31edd3b8d2546a2a98a8a8d3d038f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:52.043156  1047 tablet_service.cc:1511] Processing CreateTablet for tablet 8acf44a40c084675a4f076a7acbda99d (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:11:52.044634  1124 tablet_service.cc:1511] Processing CreateTablet for tablet f31edd3b8d2546a2a98a8a8d3d038f8b (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:11:52.050777  1257 tablet_bootstrap.cc:492] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:52.076683  1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8acf44a40c084675a4f076a7acbda99d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:52.047854  1199 tablet_service.cc:1511] Processing CreateTablet for tablet 8acf44a40c084675a4f076a7acbda99d (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:11:52.078898  1199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8acf44a40c084675a4f076a7acbda99d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:11:52.029457  1101 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 53690us
I20260709 13:11:52.081597  1025 leader_election.cc:304] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:11:52.082558  1288 raft_consensus.cc:2804] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:52.056752  1286 raft_consensus.cc:493] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:52.084486  1257 ts_tablet_manager.cc:1403] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.233s	user 0.015s	sys 0.019s
I20260709 13:11:52.083885   882 catalog_manager.cc:5697] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:52.086324  1288 raft_consensus.cc:697] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:11:52.098071  1124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31edd3b8d2546a2a98a8a8d3d038f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:52.097854  1288 consensus_queue.cc:237] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.100565  1317 raft_consensus.cc:493] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.101083  1317 raft_consensus.cc:515] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.103232  1317 leader_election.cc:290] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:52.105183  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:52.107270  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:52.107949  1209 raft_consensus.cc:2468] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
W20260709 13:11:52.108979  1099 leader_election.cc:343] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:52.113204  1256 tablet_bootstrap.cc:654] T 161b9f09b3fb47febfabacf867781986 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.120682  1255 tablet_bootstrap.cc:654] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.121806  1099 leader_election.cc:304] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:52.088932  1286 raft_consensus.cc:3060] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.123556  1257 raft_consensus.cc:359] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.127200  1317 raft_consensus.cc:2804] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:52.082082  1045 tablet_service.cc:1511] Processing CreateTablet for tablet f31edd3b8d2546a2a98a8a8d3d038f8b (DEFAULT_TABLE table=test-workload [id=37f1d21c46bb49e88be6c2ec07c11778]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:11:52.141928  1317 raft_consensus.cc:493] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:52.147027  1317 raft_consensus.cc:3060] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.146682  1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31edd3b8d2546a2a98a8a8d3d038f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:11:52.141870  1257 raft_consensus.cc:385] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.153589  1286 raft_consensus.cc:515] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.163964  1254 tablet_bootstrap.cc:492] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:52.164546  1254 ts_tablet_manager.cc:1403] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.159s	user 0.013s	sys 0.065s
I20260709 13:11:52.171947   885 catalog_manager.cc:5697] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 0 to 1, leader changed from <none> to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:52.157815  1257 raft_consensus.cc:740] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.175853  1257 consensus_queue.cc:260] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.178454  1257 ts_tablet_manager.cc:1434] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.091s	user 0.003s	sys 0.005s
I20260709 13:11:52.172703  1254 raft_consensus.cc:359] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.179113  1254 raft_consensus.cc:385] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.179414  1254 raft_consensus.cc:740] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.180193  1254 consensus_queue.cc:260] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.187546  1257 tablet_bootstrap.cc:492] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:52.190449  1317 raft_consensus.cc:515] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.198879  1254 ts_tablet_manager.cc:1434] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.034s	user 0.005s	sys 0.003s
I20260709 13:11:52.200052   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b917eb6128a4341b3e3288130a65db1" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:52.200724   986 raft_consensus.cc:3060] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.203351  1256 tablet_bootstrap.cc:492] T 161b9f09b3fb47febfabacf867781986 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:52.204423  1256 ts_tablet_manager.cc:1403] T 161b9f09b3fb47febfabacf867781986 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.234s	user 0.015s	sys 0.013s
I20260709 13:11:52.205648  1286 leader_election.cc:290] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:52.207608  1256 raft_consensus.cc:359] T 161b9f09b3fb47febfabacf867781986 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.208325  1256 raft_consensus.cc:385] T 161b9f09b3fb47febfabacf867781986 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.208626  1256 raft_consensus.cc:740] T 161b9f09b3fb47febfabacf867781986 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.208809   986 raft_consensus.cc:2468] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:52.209455  1256 consensus_queue.cc:260] T 161b9f09b3fb47febfabacf867781986 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.210199  1027 leader_election.cc:304] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:52.211627  1254 tablet_bootstrap.cc:492] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:52.211573  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b917eb6128a4341b3e3288130a65db1" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:52.212383  1136 raft_consensus.cc:3060] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.212073   884 catalog_manager.cc:5697] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:52.216140  1257 tablet_bootstrap.cc:654] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.217953  1317 leader_election.cc:290] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:52.218478  1255 tablet_bootstrap.cc:492] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:52.220781  1254 tablet_bootstrap.cc:654] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.231447  1255 ts_tablet_manager.cc:1403] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.195s	user 0.015s	sys 0.000s
I20260709 13:11:52.221349  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:52.221360  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:52.234629  1255 raft_consensus.cc:359] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.235280  1209 raft_consensus.cc:3060] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.226162  1286 raft_consensus.cc:2804] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:52.228641  1136 raft_consensus.cc:2468] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:52.237326  1286 raft_consensus.cc:697] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:11:52.238235  1286 consensus_queue.cc:237] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.238802  1257 tablet_bootstrap.cc:492] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:52.239338  1257 ts_tablet_manager.cc:1403] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.052s	user 0.004s	sys 0.014s
W20260709 13:11:52.240437  1099 leader_election.cc:343] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:52.235392  1255 raft_consensus.cc:385] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.241530  1255 raft_consensus.cc:740] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.242775  1209 raft_consensus.cc:2468] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:52.242506  1255 consensus_queue.cc:260] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.227952  1256 ts_tablet_manager.cc:1434] T 161b9f09b3fb47febfabacf867781986 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.000s
I20260709 13:11:52.244134  1099 leader_election.cc:304] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:52.244648  1256 tablet_bootstrap.cc:492] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:52.245337  1317 raft_consensus.cc:2804] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:52.245852  1317 raft_consensus.cc:697] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:11:52.246641  1317 consensus_queue.cc:237] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.248556  1254 tablet_bootstrap.cc:492] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:52.249068  1254 ts_tablet_manager.cc:1403] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.038s	user 0.008s	sys 0.004s
I20260709 13:11:52.252087  1254 raft_consensus.cc:359] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.252828  1254 raft_consensus.cc:385] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.253149  1254 raft_consensus.cc:740] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.253870  1254 consensus_queue.cc:260] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.254338  1255 ts_tablet_manager.cc:1434] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.001s
I20260709 13:11:52.255987  1255 tablet_bootstrap.cc:492] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:52.258584  1256 tablet_bootstrap.cc:654] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.256356  1254 ts_tablet_manager.cc:1434] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 13:11:52.264761  1254 tablet_bootstrap.cc:492] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:52.257651  1257 raft_consensus.cc:359] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.264879   882 catalog_manager.cc:5697] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:52.266002  1257 raft_consensus.cc:385] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.266353  1257 raft_consensus.cc:740] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.267228  1257 consensus_queue.cc:260] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.272017  1254 tablet_bootstrap.cc:654] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.272074  1257 ts_tablet_manager.cc:1434] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.000s
I20260709 13:11:52.273283  1257 tablet_bootstrap.cc:492] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:52.279063  1255 tablet_bootstrap.cc:654] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.283205  1257 tablet_bootstrap.cc:654] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.284502  1256 tablet_bootstrap.cc:492] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:52.285038  1256 ts_tablet_manager.cc:1403] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.041s	user 0.010s	sys 0.002s
I20260709 13:11:52.288412  1256 raft_consensus.cc:359] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.289268  1256 raft_consensus.cc:385] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.289641  1256 raft_consensus.cc:740] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.290714  1256 consensus_queue.cc:260] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.291338  1257 tablet_bootstrap.cc:492] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:52.291853  1257 ts_tablet_manager.cc:1403] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.019s	user 0.009s	sys 0.000s
I20260709 13:11:52.293154  1256 ts_tablet_manager.cc:1434] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:11:52.295980  1256 tablet_bootstrap.cc:492] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:52.294579  1257 raft_consensus.cc:359] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.297392  1254 tablet_bootstrap.cc:492] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:52.297657  1257 raft_consensus.cc:385] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.298130  1254 ts_tablet_manager.cc:1403] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.034s	user 0.017s	sys 0.012s
I20260709 13:11:52.298086  1257 raft_consensus.cc:740] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.298966  1257 consensus_queue.cc:260] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.300776  1257 ts_tablet_manager.cc:1434] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 13:11:52.297362  1255 tablet_bootstrap.cc:492] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:52.294026   884 catalog_manager.cc:5697] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:52.301155  1254 raft_consensus.cc:359] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.301921  1254 raft_consensus.cc:385] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.302211  1254 raft_consensus.cc:740] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.302510  1257 tablet_bootstrap.cc:492] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:52.301636  1255 ts_tablet_manager.cc:1403] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.046s	user 0.008s	sys 0.005s
I20260709 13:11:52.305449  1255 raft_consensus.cc:359] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.306078  1255 raft_consensus.cc:385] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.306329  1255 raft_consensus.cc:740] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.307292  1307 raft_consensus.cc:493] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.307073  1254 consensus_queue.cc:260] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.307756  1307 raft_consensus.cc:515] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.308167  1255 consensus_queue.cc:260] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.308966  1257 tablet_bootstrap.cc:654] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.309872  1307 leader_election.cc:290] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:52.310438  1317 raft_consensus.cc:493] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.310884  1317 raft_consensus.cc:515] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.312978  1317 leader_election.cc:290] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:52.314795  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "240eb8c24e0142c8a23a0afdd5dccda6" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:52.315059  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:52.315322  1062 raft_consensus.cc:2468] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:52.316381  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:52.316668  1099 leader_election.cc:304] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:52.317709  1254 ts_tablet_manager.cc:1434] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 13:11:52.317848  1256 tablet_bootstrap.cc:654] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.320153  1255 ts_tablet_manager.cc:1434] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.000s
I20260709 13:11:52.320675  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "240eb8c24e0142c8a23a0afdd5dccda6" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:52.321056  1255 tablet_bootstrap.cc:492] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
W20260709 13:11:52.322019   950 leader_election.cc:343] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:11:52.322757   950 leader_election.cc:343] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:52.323186   950 leader_election.cc:304] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:52.327104  1254 tablet_bootstrap.cc:492] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:52.327831  1307 raft_consensus.cc:2749] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:52.334740  1254 tablet_bootstrap.cc:654] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.334820  1317 raft_consensus.cc:2804] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:52.335461  1317 raft_consensus.cc:493] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:52.335924  1317 raft_consensus.cc:3060] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.326829  1255 tablet_bootstrap.cc:654] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
W20260709 13:11:52.339880   932 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260709 13:11:52.334369  1099 leader_election.cc:343] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 862b41697c74421fbac82c10c245434d (127.23.140.68:34477): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:52.353484  1254 tablet_bootstrap.cc:492] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:52.354094  1254 ts_tablet_manager.cc:1403] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.036s	user 0.012s	sys 0.002s
I20260709 13:11:52.354640  1255 tablet_bootstrap.cc:492] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:52.355167  1255 ts_tablet_manager.cc:1403] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.034s	user 0.010s	sys 0.000s
I20260709 13:11:52.357007  1254 raft_consensus.cc:359] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.358331  1254 raft_consensus.cc:385] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.357172  1317 raft_consensus.cc:515] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.358811  1254 raft_consensus.cc:740] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.359647  1254 consensus_queue.cc:260] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.360322  1317 leader_election.cc:290] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:52.361442  1255 raft_consensus.cc:359] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.361894  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "240eb8c24e0142c8a23a0afdd5dccda6" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:52.362142  1255 raft_consensus.cc:385] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.362541  1061 raft_consensus.cc:3060] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.362574  1255 raft_consensus.cc:740] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.362820  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "240eb8c24e0142c8a23a0afdd5dccda6" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:52.362201  1257 tablet_bootstrap.cc:492] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:52.363485  1255 consensus_queue.cc:260] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.364101  1257 ts_tablet_manager.cc:1403] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.062s	user 0.025s	sys 0.023s
I20260709 13:11:52.364712  1254 ts_tablet_manager.cc:1434] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.001s
I20260709 13:11:52.365664  1256 tablet_bootstrap.cc:492] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:52.365737  1254 tablet_bootstrap.cc:492] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:52.366248  1256 ts_tablet_manager.cc:1403] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.071s	user 0.005s	sys 0.013s
W20260709 13:11:52.366638  1099 leader_election.cc:343] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 862b41697c74421fbac82c10c245434d (127.23.140.68:34477): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:52.368697  1257 raft_consensus.cc:359] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.369391  1257 raft_consensus.cc:385] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.369670  1257 raft_consensus.cc:740] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.370474  1257 consensus_queue.cc:260] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.372542  1257 ts_tablet_manager.cc:1434] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.000s
I20260709 13:11:52.373288  1255 ts_tablet_manager.cc:1434] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.001s
I20260709 13:11:52.373924  1307 raft_consensus.cc:493] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.373792  1256 raft_consensus.cc:359] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.374542  1256 raft_consensus.cc:385] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.374455  1307 raft_consensus.cc:515] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.374922  1256 raft_consensus.cc:740] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.375784  1256 consensus_queue.cc:260] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.376629  1254 tablet_bootstrap.cc:654] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.378027  1256 ts_tablet_manager.cc:1434] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.003s
I20260709 13:11:52.380260  1061 raft_consensus.cc:2468] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:52.380419  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:52.381099  1136 raft_consensus.cc:2468] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:52.381325  1099 leader_election.cc:304] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d
I20260709 13:11:52.382369  1291 raft_consensus.cc:2804] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:52.382860  1291 raft_consensus.cc:697] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:11:52.376636  1307 leader_election.cc:290] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:52.383708  1291 consensus_queue.cc:237] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.386519  1257 tablet_bootstrap.cc:492] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:52.386678  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:11:52.387946   950 leader_election.cc:343] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:52.388548  1256 tablet_bootstrap.cc:492] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:52.388762  1266 raft_consensus.cc:493] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.389040  1307 raft_consensus.cc:493] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.389303  1266 raft_consensus.cc:515] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.389592  1307 raft_consensus.cc:515] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.391559  1266 leader_election.cc:290] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:52.392520  1307 leader_election.cc:290] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:52.394016  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:52.394738  1209 raft_consensus.cc:2468] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:52.395356  1256 tablet_bootstrap.cc:654] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.396018   950 leader_election.cc:304] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 
I20260709 13:11:52.396826  1307 raft_consensus.cc:2804] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:52.398095  1307 raft_consensus.cc:493] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:52.398478  1307 raft_consensus.cc:3060] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.394057  1257 tablet_bootstrap.cc:654] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.398901  1317 consensus_queue.cc:1048] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:52.401490   950 leader_election.cc:304] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:52.402614  1313 raft_consensus.cc:2804] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:52.402976  1313 raft_consensus.cc:493] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:52.403298  1313 raft_consensus.cc:3060] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.406168  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f95cbeb7360c4c61b442e75e35d93e73" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:52.409384  1254 tablet_bootstrap.cc:492] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:52.410717  1254 ts_tablet_manager.cc:1403] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.045s	user 0.011s	sys 0.006s
I20260709 13:11:52.410665  1313 raft_consensus.cc:515] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.411285  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f95cbeb7360c4c61b442e75e35d93e73" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:52.411906  1209 raft_consensus.cc:2468] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:52.411814  1255 tablet_bootstrap.cc:492] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:52.413821  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
W20260709 13:11:52.414136  1099 leader_election.cc:343] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:52.414271  1254 raft_consensus.cc:359] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.414897  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:52.415041  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:52.415179  1254 raft_consensus.cc:385] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.414429  1136 raft_consensus.cc:2468] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:52.417745  1254 raft_consensus.cc:740] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.418251  1135 raft_consensus.cc:3060] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:11:52.418601   950 leader_election.cc:343] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:52.418864  1313 leader_election.cc:290] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:52.418584  1254 consensus_queue.cc:260] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.406826  1307 raft_consensus.cc:515] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.425912  1135 raft_consensus.cc:2468] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:52.425822  1317 consensus_queue.cc:1048] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:52.427794   883 catalog_manager.cc:5697] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:52.430702  1307 leader_election.cc:290] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:52.432379  1257 tablet_bootstrap.cc:492] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:52.433027  1257 ts_tablet_manager.cc:1403] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.047s	user 0.013s	sys 0.000s
I20260709 13:11:52.436117  1255 tablet_bootstrap.cc:654] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.436187  1257 raft_consensus.cc:359] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.436993  1257 raft_consensus.cc:385] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.437290  1257 raft_consensus.cc:740] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.438192  1257 consensus_queue.cc:260] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.440722  1254 ts_tablet_manager.cc:1434] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.029s	user 0.003s	sys 0.004s
I20260709 13:11:52.440722  1257 ts_tablet_manager.cc:1434] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:11:52.442011  1257 tablet_bootstrap.cc:492] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:52.442427  1099 leader_election.cc:304] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:52.444145  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:52.445745  1317 raft_consensus.cc:2804] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:52.442018  1254 tablet_bootstrap.cc:492] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:52.446197  1317 raft_consensus.cc:493] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:52.446556  1317 raft_consensus.cc:3060] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.450011  1257 tablet_bootstrap.cc:654] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.449783   950 leader_election.cc:304] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:52.450350  1293 raft_consensus.cc:493] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.450822  1293 raft_consensus.cc:515] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.451205  1340 raft_consensus.cc:2804] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:52.451512  1256 tablet_bootstrap.cc:492] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:52.451797  1340 raft_consensus.cc:697] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:11:52.452021  1256 ts_tablet_manager.cc:1403] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.064s	user 0.008s	sys 0.003s
I20260709 13:11:52.452768  1340 consensus_queue.cc:237] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.455123   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:52.457860  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:52.458715  1062 raft_consensus.cc:2393] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 1: Already voted for candidate 294145900dd3496cb4a1d722613e1558 in this term.
I20260709 13:11:52.458415  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:52.459628  1135 raft_consensus.cc:3060] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.460150  1061 raft_consensus.cc:3060] T 161b9f09b3fb47febfabacf867781986 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.461997  1256 raft_consensus.cc:359] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.462718  1256 raft_consensus.cc:385] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.462981  1256 raft_consensus.cc:740] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.463313  1173 leader_election.cc:304] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:52.452979  1293 leader_election.cc:290] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:52.465751  1349 raft_consensus.cc:3060] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.444859  1209 raft_consensus.cc:3060] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.472750  1317 raft_consensus.cc:515] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.477510  1256 consensus_queue.cc:260] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.478835  1135 raft_consensus.cc:2468] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:52.480394   950 leader_election.cc:304] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:52.481292  1340 raft_consensus.cc:2804] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:52.481808  1340 raft_consensus.cc:697] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:11:52.482872  1340 consensus_queue.cc:237] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.488566  1209 raft_consensus.cc:2468] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:52.457887  1254 tablet_bootstrap.cc:654] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.496065  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f95cbeb7360c4c61b442e75e35d93e73" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
W20260709 13:11:52.497696  1099 leader_election.cc:343] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:52.498080  1254 tablet_bootstrap.cc:492] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:52.502441  1254 ts_tablet_manager.cc:1403] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.061s	user 0.012s	sys 0.000s
I20260709 13:11:52.502985  1317 leader_election.cc:290] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:52.505246  1254 raft_consensus.cc:359] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.505988  1254 raft_consensus.cc:385] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.506253  1254 raft_consensus.cc:740] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.506918  1254 consensus_queue.cc:260] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.498713  1256 ts_tablet_manager.cc:1434] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.046s	user 0.009s	sys 0.000s
I20260709 13:11:52.508934  1256 tablet_bootstrap.cc:492] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:52.509975  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f95cbeb7360c4c61b442e75e35d93e73" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:52.509883  1340 consensus_queue.cc:1048] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:52.510782  1210 raft_consensus.cc:3060] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.511924  1349 raft_consensus.cc:2749] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:52.516297  1256 tablet_bootstrap.cc:654] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.517098  1307 consensus_queue.cc:1048] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:52.500558  1255 tablet_bootstrap.cc:492] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:52.519855  1255 ts_tablet_manager.cc:1403] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.108s	user 0.021s	sys 0.000s
I20260709 13:11:52.522042  1263 raft_consensus.cc:493] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.522316  1210 raft_consensus.cc:2468] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:52.522874  1263 raft_consensus.cc:515] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.523701  1099 leader_election.cc:304] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:52.524567  1291 raft_consensus.cc:2804] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:52.525031  1291 raft_consensus.cc:697] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:11:52.525847  1291 consensus_queue.cc:237] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.530637  1254 ts_tablet_manager.cc:1434] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.028s	user 0.006s	sys 0.000s
I20260709 13:11:52.530387   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:52.523293  1255 raft_consensus.cc:359] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.532442  1255 raft_consensus.cc:385] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.532708  1255 raft_consensus.cc:740] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.534619  1263 leader_election.cc:290] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:52.534865  1255 consensus_queue.cc:260] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.540259  1255 ts_tablet_manager.cc:1434] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.020s	user 0.003s	sys 0.003s
I20260709 13:11:52.539822   882 catalog_manager.cc:5697] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 0 to 1, leader changed from <none> to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:52.541234  1255 tablet_bootstrap.cc:492] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:52.541939   882 catalog_manager.cc:5697] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 0 to 1, leader changed from <none> to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:52.549338  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:52.550654  1135 raft_consensus.cc:2393] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 1: Already voted for candidate 294145900dd3496cb4a1d722613e1558 in this term.
I20260709 13:11:52.552157  1025 leader_election.cc:304] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:52.553424  1255 tablet_bootstrap.cc:654] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.554329  1263 raft_consensus.cc:3060] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.561883  1266 raft_consensus.cc:493] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.562526  1266 raft_consensus.cc:515] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.564821  1254 tablet_bootstrap.cc:492] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:52.564939  1266 leader_election.cc:290] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:52.567792  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
W20260709 13:11:52.569542  1099 leader_election.cc:343] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 862b41697c74421fbac82c10c245434d (127.23.140.68:34477): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:52.570482  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:11:52.571954  1099 leader_election.cc:343] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:52.572492  1099 leader_election.cc:304] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:52.573516  1266 raft_consensus.cc:2749] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:52.574069   881 catalog_manager.cc:5697] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:52.578091  1255 tablet_bootstrap.cc:492] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:52.578645  1255 ts_tablet_manager.cc:1403] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.038s	user 0.008s	sys 0.004s
I20260709 13:11:52.583138  1286 consensus_queue.cc:1048] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:52.596452  1349 raft_consensus.cc:493] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.597225  1349 raft_consensus.cc:515] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.599074  1286 raft_consensus.cc:493] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.599658  1286 raft_consensus.cc:515] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.599773  1255 raft_consensus.cc:359] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.600818  1255 raft_consensus.cc:385] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.601217  1255 raft_consensus.cc:740] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.601836  1349 leader_election.cc:290] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:52.602515  1286 leader_election.cc:290] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:52.603849  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:52.602870  1255 consensus_queue.cc:260] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
W20260709 13:11:52.605809  1025 leader_election.cc:343] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:52.607769  1255 ts_tablet_manager.cc:1434] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.029s	user 0.004s	sys 0.003s
I20260709 13:11:52.608996  1255 tablet_bootstrap.cc:492] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:52.610790  1263 raft_consensus.cc:2749] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:52.612933  1257 tablet_bootstrap.cc:492] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:52.613559  1257 ts_tablet_manager.cc:1403] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.172s	user 0.041s	sys 0.041s
I20260709 13:11:52.614756  1254 tablet_bootstrap.cc:654] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.617584  1257 raft_consensus.cc:359] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.618460  1257 raft_consensus.cc:385] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.618763  1257 raft_consensus.cc:740] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.619546  1257 consensus_queue.cc:260] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.623905  1257 ts_tablet_manager.cc:1434] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.001s
I20260709 13:11:52.625217  1256 tablet_bootstrap.cc:492] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:52.625257  1257 tablet_bootstrap.cc:492] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:52.625860  1256 ts_tablet_manager.cc:1403] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.117s	user 0.027s	sys 0.019s
I20260709 13:11:52.626458  1255 tablet_bootstrap.cc:654] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.627764  1254 tablet_bootstrap.cc:492] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:52.628268  1254 ts_tablet_manager.cc:1403] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.064s	user 0.014s	sys 0.005s
I20260709 13:11:52.669603  1286 raft_consensus.cc:493] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.670069   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:52.671211  1286 raft_consensus.cc:515] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.671815   986 raft_consensus.cc:2468] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:52.674750   987 raft_consensus.cc:3060] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.678234  1349 raft_consensus.cc:493] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.678706  1027 leader_election.cc:304] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:11:52.679739  1307 raft_consensus.cc:493] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.679430  1349 raft_consensus.cc:515] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.680895  1307 raft_consensus.cc:515] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.686993  1317 raft_consensus.cc:493] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.629561  1256 raft_consensus.cc:359] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.688047  1256 raft_consensus.cc:385] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.688663  1256 raft_consensus.cc:740] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.684726  1263 raft_consensus.cc:2804] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:52.689378  1263 raft_consensus.cc:493] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:52.688064  1317 raft_consensus.cc:515] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.696280  1317 leader_election.cc:290] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:52.667985   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dde48ef51c4666a28b97181ec2ee21" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:52.699261  1208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dde48ef51c4666a28b97181ec2ee21" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:52.700173   985 raft_consensus.cc:2468] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 0.
I20260709 13:11:52.700314  1208 raft_consensus.cc:2468] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:52.691903  1263 raft_consensus.cc:3060] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.707173  1349 leader_election.cc:290] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:52.708109  1099 leader_election.cc:304] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:52.708297  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dde48ef51c4666a28b97181ec2ee21" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:52.709313  1135 raft_consensus.cc:2468] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 0.
I20260709 13:11:52.709862  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:52.710824  1136 raft_consensus.cc:2468] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 0.
I20260709 13:11:52.711174  1307 leader_election.cc:290] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:52.712518  1256 consensus_queue.cc:260] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.713425  1291 raft_consensus.cc:2804] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:52.713891  1291 raft_consensus.cc:493] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:52.718878  1291 raft_consensus.cc:3060] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.719487  1175 leader_election.cc:304] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 
I20260709 13:11:52.722748  1349 raft_consensus.cc:2804] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:52.723266  1349 raft_consensus.cc:493] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:52.723683  1349 raft_consensus.cc:3060] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.726549  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:52.727273  1136 raft_consensus.cc:2468] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:52.728389  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:52.729189  1207 raft_consensus.cc:2468] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:52.732630  1173 leader_election.cc:304] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:52.736452   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dde48ef51c4666a28b97181ec2ee21" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:52.737282   985 raft_consensus.cc:2468] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:52.738833   950 leader_election.cc:304] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 
I20260709 13:11:52.742480  1293 raft_consensus.cc:2804] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:52.742961  1293 raft_consensus.cc:493] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:52.743301  1293 raft_consensus.cc:3060] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.743863  1313 raft_consensus.cc:2804] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:52.744293  1313 raft_consensus.cc:493] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:52.744637  1313 raft_consensus.cc:3060] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.737118  1348 raft_consensus.cc:493] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.745570  1348 raft_consensus.cc:515] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.737246   984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:52.748186  1286 leader_election.cc:290] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:52.748811  1286 consensus_queue.cc:1048] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:52.752257  1254 raft_consensus.cc:359] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.753149  1254 raft_consensus.cc:385] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.753443  1254 raft_consensus.cc:740] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.754578  1254 consensus_queue.cc:260] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.759011  1348 leader_election.cc:290] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:52.761757  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "240eb8c24e0142c8a23a0afdd5dccda6" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:52.762168  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "240eb8c24e0142c8a23a0afdd5dccda6" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:52.763242  1062 raft_consensus.cc:2393] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 1: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:11:52.766141  1173 leader_election.cc:304] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:52.769893  1348 raft_consensus.cc:3060] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.772521  1255 tablet_bootstrap.cc:492] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:52.773998  1255 ts_tablet_manager.cc:1403] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.165s	user 0.008s	sys 0.006s
I20260709 13:11:52.775722  1254 ts_tablet_manager.cc:1434] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.147s	user 0.008s	sys 0.004s
I20260709 13:11:52.777428  1254 tablet_bootstrap.cc:492] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
W20260709 13:11:52.736801  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 53479.4us
I20260709 13:11:52.786270  1208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:52.787281  1208 raft_consensus.cc:2393] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 1: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:11:52.782572  1348 raft_consensus.cc:2749] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:52.786032  1255 raft_consensus.cc:359] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.790726  1255 raft_consensus.cc:385] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.791047  1255 raft_consensus.cc:740] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.791785  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:52.796015  1293 raft_consensus.cc:515] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.803158  1293 leader_election.cc:290] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:52.802659  1291 raft_consensus.cc:515] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.805941  1291 leader_election.cc:290] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:52.805927  1255 consensus_queue.cc:260] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.806836   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:52.807142   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dde48ef51c4666a28b97181ec2ee21" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:52.807826   986 raft_consensus.cc:3060] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.809116  1025 leader_election.cc:304] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:52.795933  1288 consensus_queue.cc:1048] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260709 13:11:52.811357  1367 raft_consensus.cc:3060] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.807758  1255 ts_tablet_manager.cc:1434] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.033s	user 0.006s	sys 0.001s
I20260709 13:11:52.813336  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:52.821970  1136 raft_consensus.cc:3060] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.812551  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dde48ef51c4666a28b97181ec2ee21" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:52.825459  1256 ts_tablet_manager.cc:1434] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.199s	user 0.015s	sys 0.022s
I20260709 13:11:52.827142  1256 tablet_bootstrap.cc:492] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:52.813628  1255 tablet_bootstrap.cc:492] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:52.816983  1257 tablet_bootstrap.cc:654] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.832588  1254 tablet_bootstrap.cc:654] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.843255  1313 raft_consensus.cc:515] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.848129  1256 tablet_bootstrap.cc:654] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.848626  1307 consensus_queue.cc:1048] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:52.851171  1348 raft_consensus.cc:493] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.855010  1348 raft_consensus.cc:515] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.857412  1348 leader_election.cc:290] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:52.864401   983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:52.858129  1288 consensus_queue.cc:1048] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
W20260709 13:11:52.868258  1175 leader_election.cc:343] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:52.875661   984 raft_consensus.cc:2393] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 1: Already voted for candidate 294145900dd3496cb4a1d722613e1558 in this term.
I20260709 13:11:52.876722  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:11:52.878851  1173 leader_election.cc:343] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:52.879930  1173 leader_election.cc:304] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:52.885003  1293 raft_consensus.cc:2749] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:52.886142  1366 raft_consensus.cc:493] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.886817  1366 raft_consensus.cc:515] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:52.891460  1366 leader_election.cc:290] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:52.862847  1263 raft_consensus.cc:515] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.906455  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2312d768cbf4b3495d5b1574b0b6eb5" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:52.907265  1255 tablet_bootstrap.cc:654] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:52.907840  1368 raft_consensus.cc:493] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.908460   984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:52.908371  1368 raft_consensus.cc:515] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:52.909080   984 raft_consensus.cc:3060] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.910686  1368 leader_election.cc:290] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:52.913625  1263 leader_election.cc:290] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:52.914134  1317 consensus_queue.cc:1048] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:52.916074  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:52.919263  1313 leader_election.cc:290] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:52.920648  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2312d768cbf4b3495d5b1574b0b6eb5" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:52.921046  1367 raft_consensus.cc:2749] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:52.922125  1136 raft_consensus.cc:2468] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:11:52.922892  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f95cbeb7360c4c61b442e75e35d93e73" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:52.923617  1208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:52.923540  1173 leader_election.cc:304] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:11:52.925673  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f95cbeb7360c4c61b442e75e35d93e73" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:52.927660  1208 raft_consensus.cc:2393] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 in current term 1: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
I20260709 13:11:52.927850  1209 raft_consensus.cc:2393] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 1: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:11:52.928092  1293 raft_consensus.cc:2804] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:52.926829   950 leader_election.cc:304] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:52.928862  1293 raft_consensus.cc:697] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:11:52.929706   950 leader_election.cc:304] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:52.930182  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:52.930569  1307 raft_consensus.cc:2749] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:11:52.931386  1307 raft_consensus.cc:2749] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:11:52.930399  1293 consensus_queue.cc:237] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.933568  1254 tablet_bootstrap.cc:492] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:52.934082  1254 ts_tablet_manager.cc:1403] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.157s	user 0.012s	sys 0.007s
I20260709 13:11:52.935056   986 raft_consensus.cc:2468] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:52.936352  1101 leader_election.cc:304] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:52.937273  1291 raft_consensus.cc:2804] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:52.937049  1254 raft_consensus.cc:359] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.937839  1291 raft_consensus.cc:697] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:11:52.940553  1291 consensus_queue.cc:237] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.903098  1349 raft_consensus.cc:515] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.942384  1207 raft_consensus.cc:2393] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 1: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
I20260709 13:11:52.943960  1349 leader_election.cc:290] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:52.945003  1254 raft_consensus.cc:385] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.945384  1254 raft_consensus.cc:740] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.954746  1257 tablet_bootstrap.cc:492] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:52.955538  1257 ts_tablet_manager.cc:1403] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.331s	user 0.015s	sys 0.026s
I20260709 13:11:52.963510   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dde48ef51c4666a28b97181ec2ee21" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:52.966279   987 raft_consensus.cc:2393] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 1: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:11:52.966192  1025 leader_election.cc:304] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:52.967039  1257 raft_consensus.cc:359] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.968035  1257 raft_consensus.cc:385] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:52.968364  1257 raft_consensus.cc:740] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.968672  1263 raft_consensus.cc:3060] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:52.969333  1257 consensus_queue.cc:260] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:52.970959   984 raft_consensus.cc:2468] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:52.972931  1027 leader_election.cc:304] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:52.973855  1367 raft_consensus.cc:2804] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:52.974658  1367 raft_consensus.cc:697] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:11:52.975548  1254 consensus_queue.cc:260] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.977188  1257 ts_tablet_manager.cc:1434] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.021s	user 0.009s	sys 0.003s
I20260709 13:11:52.978852  1257 tablet_bootstrap.cc:492] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:52.978855  1367 consensus_queue.cc:237] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.986662  1266 raft_consensus.cc:493] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:52.986900  1317 consensus_queue.cc:1048] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:52.988150  1266 raft_consensus.cc:515] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:52.990666  1266 leader_election.cc:290] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:52.991624  1255 tablet_bootstrap.cc:492] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:52.992302  1255 ts_tablet_manager.cc:1403] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.179s	user 0.023s	sys 0.020s
I20260709 13:11:52.993163  1256 tablet_bootstrap.cc:492] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:52.993834  1256 ts_tablet_manager.cc:1403] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.167s	user 0.016s	sys 0.009s
I20260709 13:11:52.996158  1307 consensus_queue.cc:1048] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:52.999953  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:53.001071   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:53.003983  1256 raft_consensus.cc:359] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:53.005424  1256 raft_consensus.cc:385] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.005798  1256 raft_consensus.cc:740] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:52.985821  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dde48ef51c4666a28b97181ec2ee21" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:53.016969  1266 consensus_queue.cc:1048] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:53.017024  1256 consensus_queue.cc:260] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:53.021661  1256 ts_tablet_manager.cc:1434] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.003s
W20260709 13:11:53.021703  1101 leader_election.cc:343] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:53.023190  1256 tablet_bootstrap.cc:492] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
W20260709 13:11:53.005035  1025 leader_election.cc:343] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:53.032768  1255 raft_consensus.cc:359] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:53.038931  1255 raft_consensus.cc:385] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.039252  1255 raft_consensus.cc:740] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.047227  1254 ts_tablet_manager.cc:1434] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.113s	user 0.010s	sys 0.000s
I20260709 13:11:53.050333  1366 consensus_queue.cc:1048] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:11:53.047695  1099 leader_election.cc:343] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:53.052259  1099 leader_election.cc:304] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:53.052685  1255 consensus_queue.cc:260] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:53.055316  1255 ts_tablet_manager.cc:1434] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.063s	user 0.006s	sys 0.003s
I20260709 13:11:53.056881  1255 tablet_bootstrap.cc:492] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:53.057888  1256 tablet_bootstrap.cc:654] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.058210  1263 raft_consensus.cc:2749] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:53.058352  1266 raft_consensus.cc:2749] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:53.067932  1266 consensus_queue.cc:1048] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:53.059064  1317 raft_consensus.cc:493] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:53.072227  1257 tablet_bootstrap.cc:654] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.071786  1317 raft_consensus.cc:515] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:53.076546  1317 leader_election.cc:290] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:53.081882  1254 tablet_bootstrap.cc:492] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:53.083750   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:53.084714   985 raft_consensus.cc:2393] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 1: Already voted for candidate 9c45b6d917734835a8a37b0f4c789ae6 in this term.
I20260709 13:11:53.087201   880 catalog_manager.cc:5697] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:53.087641  1255 tablet_bootstrap.cc:654] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.111970  1173 leader_election.cc:304] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:53.113003  1293 raft_consensus.cc:2749] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:11:53.120627  1256 tablet_bootstrap.cc:492] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:53.121274  1256 ts_tablet_manager.cc:1403] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.098s	user 0.015s	sys 0.009s
I20260709 13:11:53.121600  1254 tablet_bootstrap.cc:654] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.131412  1266 consensus_queue.cc:1048] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:53.134756  1307 consensus_queue.cc:1048] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:53.135042  1266 consensus_queue.cc:1048] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:53.164911  1255 tablet_bootstrap.cc:492] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:53.165746  1255 ts_tablet_manager.cc:1403] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.109s	user 0.027s	sys 0.003s
I20260709 13:11:53.148850  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:53.179564  1293 raft_consensus.cc:493] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:53.180310  1293 raft_consensus.cc:515] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.181813  1255 raft_consensus.cc:359] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.182545  1255 raft_consensus.cc:385] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.182811  1255 raft_consensus.cc:740] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.183636  1255 consensus_queue.cc:260] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.185349  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11cd8a46e76f4bc685aa01885b27cc57" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:53.186951  1255 ts_tablet_manager.cc:1434] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 13:11:53.186118  1293 leader_election.cc:290] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:53.188165  1255 tablet_bootstrap.cc:492] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:53.187981   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11cd8a46e76f4bc685aa01885b27cc57" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:53.191735   985 raft_consensus.cc:2468] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 0.
I20260709 13:11:53.196127  1256 raft_consensus.cc:359] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:53.197465  1256 raft_consensus.cc:385] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.197661  1256 raft_consensus.cc:740] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
W20260709 13:11:53.188673  1173 leader_election.cc:343] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:53.212481  1263 raft_consensus.cc:493] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:53.213215  1263 raft_consensus.cc:515] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:53.214946  1099 leader_election.cc:304] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:53.215252  1175 leader_election.cc:304] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:53.216665  1293 raft_consensus.cc:2804] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:53.218693  1293 raft_consensus.cc:493] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:53.219244  1293 raft_consensus.cc:3060] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.217257   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:53.223688  1256 consensus_queue.cc:260] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:53.226540   985 raft_consensus.cc:2468] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:53.229838  1027 leader_election.cc:304] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:53.230454  1266 raft_consensus.cc:3060] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.230768  1367 raft_consensus.cc:2804] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:53.231199  1367 raft_consensus.cc:493] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:53.228840  1313 raft_consensus.cc:493] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:53.231565  1367 raft_consensus.cc:3060] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.232208  1313 raft_consensus.cc:515] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.233523  1263 leader_election.cc:290] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:53.238413  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:53.234689  1313 leader_election.cc:290] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:53.236639  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:53.243359  1210 raft_consensus.cc:2468] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:53.246838  1256 ts_tablet_manager.cc:1434] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.125s	user 0.008s	sys 0.001s
I20260709 13:11:53.254140  1254 tablet_bootstrap.cc:492] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:53.254662  1254 ts_tablet_manager.cc:1403] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.174s	user 0.017s	sys 0.002s
I20260709 13:11:53.245194   950 leader_election.cc:304] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 
W20260709 13:11:53.256341   950 leader_election.cc:343] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:53.256363  1313 raft_consensus.cc:2804] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:53.256978  1313 raft_consensus.cc:493] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:53.248793  1349 raft_consensus.cc:493] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:53.257473  1313 raft_consensus.cc:3060] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.257633  1349 raft_consensus.cc:515] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:53.258265  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:53.258925  1135 raft_consensus.cc:2468] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:53.260264  1349 leader_election.cc:290] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:53.261533  1266 raft_consensus.cc:2749] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:53.261438   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:53.257726  1254 raft_consensus.cc:359] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:53.265585  1256 tablet_bootstrap.cc:492] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:53.268312  1367 raft_consensus.cc:515] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:53.271323  1257 tablet_bootstrap.cc:492] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:53.271919  1257 ts_tablet_manager.cc:1403] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.293s	user 0.011s	sys 0.008s
I20260709 13:11:53.273283  1367 leader_election.cc:290] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:53.270362  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:53.274066  1256 tablet_bootstrap.cc:654] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.274155   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:53.274902   985 raft_consensus.cc:3060] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.264346  1254 raft_consensus.cc:385] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.275635  1254 raft_consensus.cc:740] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.276852  1255 tablet_bootstrap.cc:654] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.276695  1254 consensus_queue.cc:260] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:53.279461  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:53.279754  1254 ts_tablet_manager.cc:1434] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.000s
W20260709 13:11:53.265049  1175 leader_election.cc:343] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:11:53.281082  1173 leader_election.cc:343] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:53.282665  1257 raft_consensus.cc:359] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:53.283658  1257 raft_consensus.cc:385] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.283999  1257 raft_consensus.cc:740] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.287449  1257 consensus_queue.cc:260] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:53.289919  1257 ts_tablet_manager.cc:1434] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.018s	user 0.009s	sys 0.001s
I20260709 13:11:53.291162  1257 tablet_bootstrap.cc:492] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:53.294756  1173 leader_election.cc:304] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:53.296142  1348 raft_consensus.cc:2749] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:53.284658  1135 raft_consensus.cc:3060] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.303206  1257 tablet_bootstrap.cc:654] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.304564   985 raft_consensus.cc:2468] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:53.284821  1293 raft_consensus.cc:515] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.308105  1254 tablet_bootstrap.cc:492] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:53.308847  1027 leader_election.cc:304] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:53.311723  1367 raft_consensus.cc:2804] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:53.312273  1367 raft_consensus.cc:697] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:11:53.314996   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11cd8a46e76f4bc685aa01885b27cc57" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:53.315646   985 raft_consensus.cc:3060] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.316944  1293 leader_election.cc:290] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:53.318378  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11cd8a46e76f4bc685aa01885b27cc57" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:53.322680  1255 tablet_bootstrap.cc:492] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:53.323410  1254 tablet_bootstrap.cc:654] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.324236   885 catalog_manager.cc:5697] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:53.325883   985 raft_consensus.cc:2468] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:11:53.326354  1263 raft_consensus.cc:493] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:53.340837  1263 raft_consensus.cc:515] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:53.343062  1263 leader_election.cc:290] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:53.326534  1367 consensus_queue.cc:237] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:53.327093   882 catalog_manager.cc:5697] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 0 to 1, leader changed from <none> to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 1 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:53.325083  1135 raft_consensus.cc:2468] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:53.328883  1255 ts_tablet_manager.cc:1403] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.141s	user 0.011s	sys 0.005s
W20260709 13:11:53.358153   932 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260709 13:11:53.329896  1173 leader_election.cc:343] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:53.362326  1255 raft_consensus.cc:359] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:53.363049  1255 raft_consensus.cc:385] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.363093  1175 leader_election.cc:304] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:53.330510  1257 tablet_bootstrap.cc:492] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:53.365459  1257 ts_tablet_manager.cc:1403] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.075s	user 0.014s	sys 0.015s
I20260709 13:11:53.366125  1293 raft_consensus.cc:2804] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:53.366966  1293 raft_consensus.cc:697] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:11:53.368258  1257 raft_consensus.cc:359] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:53.369544  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:53.335551  1401 consensus_queue.cc:1048] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.034s
I20260709 13:11:53.368966  1257 raft_consensus.cc:385] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.372530  1257 raft_consensus.cc:740] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.373274  1257 consensus_queue.cc:260] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:53.363330  1255 raft_consensus.cc:740] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.339071  1313 raft_consensus.cc:515] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.375691  1255 consensus_queue.cc:260] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:53.376998  1135 raft_consensus.cc:2468] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:53.377825  1255 ts_tablet_manager.cc:1434] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.000s
I20260709 13:11:53.378995  1255 tablet_bootstrap.cc:492] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:53.381214  1025 leader_election.cc:304] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:53.381910  1257 ts_tablet_manager.cc:1434] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20260709 13:11:53.389515  1257 tablet_bootstrap.cc:492] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:53.393321  1313 leader_election.cc:290] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:53.395221  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:53.395897  1210 raft_consensus.cc:3060] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.398808  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:53.399577  1255 tablet_bootstrap.cc:654] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.399688  1207 raft_consensus.cc:2468] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:53.404304  1210 raft_consensus.cc:2468] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:53.407058  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:53.407141   885 catalog_manager.cc:5697] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
W20260709 13:11:53.409081   950 leader_election.cc:343] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:53.368465  1293 consensus_queue.cc:237] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.413316  1263 raft_consensus.cc:2804] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:53.413880  1263 raft_consensus.cc:493] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:53.414354  1263 raft_consensus.cc:3060] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.395529  1257 tablet_bootstrap.cc:654] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.429550   950 leader_election.cc:304] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:53.430929  1340 raft_consensus.cc:2804] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:53.431443  1340 raft_consensus.cc:697] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:11:53.432288  1340 consensus_queue.cc:237] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.469336  1255 tablet_bootstrap.cc:492] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:53.470116  1255 ts_tablet_manager.cc:1403] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.091s	user 0.014s	sys 0.021s
I20260709 13:11:53.474053  1254 tablet_bootstrap.cc:492] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:53.474560  1254 ts_tablet_manager.cc:1403] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.167s	user 0.011s	sys 0.029s
I20260709 13:11:53.478101  1257 tablet_bootstrap.cc:492] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:53.480186  1256 tablet_bootstrap.cc:492] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:53.481617  1256 ts_tablet_manager.cc:1403] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.232s	user 0.039s	sys 0.084s
I20260709 13:11:53.484668  1256 raft_consensus.cc:359] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.485469  1256 raft_consensus.cc:385] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.486363  1257 ts_tablet_manager.cc:1403] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.097s	user 0.014s	sys 0.005s
I20260709 13:11:53.485742  1256 raft_consensus.cc:740] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.489030  1256 consensus_queue.cc:260] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.489733  1401 consensus_queue.cc:1048] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:53.494607  1256 ts_tablet_manager.cc:1434] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 13:11:53.495832  1256 tablet_bootstrap.cc:492] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:53.501945  1365 consensus_queue.cc:1048] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:53.502665  1256 tablet_bootstrap.cc:654] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.513651  1254 raft_consensus.cc:359] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:53.514258  1254 raft_consensus.cc:385] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.517004  1263 raft_consensus.cc:515] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:53.517371  1257 raft_consensus.cc:359] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:53.514509  1254 raft_consensus.cc:740] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.518184  1257 raft_consensus.cc:385] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.518872  1254 consensus_queue.cc:260] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:53.519062   882 catalog_manager.cc:5697] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 0 to 1, leader changed from <none> to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 1 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:53.527347  1254 ts_tablet_manager.cc:1434] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.053s	user 0.005s	sys 0.001s
I20260709 13:11:53.528656  1254 tablet_bootstrap.cc:492] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:53.520900  1257 raft_consensus.cc:740] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.534579  1293 consensus_queue.cc:1048] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:53.535817  1257 consensus_queue.cc:260] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:53.536141  1288 consensus_queue.cc:1048] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:53.538412  1257 ts_tablet_manager.cc:1434] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.052s	user 0.006s	sys 0.000s
I20260709 13:11:53.538992  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:53.539654  1257 tablet_bootstrap.cc:492] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:53.539665  1209 raft_consensus.cc:3060] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.541356  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:53.541502  1255 raft_consensus.cc:359] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:53.541963  1136 raft_consensus.cc:3060] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.542248  1255 raft_consensus.cc:385] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.542593  1255 raft_consensus.cc:740] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.543304  1255 consensus_queue.cc:260] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:53.546545  1257 tablet_bootstrap.cc:654] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.547611  1293 consensus_queue.cc:1048] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:53.550865  1254 tablet_bootstrap.cc:654] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.560281  1256 tablet_bootstrap.cc:492] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:53.559118   879 catalog_manager.cc:5697] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 0 to 1, leader changed from <none> to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:53.561769  1291 raft_consensus.cc:493] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:53.562503  1291 raft_consensus.cc:515] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:53.561039  1256 ts_tablet_manager.cc:1403] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.066s	user 0.013s	sys 0.006s
I20260709 13:11:53.569798   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:53.572283   985 raft_consensus.cc:2468] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:53.570902  1291 leader_election.cc:290] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:53.572822  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:53.574074  1101 leader_election.cc:304] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
W20260709 13:11:53.574705  1099 leader_election.cc:343] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:53.575255  1291 raft_consensus.cc:2804] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:53.575737  1291 raft_consensus.cc:493] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:53.576706  1291 raft_consensus.cc:3060] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.581851  1257 tablet_bootstrap.cc:492] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:53.578208  1256 raft_consensus.cc:359] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.582396  1257 ts_tablet_manager.cc:1403] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.043s	user 0.031s	sys 0.000s
I20260709 13:11:53.582432  1256 raft_consensus.cc:385] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.582959  1256 raft_consensus.cc:740] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.551388  1209 raft_consensus.cc:2468] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:53.553922  1263 leader_election.cc:290] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:53.591418  1257 raft_consensus.cc:359] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.592423  1257 raft_consensus.cc:385] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.592713  1257 raft_consensus.cc:740] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.593331  1025 leader_election.cc:304] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:53.593503  1257 consensus_queue.cc:260] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.594229  1263 raft_consensus.cc:2804] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:53.594693  1263 raft_consensus.cc:697] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:11:53.595571  1257 ts_tablet_manager.cc:1434] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 13:11:53.595490  1263 consensus_queue.cc:237] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:53.596614  1257 tablet_bootstrap.cc:492] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:53.598187  1255 ts_tablet_manager.cc:1434] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.128s	user 0.007s	sys 0.001s
I20260709 13:11:53.583880  1313 raft_consensus.cc:493] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:53.588322  1256 consensus_queue.cc:260] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.604202  1256 ts_tablet_manager.cc:1434] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.037s	user 0.007s	sys 0.000s
I20260709 13:11:53.605623  1256 tablet_bootstrap.cc:492] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:53.605747  1313 raft_consensus.cc:515] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:53.607434  1313 leader_election.cc:290] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:53.609948  1255 tablet_bootstrap.cc:492] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:53.614429  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:53.617913  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
W20260709 13:11:53.618415   950 leader_election.cc:343] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:53.627442  1288 raft_consensus.cc:493] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:53.635416  1288 raft_consensus.cc:515] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.638404  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:53.640334  1209 raft_consensus.cc:2393] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 1: Already voted for candidate 294145900dd3496cb4a1d722613e1558 in this term.
I20260709 13:11:53.640946  1288 leader_election.cc:290] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:53.644466   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:53.650314  1025 leader_election.cc:304] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:11:53.653834  1255 tablet_bootstrap.cc:654] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.654277  1288 raft_consensus.cc:3060] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.655421  1254 tablet_bootstrap.cc:492] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:53.656090  1254 ts_tablet_manager.cc:1403] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.128s	user 0.017s	sys 0.005s
I20260709 13:11:53.657853  1136 raft_consensus.cc:2468] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:53.660785  1254 raft_consensus.cc:359] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:53.661552  1254 raft_consensus.cc:385] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.661818  1254 raft_consensus.cc:740] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.662587  1254 consensus_queue.cc:260] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:53.666700  1254 ts_tablet_manager.cc:1434] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 13:11:53.667764  1254 tablet_bootstrap.cc:492] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:53.671720  1291 raft_consensus.cc:515] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:53.674026  1254 tablet_bootstrap.cc:654] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.679176  1288 raft_consensus.cc:2749] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:53.686702  1256 tablet_bootstrap.cc:654] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.689018  1133 raft_consensus.cc:2393] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 in current term 1: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:11:53.692685   950 leader_election.cc:304] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:53.692489   882 catalog_manager.cc:5697] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:53.693925  1313 raft_consensus.cc:3060] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.698038  1291 leader_election.cc:290] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:53.698889  1257 tablet_bootstrap.cc:654] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.706000  1293 raft_consensus.cc:493] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:53.706516  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:53.706722  1293 raft_consensus.cc:515] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:53.709348  1293 leader_election.cc:290] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:53.709662  1366 raft_consensus.cc:493] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:53.710175  1366 raft_consensus.cc:515] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:53.713224  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:53.718214  1255 tablet_bootstrap.cc:492] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:53.718685  1255 ts_tablet_manager.cc:1403] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.109s	user 0.005s	sys 0.009s
W20260709 13:11:53.709374  1099 leader_election.cc:343] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:53.727562  1255 raft_consensus.cc:359] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.728276  1255 raft_consensus.cc:385] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.721884  1366 leader_election.cc:290] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
W20260709 13:11:53.717628  1173 leader_election.cc:343] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:53.730506  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b89f56a91e004f408b36c36d7aee0076" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:53.731211  1133 raft_consensus.cc:2468] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:53.724298  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b89f56a91e004f408b36c36d7aee0076" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:53.732456   984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:53.732475  1207 raft_consensus.cc:2468] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:53.744035  1257 tablet_bootstrap.cc:492] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:53.728542  1255 raft_consensus.cc:740] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.744573  1257 ts_tablet_manager.cc:1403] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.148s	user 0.022s	sys 0.006s
I20260709 13:11:53.739353  1313 raft_consensus.cc:2749] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:53.740425  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:53.745191  1255 consensus_queue.cc:260] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.745652  1134 raft_consensus.cc:2468] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 0.
I20260709 13:11:53.741583   950 leader_election.cc:304] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:53.748018  1340 raft_consensus.cc:2804] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:53.747943  1257 raft_consensus.cc:359] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.748536  1340 raft_consensus.cc:493] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:53.748769  1257 raft_consensus.cc:385] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.748940  1340 raft_consensus.cc:3060] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.749549  1257 raft_consensus.cc:740] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.750708  1255 ts_tablet_manager.cc:1434] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.000s
I20260709 13:11:53.750352  1257 consensus_queue.cc:260] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.751792  1255 tablet_bootstrap.cc:492] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:53.752508  1257 ts_tablet_manager.cc:1434] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 13:11:53.754470  1173 leader_election.cc:304] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:53.755656  1293 raft_consensus.cc:2804] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:53.756022  1293 raft_consensus.cc:493] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:53.756341  1293 raft_consensus.cc:3060] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.759886  1257 tablet_bootstrap.cc:492] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:53.761936  1254 tablet_bootstrap.cc:492] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:53.762521  1254 ts_tablet_manager.cc:1403] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.095s	user 0.015s	sys 0.013s
I20260709 13:11:53.765998  1254 raft_consensus.cc:359] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:53.766806  1254 raft_consensus.cc:385] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.767138  1254 raft_consensus.cc:740] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.767979  1254 consensus_queue.cc:260] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:53.769384  1257 tablet_bootstrap.cc:654] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.773321  1256 tablet_bootstrap.cc:492] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:53.773836  1256 ts_tablet_manager.cc:1403] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.169s	user 0.016s	sys 0.016s
I20260709 13:11:53.760115  1340 raft_consensus.cc:515] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:53.760944  1255 tablet_bootstrap.cc:654] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.783149  1254 ts_tablet_manager.cc:1434] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.000s
I20260709 13:11:53.783008  1293 raft_consensus.cc:515] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:53.786129  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
W20260709 13:11:53.787724  1173 leader_election.cc:343] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:53.789412  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b89f56a91e004f408b36c36d7aee0076" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:53.789685  1340 leader_election.cc:290] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:53.790122  1134 raft_consensus.cc:3060] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.790057  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b89f56a91e004f408b36c36d7aee0076" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:53.791189  1209 raft_consensus.cc:3060] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.791226  1256 raft_consensus.cc:359] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:53.792186  1256 raft_consensus.cc:385] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.792465  1256 raft_consensus.cc:740] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.797739  1256 consensus_queue.cc:260] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:53.800201  1255 tablet_bootstrap.cc:492] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:53.800771  1255 ts_tablet_manager.cc:1403] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.049s	user 0.015s	sys 0.001s
I20260709 13:11:53.803738  1255 raft_consensus.cc:359] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:53.804456  1255 raft_consensus.cc:385] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.804728  1255 raft_consensus.cc:740] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.809839  1134 raft_consensus.cc:2468] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:53.810936   950 leader_election.cc:304] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:53.811873  1313 raft_consensus.cc:2804] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:53.812580  1313 raft_consensus.cc:697] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:11:53.813967  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:53.813714  1313 consensus_queue.cc:237] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:53.814612  1133 raft_consensus.cc:3060] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.818192  1256 ts_tablet_manager.cc:1434] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.044s	user 0.007s	sys 0.000s
I20260709 13:11:53.811404  1254 tablet_bootstrap.cc:492] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:53.822994  1133 raft_consensus.cc:2468] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:11:53.824205  1173 leader_election.cc:304] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:53.825140  1349 raft_consensus.cc:2804] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:53.825058  1401 raft_consensus.cc:493] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:53.825934  1349 raft_consensus.cc:697] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:11:53.826061  1401 raft_consensus.cc:515] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:53.826856  1349 consensus_queue.cc:237] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:53.829885  1256 tablet_bootstrap.cc:492] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:53.830899   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:53.832044  1401 leader_election.cc:290] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:53.832914  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:53.833900  1207 raft_consensus.cc:2468] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:53.835623  1099 leader_election.cc:304] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:53.835470  1255 consensus_queue.cc:260] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:53.837237  1293 leader_election.cc:290] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:53.837970  1255 ts_tablet_manager.cc:1434] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.037s	user 0.008s	sys 0.000s
I20260709 13:11:53.838753   985 raft_consensus.cc:2468] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:53.838246   984 raft_consensus.cc:2468] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:53.840044  1255 tablet_bootstrap.cc:492] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:53.843362  1101 leader_election.cc:304] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:53.845213  1291 raft_consensus.cc:2804] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:53.845570  1256 tablet_bootstrap.cc:654] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.845757  1291 raft_consensus.cc:493] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:53.846328  1291 raft_consensus.cc:3060] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.846009  1349 raft_consensus.cc:493] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:53.850535  1263 consensus_queue.cc:1048] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:53.851267  1349 raft_consensus.cc:515] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.855427  1255 tablet_bootstrap.cc:654] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.857431  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:11:53.859000  1173 leader_election.cc:343] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:53.859674  1349 leader_election.cc:290] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:53.857488   984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:53.868847  1317 raft_consensus.cc:2804] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:53.869511  1317 raft_consensus.cc:697] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
W20260709 13:11:53.869788  1175 leader_election.cc:343] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:53.870286  1175 leader_election.cc:304] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:53.870711  1317 consensus_queue.cc:237] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:53.861493   879 catalog_manager.cc:5697] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 0 to 1, leader changed from <none> to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 1 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:53.873658  1340 raft_consensus.cc:493] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:53.874195  1340 raft_consensus.cc:515] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.875877  1288 consensus_queue.cc:1048] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:53.878942  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:53.879824  1207 raft_consensus.cc:2468] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:53.878968  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:53.881237   950 leader_election.cc:304] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 
I20260709 13:11:53.876247  1340 leader_election.cc:290] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:53.882208  1340 raft_consensus.cc:2804] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:53.882653  1340 raft_consensus.cc:493] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:11:53.882737   950 leader_election.cc:343] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:53.883188  1340 raft_consensus.cc:3060] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.886620  1348 raft_consensus.cc:2749] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:53.890882  1348 consensus_queue.cc:1048] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:53.824429  1209 raft_consensus.cc:2468] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:53.906364  1291 raft_consensus.cc:515] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:53.903329   879 catalog_manager.cc:5697] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 0 to 1, leader changed from <none> to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:53.913079  1254 tablet_bootstrap.cc:654] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:53.914700  1340 raft_consensus.cc:515] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.922766  1062 raft_consensus.cc:3060] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.923148  1340 leader_election.cc:290] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:53.927881  1291 leader_election.cc:290] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:53.930497  1257 tablet_bootstrap.cc:492] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:53.931030  1257 ts_tablet_manager.cc:1403] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.171s	user 0.041s	sys 0.083s
I20260709 13:11:53.934818  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:53.935415  1207 raft_consensus.cc:3060] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.935494   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:53.936134   985 raft_consensus.cc:3060] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.940265  1257 raft_consensus.cc:359] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:53.941552  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:53.943358  1207 raft_consensus.cc:2468] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:53.947713   985 raft_consensus.cc:2468] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:53.957300   880 catalog_manager.cc:5697] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:53.959281  1101 leader_election.cc:304] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:53.961175  1291 raft_consensus.cc:2804] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 13:11:53.959337   950 leader_election.cc:343] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:53.940989  1257 raft_consensus.cc:385] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:53.961746  1291 raft_consensus.cc:697] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:11:53.930919  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:53.962543  1257 raft_consensus.cc:740] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:53.965166  1257 consensus_queue.cc:260] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:53.966732  1209 raft_consensus.cc:3060] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:53.967384  1291 consensus_queue.cc:237] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:53.973785  1257 ts_tablet_manager.cc:1434] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.042s	user 0.008s	sys 0.000s
I20260709 13:11:53.977346  1256 tablet_bootstrap.cc:492] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:53.977926  1256 ts_tablet_manager.cc:1403] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.148s	user 0.016s	sys 0.013s
I20260709 13:11:53.979599  1255 tablet_bootstrap.cc:492] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:53.980050  1255 ts_tablet_manager.cc:1403] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.140s	user 0.023s	sys 0.003s
I20260709 13:11:53.980927  1257 tablet_bootstrap.cc:492] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:53.982472  1340 consensus_queue.cc:1048] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:11:53.974534  1432 log.cc:927] Time spent T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 13:11:53.992501  1288 raft_consensus.cc:493] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:53.993057  1288 raft_consensus.cc:515] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:53.995666  1254 tablet_bootstrap.cc:492] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:53.996418  1254 ts_tablet_manager.cc:1403] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.185s	user 0.009s	sys 0.020s
I20260709 13:11:53.999987  1307 consensus_queue.cc:1048] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:54.000547  1254 raft_consensus.cc:359] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.000800  1288 leader_election.cc:290] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:54.002519  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11cd8a46e76f4bc685aa01885b27cc57" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:54.008127  1348 consensus_queue.cc:1048] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:54.008354  1255 raft_consensus.cc:359] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.010596  1255 raft_consensus.cc:385] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.010659  1254 raft_consensus.cc:385] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.011219  1254 raft_consensus.cc:740] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.012212  1254 consensus_queue.cc:260] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.014829  1257 tablet_bootstrap.cc:654] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.015829  1254 ts_tablet_manager.cc:1434] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20260709 13:11:54.020959  1254 tablet_bootstrap.cc:492] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:54.022200   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11cd8a46e76f4bc685aa01885b27cc57" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:54.010916  1255 raft_consensus.cc:740] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.038623  1254 tablet_bootstrap.cc:654] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.038376  1256 raft_consensus.cc:359] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.044909  1027 leader_election.cc:304] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:11:54.039618  1256 raft_consensus.cc:385] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.046335  1256 raft_consensus.cc:740] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.035061  1255 consensus_queue.cc:260] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.056432  1288 raft_consensus.cc:2749] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:11:54.061051  1256 consensus_queue.cc:260] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.055450  1209 raft_consensus.cc:2468] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:54.066676  1255 ts_tablet_manager.cc:1434] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.086s	user 0.008s	sys 0.001s
I20260709 13:11:54.068100   950 leader_election.cc:304] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:54.069715  1340 raft_consensus.cc:2804] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:54.070245  1340 raft_consensus.cc:697] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:11:54.070986  1340 consensus_queue.cc:237] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.078598  1256 ts_tablet_manager.cc:1434] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.100s	user 0.009s	sys 0.000s
I20260709 13:11:54.079686  1255 tablet_bootstrap.cc:492] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:54.082508  1256 tablet_bootstrap.cc:492] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:54.087342  1257 tablet_bootstrap.cc:492] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:54.087831  1257 ts_tablet_manager.cc:1403] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.107s	user 0.021s	sys 0.024s
I20260709 13:11:54.088009  1254 tablet_bootstrap.cc:492] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:54.088553  1254 ts_tablet_manager.cc:1403] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.068s	user 0.015s	sys 0.008s
I20260709 13:11:54.089653  1256 tablet_bootstrap.cc:654] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.090438  1257 raft_consensus.cc:359] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.091197  1257 raft_consensus.cc:385] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.091477  1257 raft_consensus.cc:740] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.091354  1254 raft_consensus.cc:359] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:54.092234  1254 raft_consensus.cc:385] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.091687   882 catalog_manager.cc:5697] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:54.092545  1254 raft_consensus.cc:740] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.092481  1257 consensus_queue.cc:260] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.093741  1254 consensus_queue.cc:260] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:54.095006  1257 ts_tablet_manager.cc:1434] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260709 13:11:54.096122  1257 tablet_bootstrap.cc:492] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:54.105098  1254 ts_tablet_manager.cc:1434] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.016s	user 0.003s	sys 0.003s
I20260709 13:11:54.106462  1288 consensus_queue.cc:1048] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:54.106751  1254 tablet_bootstrap.cc:492] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:54.115373  1340 raft_consensus.cc:493] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:54.115688  1288 consensus_queue.cc:1048] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:54.115963  1340 raft_consensus.cc:515] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.118144  1340 leader_election.cc:290] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:54.119839  1257 tablet_bootstrap.cc:654] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.120199  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:54.121109  1207 raft_consensus.cc:2468] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:54.123075   950 leader_election.cc:304] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 
I20260709 13:11:54.124979  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:54.125689  1256 tablet_bootstrap.cc:492] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:54.126160  1256 ts_tablet_manager.cc:1403] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.044s	user 0.025s	sys 0.002s
W20260709 13:11:54.126866   950 leader_election.cc:343] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:54.128065  1340 raft_consensus.cc:2804] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:54.128793  1340 raft_consensus.cc:493] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:54.129266  1340 raft_consensus.cc:3060] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.129019  1256 raft_consensus.cc:359] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.134217  1256 raft_consensus.cc:385] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.134497  1256 raft_consensus.cc:740] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.135206  1256 consensus_queue.cc:260] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.136765   882 catalog_manager.cc:5697] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 0 to 1, leader changed from <none> to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:54.147850  1255 tablet_bootstrap.cc:654] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.149205  1256 ts_tablet_manager.cc:1434] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.000s
I20260709 13:11:54.151583  1256 tablet_bootstrap.cc:492] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:54.158195  1340 raft_consensus.cc:515] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.160038  1254 tablet_bootstrap.cc:654] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.164163  1256 tablet_bootstrap.cc:654] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.165148  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:54.165796  1209 raft_consensus.cc:3060] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.169170  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
W20260709 13:11:54.172197   950 leader_election.cc:343] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:54.173275  1340 leader_election.cc:290] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:54.175009  1209 raft_consensus.cc:2468] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:54.176203  1256 tablet_bootstrap.cc:492] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:54.176282   950 leader_election.cc:304] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:54.176766  1256 ts_tablet_manager.cc:1403] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.025s	user 0.013s	sys 0.003s
I20260709 13:11:54.177731  1340 raft_consensus.cc:2804] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:54.179075  1340 raft_consensus.cc:697] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:11:54.179764  1256 raft_consensus.cc:359] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.179904  1340 consensus_queue.cc:237] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.180531  1256 raft_consensus.cc:385] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.180847  1256 raft_consensus.cc:740] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.181459  1254 tablet_bootstrap.cc:492] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:54.181605  1256 consensus_queue.cc:260] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.182199  1254 ts_tablet_manager.cc:1403] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.076s	user 0.014s	sys 0.000s
I20260709 13:11:54.183473  1256 ts_tablet_manager.cc:1434] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:11:54.184798  1256 tablet_bootstrap.cc:492] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:54.184960  1254 raft_consensus.cc:359] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.185665  1254 raft_consensus.cc:385] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.187736  1254 raft_consensus.cc:740] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.188508  1254 consensus_queue.cc:260] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.190639  1254 ts_tablet_manager.cc:1434] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:11:54.191907  1254 tablet_bootstrap.cc:492] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:54.196240  1256 tablet_bootstrap.cc:654] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.203799  1263 raft_consensus.cc:493] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:54.204571  1263 raft_consensus.cc:515] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.207463  1256 tablet_bootstrap.cc:492] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:54.207541  1263 leader_election.cc:290] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:54.208161  1256 ts_tablet_manager.cc:1403] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.024s	user 0.009s	sys 0.006s
I20260709 13:11:54.209452   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:54.210294   985 raft_consensus.cc:2393] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 1: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:11:54.211055  1256 raft_consensus.cc:359] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.211738  1256 raft_consensus.cc:385] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.212011  1256 raft_consensus.cc:740] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.212366  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:54.212921  1256 consensus_queue.cc:260] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.214913  1256 ts_tablet_manager.cc:1434] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 13:11:54.214972  1025 leader_election.cc:304] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:54.216251  1256 tablet_bootstrap.cc:492] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:54.217367  1263 raft_consensus.cc:3060] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.222851  1256 tablet_bootstrap.cc:654] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.233783  1256 tablet_bootstrap.cc:492] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:54.234263  1256 ts_tablet_manager.cc:1403] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.018s	user 0.012s	sys 0.004s
I20260709 13:11:54.237486  1348 raft_consensus.cc:493] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:54.238003  1256 raft_consensus.cc:359] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.239077  1256 raft_consensus.cc:385] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.239274  1256 raft_consensus.cc:740] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.238430  1348 raft_consensus.cc:515] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.240576  1256 consensus_queue.cc:260] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.242084  1348 leader_election.cc:290] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:54.248395   883 catalog_manager.cc:5697] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 0 to 1, leader changed from <none> to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:54.246892  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:54.248172  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:54.243048  1367 raft_consensus.cc:493] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:54.251444  1062 raft_consensus.cc:2468] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 0.
I20260709 13:11:54.251523  1367 raft_consensus.cc:515] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.245000  1256 ts_tablet_manager.cc:1434] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 13:11:54.253309  1256 tablet_bootstrap.cc:492] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:54.255337   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "160ad44adbce4b7f97094b248fd05415" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:54.256366  1367 leader_election.cc:290] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:54.251544  1133 raft_consensus.cc:2468] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 0.
I20260709 13:11:54.258049  1173 leader_election.cc:304] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:54.258935  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "160ad44adbce4b7f97094b248fd05415" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
W20260709 13:11:54.259167  1027 leader_election.cc:343] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:54.259632  1134 raft_consensus.cc:2468] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:54.264495  1254 tablet_bootstrap.cc:654] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.270612  1025 leader_election.cc:304] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:11:54.271029  1257 tablet_bootstrap.cc:492] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:54.271567  1257 ts_tablet_manager.cc:1403] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.176s	user 0.032s	sys 0.046s
I20260709 13:11:54.273375  1256 tablet_bootstrap.cc:654] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.276535  1257 raft_consensus.cc:359] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.278036  1257 raft_consensus.cc:385] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.278390  1257 raft_consensus.cc:740] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.274349  1368 raft_consensus.cc:2804] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:54.269096  1348 raft_consensus.cc:2804] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:54.279981  1368 raft_consensus.cc:493] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:54.280182  1348 raft_consensus.cc:493] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:54.280470  1368 raft_consensus.cc:3060] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.280826  1348 raft_consensus.cc:3060] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.284471  1257 consensus_queue.cc:260] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.274024  1367 raft_consensus.cc:493] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:54.285707  1367 raft_consensus.cc:515] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.292212  1367 leader_election.cc:290] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:54.292387  1257 ts_tablet_manager.cc:1434] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.001s
I20260709 13:11:54.293519  1257 tablet_bootstrap.cc:492] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:54.294847  1368 raft_consensus.cc:515] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.297010  1368 leader_election.cc:290] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:54.302479   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "160ad44adbce4b7f97094b248fd05415" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:54.301177  1263 raft_consensus.cc:2749] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:54.304834  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:54.305629  1134 raft_consensus.cc:2393] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 1: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
I20260709 13:11:54.305845  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
W20260709 13:11:54.305524  1027 leader_election.cc:343] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:54.309955  1025 leader_election.cc:304] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:54.311815  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "160ad44adbce4b7f97094b248fd05415" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:54.312462  1134 raft_consensus.cc:3060] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.312886  1263 raft_consensus.cc:3060] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.319618  1263 raft_consensus.cc:2749] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:54.319785  1257 tablet_bootstrap.cc:654] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.320426  1134 raft_consensus.cc:2468] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:54.330809  1348 raft_consensus.cc:515] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.332590  1255 tablet_bootstrap.cc:492] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:54.333149  1255 ts_tablet_manager.cc:1403] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.254s	user 0.013s	sys 0.017s
I20260709 13:11:54.333184  1256 tablet_bootstrap.cc:492] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:54.333397  1348 leader_election.cc:290] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:54.333935  1256 ts_tablet_manager.cc:1403] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.081s	user 0.027s	sys 0.025s
I20260709 13:11:54.334067  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:54.334714  1062 raft_consensus.cc:3060] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.336442  1256 raft_consensus.cc:359] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.336689  1255 raft_consensus.cc:359] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.337344  1255 raft_consensus.cc:385] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.337445  1025 leader_election.cc:304] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:11:54.338369  1263 raft_consensus.cc:2804] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:54.339021  1257 tablet_bootstrap.cc:492] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:54.339099  1263 raft_consensus.cc:697] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:11:54.339622  1257 ts_tablet_manager.cc:1403] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.046s	user 0.011s	sys 0.000s
I20260709 13:11:54.337658  1255 raft_consensus.cc:740] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.340070  1263 consensus_queue.cc:237] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.341019  1255 consensus_queue.cc:260] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.342485  1257 raft_consensus.cc:359] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.343108  1257 raft_consensus.cc:385] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.343163  1255 ts_tablet_manager.cc:1434] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.000s
I20260709 13:11:54.343423  1257 raft_consensus.cc:740] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.343976  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:54.344159  1257 consensus_queue.cc:260] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.344605  1134 raft_consensus.cc:3060] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.345172  1256 raft_consensus.cc:385] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.345537  1256 raft_consensus.cc:740] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.346192  1256 consensus_queue.cc:260] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.348213  1256 ts_tablet_manager.cc:1434] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.014s	user 0.003s	sys 0.000s
I20260709 13:11:54.348626  1254 tablet_bootstrap.cc:492] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:54.349172  1254 ts_tablet_manager.cc:1403] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.158s	user 0.008s	sys 0.007s
I20260709 13:11:54.349295  1256 tablet_bootstrap.cc:492] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:54.350119  1062 raft_consensus.cc:2468] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:11:54.352058  1254 raft_consensus.cc:359] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.352751  1254 raft_consensus.cc:385] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.353016  1254 raft_consensus.cc:740] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.356556  1256 tablet_bootstrap.cc:654] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.358829  1257 ts_tablet_manager.cc:1434] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.003s
I20260709 13:11:54.358860  1173 leader_election.cc:304] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:54.359965  1348 raft_consensus.cc:2804] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:54.360383  1257 tablet_bootstrap.cc:492] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:54.360635  1348 raft_consensus.cc:697] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:11:54.362076  1348 consensus_queue.cc:237] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.365068  1401 consensus_queue.cc:1048] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:54.370462  1257 tablet_bootstrap.cc:654] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.372095  1293 consensus_queue.cc:1048] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:54.354195  1255 tablet_bootstrap.cc:492] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:54.361063  1254 consensus_queue.cc:260] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.387262  1340 consensus_queue.cc:1048] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:54.394462  1254 ts_tablet_manager.cc:1434] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.045s	user 0.006s	sys 0.000s
I20260709 13:11:54.395817  1254 tablet_bootstrap.cc:492] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:54.400741  1340 consensus_queue.cc:1048] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:54.403105  1349 consensus_queue.cc:1048] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:54.410575  1291 consensus_queue.cc:1048] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:54.418099  1257 tablet_bootstrap.cc:492] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:54.424413  1257 ts_tablet_manager.cc:1403] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.059s	user 0.013s	sys 0.015s
I20260709 13:11:54.433940  1257 raft_consensus.cc:359] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:54.434963  1257 raft_consensus.cc:385] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.435328  1257 raft_consensus.cc:740] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.436152  1257 consensus_queue.cc:260] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:54.437148  1401 consensus_queue.cc:1048] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:54.439033  1257 ts_tablet_manager.cc:1434] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.003s
I20260709 13:11:54.440495  1257 tablet_bootstrap.cc:492] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:54.449162  1291 raft_consensus.cc:493] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:54.449786  1291 raft_consensus.cc:515] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:54.452919  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e038f78f3e9540399aeda1f38cbea3fd" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:54.453408  1256 tablet_bootstrap.cc:492] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
W20260709 13:11:54.454457  1099 leader_election.cc:343] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 862b41697c74421fbac82c10c245434d (127.23.140.68:34477): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:54.457698   983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e038f78f3e9540399aeda1f38cbea3fd" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:54.459472  1101 leader_election.cc:343] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:54.459950  1101 leader_election.cc:304] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:11:54.454106  1256 ts_tablet_manager.cc:1403] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.105s	user 0.011s	sys 0.034s
I20260709 13:11:54.460701  1317 raft_consensus.cc:2749] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:54.464597  1256 raft_consensus.cc:359] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.465555  1256 raft_consensus.cc:385] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.465889  1256 raft_consensus.cc:740] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.466176  1291 leader_election.cc:290] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:54.469496  1256 consensus_queue.cc:260] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.471873  1257 tablet_bootstrap.cc:654] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.472321  1317 raft_consensus.cc:493] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:54.472888  1317 raft_consensus.cc:515] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.475033  1317 leader_election.cc:290] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:54.475498  1317 consensus_queue.cc:1048] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:54.476671  1256 ts_tablet_manager.cc:1434] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 13:11:54.478022  1256 tablet_bootstrap.cc:492] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:54.479655  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8aead26aca7435abf7d4af1ab2321af" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:54.480332  1060 raft_consensus.cc:2468] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:54.482373  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8aead26aca7435abf7d4af1ab2321af" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:54.483126  1210 raft_consensus.cc:2468] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:54.486756  1367 raft_consensus.cc:493] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:54.487150  1099 leader_election.cc:304] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:54.487838  1367 raft_consensus.cc:515] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.489482  1401 raft_consensus.cc:2804] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:54.489909  1401 raft_consensus.cc:493] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:54.490239  1401 raft_consensus.cc:3060] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.492049  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:54.493216  1210 raft_consensus.cc:2393] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 1: Already voted for candidate 294145900dd3496cb4a1d722613e1558 in this term.
I20260709 13:11:54.501703   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:54.503407  1027 leader_election.cc:304] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:11:54.504184  1368 raft_consensus.cc:3060] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.505416  1367 leader_election.cc:290] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:54.512825   883 catalog_manager.cc:5697] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:54.486713  1256 tablet_bootstrap.cc:654] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.524108  1256 tablet_bootstrap.cc:492] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:54.488879   880 catalog_manager.cc:5697] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 0 to 1, leader changed from <none> to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 1 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:54.527259  1255 tablet_bootstrap.cc:654] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.525688  1368 raft_consensus.cc:2749] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:54.532493  1256 ts_tablet_manager.cc:1403] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.055s	user 0.014s	sys 0.000s
I20260709 13:11:54.537050  1256 raft_consensus.cc:359] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.537739  1256 raft_consensus.cc:385] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.538013  1256 raft_consensus.cc:740] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.514690  1401 raft_consensus.cc:515] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.542613  1256 consensus_queue.cc:260] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.543730  1401 leader_election.cc:290] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:54.546314  1256 ts_tablet_manager.cc:1434] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.001s
I20260709 13:11:54.569051  1256 tablet_bootstrap.cc:492] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:54.400739  1134 raft_consensus.cc:2468] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:11:54.566291  1465 raft_consensus.cc:493] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:54.573953  1465 raft_consensus.cc:515] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.577018  1465 leader_election.cc:290] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:11:54.578681   932 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:11:54.578820  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6f69ac309b6488b81b409d785a2d465" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:54.582226  1254 tablet_bootstrap.cc:654] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
W20260709 13:11:54.582227  1173 leader_election.cc:343] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:54.585906  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8aead26aca7435abf7d4af1ab2321af" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:54.586149  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8aead26aca7435abf7d4af1ab2321af" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:54.586592  1210 raft_consensus.cc:3060] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.586741  1060 raft_consensus.cc:3060] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.586160   984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6f69ac309b6488b81b409d785a2d465" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:54.590005  1175 leader_election.cc:343] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:54.590248  1293 raft_consensus.cc:493] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:54.590420  1175 leader_election.cc:304] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:54.590840  1293 raft_consensus.cc:515] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.593350  1465 raft_consensus.cc:2749] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:54.598016  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d3595711f1e49dfb415fb2f6720e66d" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:54.599195  1366 consensus_queue.cc:1048] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:11:54.608124  1173 leader_election.cc:343] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:54.610401  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d3595711f1e49dfb415fb2f6720e66d" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:54.612509  1210 raft_consensus.cc:2468] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:54.612766  1060 raft_consensus.cc:2468] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:54.593048  1293 leader_election.cc:290] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
W20260709 13:11:54.614375  1173 leader_election.cc:343] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:54.619020  1173 leader_election.cc:304] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:54.620167  1293 raft_consensus.cc:2749] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:54.619169  1099 leader_election.cc:304] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:54.615087  1255 tablet_bootstrap.cc:492] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:54.617497  1368 raft_consensus.cc:493] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:54.621189  1368 raft_consensus.cc:515] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.621969  1401 raft_consensus.cc:2804] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:54.624600  1257 tablet_bootstrap.cc:492] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:54.625154  1257 ts_tablet_manager.cc:1403] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.185s	user 0.020s	sys 0.009s
I20260709 13:11:54.625175  1255 ts_tablet_manager.cc:1403] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.271s	user 0.016s	sys 0.011s
I20260709 13:11:54.627436   984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90e13f7e16774e68bca79e3f141ed6bc" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:54.631906  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90e13f7e16774e68bca79e3f141ed6bc" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:54.625015  1401 raft_consensus.cc:697] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:11:54.635421  1401 consensus_queue.cc:237] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.636430  1257 raft_consensus.cc:359] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:54.639330  1257 raft_consensus.cc:385] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.639637  1257 raft_consensus.cc:740] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.640218  1256 tablet_bootstrap.cc:654] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
W20260709 13:11:54.641254  1025 leader_election.cc:343] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:54.625844  1368 leader_election.cc:290] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:54.642632  1255 raft_consensus.cc:359] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.643456  1255 raft_consensus.cc:385] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.643498  1257 consensus_queue.cc:260] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:54.643944  1255 raft_consensus.cc:740] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.646185  1255 consensus_queue.cc:260] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
W20260709 13:11:54.643931  1027 leader_election.cc:343] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:54.647945  1027 leader_election.cc:304] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:54.648361  1255 ts_tablet_manager.cc:1434] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.000s
I20260709 13:11:54.648910  1368 raft_consensus.cc:2749] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:54.649675  1255 tablet_bootstrap.cc:492] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:54.656170  1257 ts_tablet_manager.cc:1434] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.030s	user 0.007s	sys 0.001s
I20260709 13:11:54.656868  1255 tablet_bootstrap.cc:654] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.657537  1257 tablet_bootstrap.cc:492] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:54.664170  1263 raft_consensus.cc:493] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:54.664783  1263 raft_consensus.cc:515] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.668738  1263 leader_election.cc:290] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:54.673231  1254 tablet_bootstrap.cc:492] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:54.673784  1254 ts_tablet_manager.cc:1403] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.278s	user 0.017s	sys 0.011s
I20260709 13:11:54.674147  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:54.674283  1256 tablet_bootstrap.cc:492] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:54.675091  1210 raft_consensus.cc:2393] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 1: Already voted for candidate 294145900dd3496cb4a1d722613e1558 in this term.
I20260709 13:11:54.675663  1256 ts_tablet_manager.cc:1403] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.113s	user 0.023s	sys 0.002s
I20260709 13:11:54.672793   984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:54.681401  1256 raft_consensus.cc:359] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.682197  1256 raft_consensus.cc:385] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.671020  1263 raft_consensus.cc:493] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:54.682581  1256 raft_consensus.cc:740] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.682024  1027 leader_election.cc:304] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:11:54.689462  1256 consensus_queue.cc:260] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.688829  1368 raft_consensus.cc:3060] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.692992  1254 raft_consensus.cc:359] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.694520  1254 raft_consensus.cc:385] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.694816  1254 raft_consensus.cc:740] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.697630  1254 consensus_queue.cc:260] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.693591  1313 consensus_queue.cc:1048] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 13:11:54.701084  1256 ts_tablet_manager.cc:1434] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.000s
I20260709 13:11:54.694063  1366 raft_consensus.cc:493] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:54.702368  1256 tablet_bootstrap.cc:492] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:54.702574  1366 raft_consensus.cc:515] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.708832  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "160ad44adbce4b7f97094b248fd05415" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:54.709460  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "160ad44adbce4b7f97094b248fd05415" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:54.688292  1263 raft_consensus.cc:515] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.712368  1254 ts_tablet_manager.cc:1434] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.038s	user 0.007s	sys 0.000s
I20260709 13:11:54.706651  1366 leader_election.cc:290] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:54.716152  1136 raft_consensus.cc:2393] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 in current term 1: Already voted for candidate 9c45b6d917734835a8a37b0f4c789ae6 in this term.
I20260709 13:11:54.716917  1254 tablet_bootstrap.cc:492] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:54.718874   950 leader_election.cc:304] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:54.720178  1366 raft_consensus.cc:3060] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.722472  1263 leader_election.cc:290] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:54.723081   879 catalog_manager.cc:5697] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:54.724515  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:54.725636   984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:54.727293  1027 leader_election.cc:343] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:11:54.729774  1025 leader_election.cc:343] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 862b41697c74421fbac82c10c245434d (127.23.140.68:34477): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:54.733349  1025 leader_election.cc:304] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:11:54.730023  1368 raft_consensus.cc:2749] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:54.730005  1366 raft_consensus.cc:2749] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:54.743331  1256 tablet_bootstrap.cc:654] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.745798  1465 raft_consensus.cc:493] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:54.746280  1465 raft_consensus.cc:515] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:54.748402  1465 leader_election.cc:290] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:54.750293  1293 consensus_queue.cc:1048] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:54.750537  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "545027a75bf749969ab288105031caff" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:54.751654  1293 raft_consensus.cc:493] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:54.752110  1293 raft_consensus.cc:515] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:54.752664   984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "545027a75bf749969ab288105031caff" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:54.753726  1173 leader_election.cc:343] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:11:54.754855  1175 leader_election.cc:343] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:54.755275  1175 leader_election.cc:304] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:54.756357  1465 raft_consensus.cc:2749] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:54.762200  1340 consensus_queue.cc:1048] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:54.762200  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e038f78f3e9540399aeda1f38cbea3fd" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:54.763648  1136 raft_consensus.cc:2468] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 0.
I20260709 13:11:54.764786  1173 leader_election.cc:304] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:54.767463  1401 raft_consensus.cc:493] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:54.768193  1401 raft_consensus.cc:515] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.768396  1465 raft_consensus.cc:2804] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:54.769341  1465 raft_consensus.cc:493] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:54.769785  1465 raft_consensus.cc:3060] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.775087  1255 tablet_bootstrap.cc:492] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:54.775781  1255 ts_tablet_manager.cc:1403] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.126s	user 0.015s	sys 0.025s
I20260709 13:11:54.737681  1263 raft_consensus.cc:2749] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:54.782254  1257 tablet_bootstrap.cc:654] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.791512  1401 leader_election.cc:290] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:54.788378   984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90e13f7e16774e68bca79e3f141ed6bc" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:54.792550   984 raft_consensus.cc:2468] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:54.792587  1313 consensus_queue.cc:1048] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:54.794801  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90e13f7e16774e68bca79e3f141ed6bc" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:54.795809  1060 raft_consensus.cc:2468] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:54.796411  1293 leader_election.cc:290] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:54.798847  1099 leader_election.cc:304] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:54.802328   984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e038f78f3e9540399aeda1f38cbea3fd" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:54.803314  1255 raft_consensus.cc:359] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:54.803648  1465 raft_consensus.cc:515] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:54.804102  1255 raft_consensus.cc:385] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 13:11:54.804291  1175 leader_election.cc:343] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:54.807265  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e038f78f3e9540399aeda1f38cbea3fd" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:54.807821  1465 leader_election.cc:290] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:54.807884  1136 raft_consensus.cc:3060] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.809952   984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e038f78f3e9540399aeda1f38cbea3fd" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:54.812230  1317 raft_consensus.cc:2804] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:54.812644  1317 raft_consensus.cc:493] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:54.812999  1317 raft_consensus.cc:3060] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:11:54.813257  1175 leader_election.cc:343] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:54.804574  1255 raft_consensus.cc:740] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.828140  1254 tablet_bootstrap.cc:654] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.825249  1255 consensus_queue.cc:260] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:54.830366  1256 tablet_bootstrap.cc:492] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:54.831284  1256 ts_tablet_manager.cc:1403] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.129s	user 0.029s	sys 0.042s
I20260709 13:11:54.844991  1136 raft_consensus.cc:2468] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:11:54.847872  1173 leader_election.cc:304] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:11:54.839571  1317 raft_consensus.cc:515] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.849874  1465 raft_consensus.cc:2804] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:54.850363  1465 raft_consensus.cc:697] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:11:54.851193  1465 consensus_queue.cc:237] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:54.851682  1317 leader_election.cc:290] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:54.853693  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90e13f7e16774e68bca79e3f141ed6bc" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:54.854576  1061 raft_consensus.cc:3060] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.854898   984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90e13f7e16774e68bca79e3f141ed6bc" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:54.855533   984 raft_consensus.cc:3060] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:54.855307  1256 raft_consensus.cc:359] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.856405  1256 raft_consensus.cc:385] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.856653  1256 raft_consensus.cc:740] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.857332  1256 consensus_queue.cc:260] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.858975  1255 ts_tablet_manager.cc:1434] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.083s	user 0.005s	sys 0.003s
I20260709 13:11:54.860064  1255 tablet_bootstrap.cc:492] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:54.871218   984 raft_consensus.cc:2468] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:54.872653  1101 leader_election.cc:304] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:54.874509  1401 raft_consensus.cc:2804] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:54.875003  1401 raft_consensus.cc:697] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:11:54.875826  1401 consensus_queue.cc:237] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:54.876185  1061 raft_consensus.cc:2468] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:54.880537  1293 consensus_queue.cc:1048] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 13:11:54.885401  1256 ts_tablet_manager.cc:1434] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.054s	user 0.007s	sys 0.000s
I20260709 13:11:54.886656  1256 tablet_bootstrap.cc:492] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:54.891636  1257 tablet_bootstrap.cc:492] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:54.892256  1257 ts_tablet_manager.cc:1403] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.235s	user 0.021s	sys 0.037s
I20260709 13:11:54.895385  1257 raft_consensus.cc:359] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:54.895965  1255 tablet_bootstrap.cc:654] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.896107  1257 raft_consensus.cc:385] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.896420  1257 raft_consensus.cc:740] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.897105  1257 consensus_queue.cc:260] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:54.903174  1263 consensus_queue.cc:1048] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:54.917423  1257 ts_tablet_manager.cc:1434] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.004s
I20260709 13:11:54.922013  1257 tablet_bootstrap.cc:492] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:54.923221  1256 tablet_bootstrap.cc:654] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.929854  1257 tablet_bootstrap.cc:654] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.935222  1254 tablet_bootstrap.cc:492] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:54.935720  1254 ts_tablet_manager.cc:1403] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.219s	user 0.009s	sys 0.009s
I20260709 13:11:54.936524  1255 tablet_bootstrap.cc:492] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:54.936997  1255 ts_tablet_manager.cc:1403] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.077s	user 0.008s	sys 0.004s
I20260709 13:11:54.938515  1254 raft_consensus.cc:359] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:54.939640  1255 raft_consensus.cc:359] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.946363  1255 raft_consensus.cc:385] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.946692  1255 raft_consensus.cc:740] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.947350  1255 consensus_queue.cc:260] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.951462  1255 ts_tablet_manager.cc:1434] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.002s
I20260709 13:11:54.951000  1254 raft_consensus.cc:385] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.953363  1254 raft_consensus.cc:740] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.954886  1254 consensus_queue.cc:260] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:54.963615  1254 ts_tablet_manager.cc:1434] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.028s	user 0.006s	sys 0.001s
I20260709 13:11:54.964825  1254 tablet_bootstrap.cc:492] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:54.970489  1255 tablet_bootstrap.cc:492] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:54.970551  1368 consensus_queue.cc:1048] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:54.971503  1256 tablet_bootstrap.cc:492] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:54.971992  1256 ts_tablet_manager.cc:1403] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.086s	user 0.009s	sys 0.004s
I20260709 13:11:54.974758  1256 raft_consensus.cc:359] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.975404  1256 raft_consensus.cc:385] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.975704  1256 raft_consensus.cc:740] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.977020  1254 tablet_bootstrap.cc:654] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.977778  1475 raft_consensus.cc:493] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:54.978062  1255 tablet_bootstrap.cc:654] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:54.978312  1475 raft_consensus.cc:515] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:54.979851  1256 consensus_queue.cc:260] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:54.980533  1475 leader_election.cc:290] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:54.982477  1256 ts_tablet_manager.cc:1434] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.001s
I20260709 13:11:54.982827   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "545027a75bf749969ab288105031caff" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:54.983153  1257 tablet_bootstrap.cc:492] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:54.983484  1256 tablet_bootstrap.cc:492] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:54.983829  1257 ts_tablet_manager.cc:1403] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.062s	user 0.013s	sys 0.014s
I20260709 13:11:54.984920  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "545027a75bf749969ab288105031caff" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:54.985615  1209 raft_consensus.cc:2468] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:54.987423  1257 raft_consensus.cc:359] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.988356  1257 raft_consensus.cc:385] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:54.988693  1257 raft_consensus.cc:740] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:54.993952  1257 consensus_queue.cc:260] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:54.996850  1257 ts_tablet_manager.cc:1434] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260709 13:11:54.998225  1257 tablet_bootstrap.cc:492] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:54.998019   879 catalog_manager.cc:5697] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 0 to 1, leader changed from <none> to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 1 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:54.999647  1465 raft_consensus.cc:493] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:55.000142  1465 raft_consensus.cc:515] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:55.003575   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:55.005263  1175 leader_election.cc:343] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:55.007714  1257 tablet_bootstrap.cc:654] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.012877  1099 leader_election.cc:304] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:55.013375  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:55.015895  1465 leader_election.cc:290] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:55.049018  1263 raft_consensus.cc:493] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:55.009902  1254 tablet_bootstrap.cc:492] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
W20260709 13:11:55.074231  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 61449us
I20260709 13:11:55.074314  1263 raft_consensus.cc:515] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.014048  1256 tablet_bootstrap.cc:654] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.073769  1401 raft_consensus.cc:2804] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:55.076365  1401 raft_consensus.cc:493] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:55.076870  1401 raft_consensus.cc:3060] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.041839   883 catalog_manager.cc:5697] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:55.084898  1313 raft_consensus.cc:493] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:55.085528  1313 raft_consensus.cc:515] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.090940  1254 ts_tablet_manager.cc:1403] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.126s	user 0.016s	sys 0.014s
I20260709 13:11:55.092811  1313 leader_election.cc:290] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:55.093458   985 raft_consensus.cc:2468] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:55.094240  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:55.097115  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3fbed7be9014cf5984a19768b6269d4" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:55.097529  1254 raft_consensus.cc:359] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.100834  1254 raft_consensus.cc:385] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.101159  1254 raft_consensus.cc:740] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.102200  1254 consensus_queue.cc:260] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.105613  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:11:55.106227  1173 leader_election.cc:343] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:55.107959  1173 leader_election.cc:304] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:55.109385  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3fbed7be9014cf5984a19768b6269d4" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:55.110025  1136 raft_consensus.cc:2468] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:55.092770  1263 leader_election.cc:290] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:55.106488  1207 raft_consensus.cc:2468] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:55.114840  1062 raft_consensus.cc:2468] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:55.114643  1465 raft_consensus.cc:2749] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:11:55.128450   950 leader_election.cc:343] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 862b41697c74421fbac82c10c245434d (127.23.140.68:34477): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:55.141918  1254 ts_tablet_manager.cc:1434] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.051s	user 0.007s	sys 0.000s
I20260709 13:11:55.151639  1254 tablet_bootstrap.cc:492] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:55.144490  1401 raft_consensus.cc:515] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:55.154671  1255 tablet_bootstrap.cc:492] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:55.161954  1255 ts_tablet_manager.cc:1403] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.207s	user 0.049s	sys 0.064s
I20260709 13:11:55.162678  1401 leader_election.cc:290] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:55.163409  1257 tablet_bootstrap.cc:492] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:55.163866  1257 ts_tablet_manager.cc:1403] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.166s	user 0.013s	sys 0.006s
I20260709 13:11:55.164742  1255 raft_consensus.cc:359] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.165447  1255 raft_consensus.cc:385] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.165712  1255 raft_consensus.cc:740] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.166282  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "545027a75bf749969ab288105031caff" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:55.166425  1255 consensus_queue.cc:260] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.168247  1257 raft_consensus.cc:359] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.168887  1257 raft_consensus.cc:385] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.169145  1257 raft_consensus.cc:740] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.169947  1257 consensus_queue.cc:260] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.157727  1025 leader_election.cc:304] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:55.174085  1368 raft_consensus.cc:2804] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:55.174717  1368 raft_consensus.cc:493] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:55.175076  1368 raft_consensus.cc:3060] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.158107  1367 raft_consensus.cc:493] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:55.160352  1256 tablet_bootstrap.cc:492] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:55.175661  1367 raft_consensus.cc:515] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.176405  1256 ts_tablet_manager.cc:1403] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.193s	user 0.022s	sys 0.032s
I20260709 13:11:55.177809  1367 leader_election.cc:290] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:55.147461  1475 consensus_queue.cc:1048] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:55.178640   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf962b21fa14634af6e810ca29f1bc7" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:55.179195  1256 raft_consensus.cc:359] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.179858  1256 raft_consensus.cc:385] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.156679   950 leader_election.cc:304] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 862b41697c74421fbac82c10c245434d
I20260709 13:11:55.166925  1210 raft_consensus.cc:3060] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.188078  1307 raft_consensus.cc:2804] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:55.188476  1307 raft_consensus.cc:493] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:55.188884  1307 raft_consensus.cc:3060] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.189234  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf962b21fa14634af6e810ca29f1bc7" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
W20260709 13:11:55.180296  1027 leader_election.cc:343] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:55.181375  1257 ts_tablet_manager.cc:1434] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.001s
I20260709 13:11:55.191761  1257 tablet_bootstrap.cc:492] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:55.188324   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "545027a75bf749969ab288105031caff" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:55.192835   986 raft_consensus.cc:3060] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.198668  1475 consensus_queue.cc:1048] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:55.200201  1210 raft_consensus.cc:2468] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:55.172613  1255 ts_tablet_manager.cc:1434] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.001s
I20260709 13:11:55.201830  1255 tablet_bootstrap.cc:492] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:55.202581  1099 leader_election.cc:304] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
W20260709 13:11:55.203605  1025 leader_election.cc:343] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:55.204069  1025 leader_election.cc:304] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:55.204857  1256 raft_consensus.cc:740] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.205742  1257 tablet_bootstrap.cc:654] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.205816  1401 raft_consensus.cc:2804] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:55.205888  1368 raft_consensus.cc:515] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.206712  1401 raft_consensus.cc:697] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:11:55.207110  1263 raft_consensus.cc:2749] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:55.207609  1401 consensus_queue.cc:237] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:55.208243  1368 leader_election.cc:290] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:55.209385  1256 consensus_queue.cc:260] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.210397  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3fbed7be9014cf5984a19768b6269d4" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:55.172582  1254 tablet_bootstrap.cc:654] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.211833  1134 raft_consensus.cc:3060] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.212029  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3fbed7be9014cf5984a19768b6269d4" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:55.212827  1210 raft_consensus.cc:3060] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.200290  1307 raft_consensus.cc:515] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.216553  1307 leader_election.cc:290] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:55.217725  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:55.219362  1209 raft_consensus.cc:3060] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.219588   986 raft_consensus.cc:2468] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:55.222116  1255 tablet_bootstrap.cc:654] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.222635  1256 ts_tablet_manager.cc:1434] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.046s	user 0.007s	sys 0.000s
I20260709 13:11:55.230468  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:55.231140  1061 raft_consensus.cc:3060] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.229338  1134 raft_consensus.cc:2468] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:55.234817  1209 raft_consensus.cc:2468] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:55.240460  1025 leader_election.cc:304] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:55.241411  1256 tablet_bootstrap.cc:492] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:55.243762   950 leader_election.cc:304] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 
I20260709 13:11:55.242539  1263 raft_consensus.cc:2804] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:55.244608  1307 raft_consensus.cc:2804] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:55.244875  1263 raft_consensus.cc:697] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:11:55.245175  1307 raft_consensus.cc:697] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:11:55.245859  1263 consensus_queue.cc:237] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.247073  1210 raft_consensus.cc:2468] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:55.245987  1307 consensus_queue.cc:237] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.262846   883 catalog_manager.cc:5697] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:55.265897  1254 tablet_bootstrap.cc:492] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:55.266402  1254 ts_tablet_manager.cc:1403] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.115s	user 0.013s	sys 0.005s
I20260709 13:11:55.271498  1061 raft_consensus.cc:2468] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:55.273921  1254 raft_consensus.cc:359] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.274749  1254 raft_consensus.cc:385] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.275095  1254 raft_consensus.cc:740] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.273967  1293 consensus_queue.cc:1048] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:55.275959  1254 consensus_queue.cc:260] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.279681  1254 ts_tablet_manager.cc:1434] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 13:11:55.279882  1255 tablet_bootstrap.cc:492] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:55.283458  1254 tablet_bootstrap.cc:492] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:55.283591  1255 ts_tablet_manager.cc:1403] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.082s	user 0.005s	sys 0.007s
I20260709 13:11:55.286808  1255 raft_consensus.cc:359] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:55.287524  1255 raft_consensus.cc:385] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.287779  1255 raft_consensus.cc:740] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.288491  1255 consensus_queue.cc:260] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:55.290972  1255 ts_tablet_manager.cc:1434] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.003s
I20260709 13:11:55.290261   880 catalog_manager.cc:5697] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:55.292044  1255 tablet_bootstrap.cc:492] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:55.291636  1254 tablet_bootstrap.cc:654] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.295534  1475 consensus_queue.cc:1048] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:55.297562  1256 tablet_bootstrap.cc:654] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.298707  1257 tablet_bootstrap.cc:492] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:55.299207  1257 ts_tablet_manager.cc:1403] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.108s	user 0.012s	sys 0.013s
W20260709 13:11:55.301895  1175 consensus_peers.cc:597] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:55.302098  1257 raft_consensus.cc:359] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.302802  1257 raft_consensus.cc:385] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.303040  1257 raft_consensus.cc:740] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.303692  1257 consensus_queue.cc:260] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.314074   883 catalog_manager.cc:5697] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 0 to 1, leader changed from <none> to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:55.305794  1257 ts_tablet_manager.cc:1434] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 13:11:55.318756  1257 tablet_bootstrap.cc:492] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:55.341380  1256 tablet_bootstrap.cc:492] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:55.341962  1256 ts_tablet_manager.cc:1403] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.102s	user 0.007s	sys 0.014s
I20260709 13:11:55.344672  1256 raft_consensus.cc:359] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.345383  1256 raft_consensus.cc:385] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.345660  1256 raft_consensus.cc:740] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.346313  1256 consensus_queue.cc:260] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.346850  1257 tablet_bootstrap.cc:654] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.348073  1256 ts_tablet_manager.cc:1434] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 13:11:55.350715  1254 tablet_bootstrap.cc:492] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:55.351233  1254 ts_tablet_manager.cc:1403] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.068s	user 0.017s	sys 0.000s
I20260709 13:11:55.353955  1254 raft_consensus.cc:359] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.354621  1254 raft_consensus.cc:385] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.354882  1254 raft_consensus.cc:740] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.355540  1254 consensus_queue.cc:260] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.357637  1254 ts_tablet_manager.cc:1434] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:11:55.358222  1401 consensus_queue.cc:1048] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:55.358700  1254 tablet_bootstrap.cc:492] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:55.360373  1255 tablet_bootstrap.cc:654] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.365360  1254 tablet_bootstrap.cc:654] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.371601  1256 tablet_bootstrap.cc:492] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:55.376690  1254 tablet_bootstrap.cc:492] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:55.377218  1254 ts_tablet_manager.cc:1403] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.019s	user 0.014s	sys 0.004s
I20260709 13:11:55.380702  1256 tablet_bootstrap.cc:654] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.380617  1254 raft_consensus.cc:359] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:55.381443  1254 raft_consensus.cc:385] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.381758  1254 raft_consensus.cc:740] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.382720  1254 consensus_queue.cc:260] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:55.385118  1254 ts_tablet_manager.cc:1434] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:11:55.387389  1254 tablet_bootstrap.cc:492] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:55.389355  1255 tablet_bootstrap.cc:492] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:55.389868  1255 ts_tablet_manager.cc:1403] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.098s	user 0.023s	sys 0.005s
I20260709 13:11:55.393812  1255 raft_consensus.cc:359] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.394632  1255 raft_consensus.cc:385] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.394942  1255 raft_consensus.cc:740] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.395968  1255 consensus_queue.cc:260] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.398243  1263 raft_consensus.cc:493] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:55.398913  1263 raft_consensus.cc:515] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.401266  1263 leader_election.cc:290] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:55.401502  1256 tablet_bootstrap.cc:492] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:55.402120  1256 ts_tablet_manager.cc:1403] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.031s	user 0.020s	sys 0.001s
I20260709 13:11:55.403015  1255 ts_tablet_manager.cc:1434] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 13:11:55.402959   984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6f69ac309b6488b81b409d785a2d465" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:55.405912   984 raft_consensus.cc:2468] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:55.407016  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6f69ac309b6488b81b409d785a2d465" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:55.407666  1210 raft_consensus.cc:2468] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:55.408010  1256 raft_consensus.cc:359] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.408655  1256 raft_consensus.cc:385] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.408900  1256 raft_consensus.cc:740] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.413782  1027 leader_election.cc:304] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:55.416561  1256 consensus_queue.cc:260] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.417693  1502 raft_consensus.cc:2804] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:55.418066  1502 raft_consensus.cc:493] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:55.418365  1502 raft_consensus.cc:3060] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.419230  1256 ts_tablet_manager.cc:1434] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.017s	user 0.004s	sys 0.003s
I20260709 13:11:55.420300  1256 tablet_bootstrap.cc:492] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:55.426735  1255 tablet_bootstrap.cc:492] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:55.428237  1348 raft_consensus.cc:493] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:55.429661  1348 raft_consensus.cc:515] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.460217   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11adc51543f7421cb3140972d9f0a424" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:55.462345  1257 tablet_bootstrap.cc:492] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:55.469527  1257 ts_tablet_manager.cc:1403] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.151s	user 0.022s	sys 0.008s
I20260709 13:11:55.472184  1257 raft_consensus.cc:359] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.473047  1257 raft_consensus.cc:385] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 13:11:55.473981  1175 leader_election.cc:343] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:55.474006  1257 raft_consensus.cc:740] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.475521  1257 consensus_queue.cc:260] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.466279  1255 tablet_bootstrap.cc:654] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.467115  1254 tablet_bootstrap.cc:654] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.480585  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11adc51543f7421cb3140972d9f0a424" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:55.481310  1061 raft_consensus.cc:2468] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 0.
I20260709 13:11:55.482908  1173 leader_election.cc:304] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:11:55.483726  1293 raft_consensus.cc:2804] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:55.484185  1293 raft_consensus.cc:493] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:55.484542  1293 raft_consensus.cc:3060] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.486620  1254 tablet_bootstrap.cc:492] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:55.487180  1254 ts_tablet_manager.cc:1403] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.100s	user 0.007s	sys 0.007s
I20260709 13:11:55.488489  1256 tablet_bootstrap.cc:654] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.490227  1254 raft_consensus.cc:359] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.491067  1254 raft_consensus.cc:385] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.491371  1254 raft_consensus.cc:740] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.493917  1502 raft_consensus.cc:515] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.496794  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6f69ac309b6488b81b409d785a2d465" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:55.497412  1209 raft_consensus.cc:3060] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.492189  1254 consensus_queue.cc:260] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.504858  1254 ts_tablet_manager.cc:1434] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.017s	user 0.004s	sys 0.002s
I20260709 13:11:55.508117  1254 tablet_bootstrap.cc:492] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:55.506963  1255 tablet_bootstrap.cc:492] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:55.508714  1255 ts_tablet_manager.cc:1403] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.082s	user 0.017s	sys 0.016s
I20260709 13:11:55.511935  1255 raft_consensus.cc:359] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:55.512718  1255 raft_consensus.cc:385] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.514003  1255 raft_consensus.cc:740] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.514643  1257 ts_tablet_manager.cc:1434] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.045s	user 0.005s	sys 0.001s
I20260709 13:11:55.517861  1348 leader_election.cc:290] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:55.519034  1257 tablet_bootstrap.cc:492] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:55.518718  1293 raft_consensus.cc:515] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.514981  1255 consensus_queue.cc:260] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:55.521462  1255 ts_tablet_manager.cc:1434] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 13:11:55.523384  1209 raft_consensus.cc:2468] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:55.524719  1025 leader_election.cc:304] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:55.525609  1293 leader_election.cc:290] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:55.531184  1307 raft_consensus.cc:493] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:55.528290  1257 tablet_bootstrap.cc:654] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.532231  1307 raft_consensus.cc:515] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:55.533914  1256 tablet_bootstrap.cc:492] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:55.534500  1307 leader_election.cc:290] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:55.536432  1307 raft_consensus.cc:493] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:55.537544  1307 raft_consensus.cc:515] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:55.538579  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e038f78f3e9540399aeda1f38cbea3fd" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:55.540602  1307 leader_election.cc:290] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:55.536610  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e038f78f3e9540399aeda1f38cbea3fd" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:55.529408   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11adc51543f7421cb3140972d9f0a424" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:55.537038  1473 raft_consensus.cc:493] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:55.542376  1255 tablet_bootstrap.cc:492] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:55.542672  1473 raft_consensus.cc:515] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.542975  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:55.543789  1210 raft_consensus.cc:2468] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:55.544564  1257 tablet_bootstrap.cc:492] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:55.544757  1473 leader_election.cc:290] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:55.545066  1257 ts_tablet_manager.cc:1403] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.026s	user 0.011s	sys 0.004s
I20260709 13:11:55.547734   950 leader_election.cc:304] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 
I20260709 13:11:55.548903  1255 tablet_bootstrap.cc:654] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.550689  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d3595711f1e49dfb415fb2f6720e66d" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:55.525653  1263 raft_consensus.cc:2804] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:55.551005  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:55.551419  1061 raft_consensus.cc:2468] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:55.551770  1263 raft_consensus.cc:697] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:11:55.551873  1133 raft_consensus.cc:2468] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:55.552623  1263 consensus_queue.cc:237] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.553035  1257 raft_consensus.cc:359] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:55.553737  1257 raft_consensus.cc:385] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.554503  1099 leader_election.cc:304] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:55.555380  1473 raft_consensus.cc:2804] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:55.555799  1473 raft_consensus.cc:493] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:55.556231  1473 raft_consensus.cc:3060] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:11:55.557024  1175 leader_election.cc:343] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:55.557116  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d3595711f1e49dfb415fb2f6720e66d" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:55.525993  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11adc51543f7421cb3140972d9f0a424" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:55.558036  1210 raft_consensus.cc:2468] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:55.558830  1307 raft_consensus.cc:2804] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:55.559288  1307 raft_consensus.cc:493] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:55.559656  1307 raft_consensus.cc:3060] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.551030   950 leader_election.cc:304] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:55.558092  1062 raft_consensus.cc:3060] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.561321  1313 raft_consensus.cc:2749] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:55.564685  1256 ts_tablet_manager.cc:1403] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.145s	user 0.036s	sys 0.001s
I20260709 13:11:55.565775  1255 tablet_bootstrap.cc:492] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:55.566350  1255 ts_tablet_manager.cc:1403] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.024s	user 0.014s	sys 0.003s
I20260709 13:11:55.567484  1256 raft_consensus.cc:359] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.568213  1256 raft_consensus.cc:385] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.568481  1256 raft_consensus.cc:740] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.570328  1502 leader_election.cc:290] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:55.570466  1255 raft_consensus.cc:359] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.571336  1255 raft_consensus.cc:385] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.571661  1255 raft_consensus.cc:740] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.572137   984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6f69ac309b6488b81b409d785a2d465" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:55.535827  1254 tablet_bootstrap.cc:654] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.572453  1255 consensus_queue.cc:260] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.573491  1313 raft_consensus.cc:493] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:55.574043  1313 raft_consensus.cc:515] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.572793   984 raft_consensus.cc:3060] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.576855  1313 leader_election.cc:290] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:55.579347  1293 raft_consensus.cc:493] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:55.579890  1293 raft_consensus.cc:515] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.581583  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf962b21fa14634af6e810ca29f1bc7" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:55.582291  1061 raft_consensus.cc:2468] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:55.584503  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7780e3bac1204186aefaa6f574717fdb" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:55.554036  1257 raft_consensus.cc:740] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.586516  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf962b21fa14634af6e810ca29f1bc7" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:55.586759   950 leader_election.cc:304] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:55.587244  1133 raft_consensus.cc:2468] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:55.587024  1257 consensus_queue.cc:260] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:55.586585  1501 raft_consensus.cc:493] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:55.589330  1293 leader_election.cc:290] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:55.589630  1257 ts_tablet_manager.cc:1434] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.044s	user 0.007s	sys 0.000s
I20260709 13:11:55.590023  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7780e3bac1204186aefaa6f574717fdb" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:55.590473  1313 raft_consensus.cc:2804] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:55.590184  1256 consensus_queue.cc:260] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.590844  1313 raft_consensus.cc:493] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:55.592089  1059 raft_consensus.cc:2468] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 0.
I20260709 13:11:55.592484  1313 raft_consensus.cc:3060] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.589165  1501 raft_consensus.cc:515] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
W20260709 13:11:55.591279  1173 leader_election.cc:343] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:55.597812  1173 leader_election.cc:304] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:11:55.599287  1256 ts_tablet_manager.cc:1434] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.034s	user 0.007s	sys 0.000s
I20260709 13:11:55.600382  1256 tablet_bootstrap.cc:492] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:55.601341   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb722a3043814db7a188e49c9861ceca" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:55.602059   986 raft_consensus.cc:2468] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:55.603219  1027 leader_election.cc:304] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:55.605862  1501 leader_election.cc:290] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:55.606001  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb722a3043814db7a188e49c9861ceca" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:55.606365  1502 raft_consensus.cc:2804] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:55.606840  1502 raft_consensus.cc:493] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:55.606945  1133 raft_consensus.cc:2468] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:55.608652  1255 ts_tablet_manager.cc:1434] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.042s	user 0.008s	sys 0.001s
I20260709 13:11:55.608299  1307 raft_consensus.cc:515] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:55.606560  1293 raft_consensus.cc:2804] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:55.609905  1293 raft_consensus.cc:493] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:55.610260  1293 raft_consensus.cc:3060] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.611256  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:55.611970  1133 raft_consensus.cc:3060] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.612908  1473 raft_consensus.cc:515] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.614025  1255 tablet_bootstrap.cc:492] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:55.615073   984 raft_consensus.cc:2468] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:55.607211  1502 raft_consensus.cc:3060] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.616250  1256 tablet_bootstrap.cc:654] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.617772  1257 tablet_bootstrap.cc:492] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:55.620863   883 catalog_manager.cc:5697] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:55.637465  1526 raft_consensus.cc:493] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:55.641227  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:55.642059  1209 raft_consensus.cc:3060] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.646816  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d3595711f1e49dfb415fb2f6720e66d" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:55.651508  1293 raft_consensus.cc:515] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.642076  1526 raft_consensus.cc:515] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.657974  1293 leader_election.cc:290] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:55.653733  1313 raft_consensus.cc:515] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.658996  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7780e3bac1204186aefaa6f574717fdb" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:55.659304  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7780e3bac1204186aefaa6f574717fdb" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:55.659876  1061 raft_consensus.cc:3060] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:11:55.661502  1173 leader_election.cc:343] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:55.645407  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d3595711f1e49dfb415fb2f6720e66d" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:55.663110  1059 raft_consensus.cc:3060] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.656129  1210 raft_consensus.cc:3060] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.666052  1313 leader_election.cc:290] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:55.647338  1473 leader_election.cc:290] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:55.671355  1062 raft_consensus.cc:2468] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:11:55.672175  1257 tablet_bootstrap.cc:654] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.672703  1526 leader_election.cc:290] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:55.672825  1173 leader_election.cc:304] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:11:55.639482  1307 leader_election.cc:290] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:55.676951  1255 tablet_bootstrap.cc:654] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.677364  1293 raft_consensus.cc:2804] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:55.679596  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf962b21fa14634af6e810ca29f1bc7" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:55.680292  1062 raft_consensus.cc:3060] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.680651  1254 tablet_bootstrap.cc:492] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:55.681185  1254 ts_tablet_manager.cc:1403] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.173s	user 0.010s	sys 0.008s
I20260709 13:11:55.681254  1293 raft_consensus.cc:697] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:11:55.682749  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf962b21fa14634af6e810ca29f1bc7" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:55.683418  1134 raft_consensus.cc:3060] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.673269  1133 raft_consensus.cc:2468] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:55.668428  1502 raft_consensus.cc:515] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.685236  1293 consensus_queue.cc:237] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.691183  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb722a3043814db7a188e49c9861ceca" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:55.692077  1060 raft_consensus.cc:2393] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 in current term 1: Already voted for candidate 9c45b6d917734835a8a37b0f4c789ae6 in this term.
I20260709 13:11:55.692684  1254 raft_consensus.cc:359] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.693675  1254 raft_consensus.cc:385] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.695027  1254 raft_consensus.cc:740] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.696048  1254 consensus_queue.cc:260] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.694640  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb722a3043814db7a188e49c9861ceca" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:55.700795  1133 raft_consensus.cc:2468] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:55.699405  1254 ts_tablet_manager.cc:1434] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.000s
I20260709 13:11:55.712250  1502 leader_election.cc:290] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:55.712735  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb722a3043814db7a188e49c9861ceca" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:55.713662  1133 raft_consensus.cc:3060] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.713783   984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb722a3043814db7a188e49c9861ceca" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:55.714498   984 raft_consensus.cc:3060] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.712177  1061 raft_consensus.cc:2468] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:11:55.716065  1173 leader_election.cc:304] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:11:55.705269   950 leader_election.cc:304] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:55.717303  1473 consensus_queue.cc:1048] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:55.718075  1526 raft_consensus.cc:2804] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:55.718544  1526 raft_consensus.cc:697] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:11:55.717303  1348 raft_consensus.cc:2804] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:55.719223  1348 raft_consensus.cc:697] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:11:55.719422  1526 consensus_queue.cc:237] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:55.720064  1348 consensus_queue.cc:237] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.724406   950 leader_election.cc:304] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:55.728986  1134 raft_consensus.cc:2468] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:55.736123  1062 raft_consensus.cc:2468] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:55.740682  1059 raft_consensus.cc:2468] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:55.740453  1502 consensus_queue.cc:1048] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:55.742753  1099 leader_election.cc:304] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:55.732833  1210 raft_consensus.cc:2468] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:55.750487  1256 tablet_bootstrap.cc:492] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:55.751374  1256 ts_tablet_manager.cc:1403] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.151s	user 0.008s	sys 0.010s
I20260709 13:11:55.752311   950 leader_election.cc:304] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:55.748994  1473 raft_consensus.cc:2804] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:55.762362  1254 tablet_bootstrap.cc:492] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:55.751696  1529 consensus_queue.cc:1048] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20260709 13:11:55.776104  1528 consensus_queue.cc:1048] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:55.777770  1473 raft_consensus.cc:697] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:11:55.785395  1473 consensus_queue.cc:237] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:55.778167  1256 raft_consensus.cc:359] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.791323  1256 raft_consensus.cc:385] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.791769  1256 raft_consensus.cc:740] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.803278  1256 consensus_queue.cc:260] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.806205   879 catalog_manager.cc:5697] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 0 to 1, leader changed from <none> to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 1 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:55.818624  1256 ts_tablet_manager.cc:1434] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.067s	user 0.006s	sys 0.003s
I20260709 13:11:55.777956  1526 raft_consensus.cc:2804] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:55.802802  1401 consensus_queue.cc:1048] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20260709 13:11:55.829859  1256 tablet_bootstrap.cc:492] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:55.840456  1133 raft_consensus.cc:2468] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:55.828356  1526 raft_consensus.cc:697] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:11:55.853282   880 catalog_manager.cc:5697] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 0 to 1, leader changed from <none> to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:55.830883   984 raft_consensus.cc:2468] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:55.861522  1209 raft_consensus.cc:2468] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:55.865801  1255 tablet_bootstrap.cc:492] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:55.866356  1255 ts_tablet_manager.cc:1403] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.253s	user 0.045s	sys 0.047s
W20260709 13:11:55.874240   932 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:11:55.874233   985 raft_consensus.cc:3060] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:55.862901  1307 raft_consensus.cc:2764] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 13:11:55.879983  1027 leader_election.cc:304] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:55.881426  1263 raft_consensus.cc:2804] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:55.882035  1263 raft_consensus.cc:697] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:11:55.885494  1257 tablet_bootstrap.cc:492] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:55.885919  1257 ts_tablet_manager.cc:1403] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.268s	user 0.013s	sys 0.018s
I20260709 13:11:55.891065  1257 raft_consensus.cc:359] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.891752  1257 raft_consensus.cc:385] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.892041  1257 raft_consensus.cc:740] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.892747  1257 consensus_queue.cc:260] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:55.883023  1263 consensus_queue.cc:237] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.895885  1254 tablet_bootstrap.cc:654] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.910953  1257 ts_tablet_manager.cc:1434] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.025s	user 0.008s	sys 0.000s
I20260709 13:11:55.904058   883 catalog_manager.cc:5697] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 0 to 1, leader changed from <none> to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 1 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:55.920922  1257 tablet_bootstrap.cc:492] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:55.909672  1502 consensus_queue.cc:1048] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260709 13:11:55.933759  1257 tablet_bootstrap.cc:654] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
W20260709 13:11:55.936640  1535 log.cc:927] Time spent T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 13:11:55.934772  1255 raft_consensus.cc:359] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:55.941846  1255 raft_consensus.cc:385] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:55.942363  1255 raft_consensus.cc:740] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:55.943250  1255 consensus_queue.cc:260] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:55.945406  1255 ts_tablet_manager.cc:1434] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.079s	user 0.008s	sys 0.000s
I20260709 13:11:55.946483  1255 tablet_bootstrap.cc:492] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:55.895306  1526 consensus_queue.cc:237] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:55.988794  1256 tablet_bootstrap.cc:654] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:55.989298  1293 raft_consensus.cc:493] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:55.991549  1293 raft_consensus.cc:515] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:56.071854  1254 tablet_bootstrap.cc:492] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:56.072412  1254 ts_tablet_manager.cc:1403] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.324s	user 0.011s	sys 0.013s
I20260709 13:11:56.082216  1254 raft_consensus.cc:359] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:56.084017  1254 raft_consensus.cc:385] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:56.084394  1254 raft_consensus.cc:740] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:56.085762  1254 consensus_queue.cc:260] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:56.086299   879 catalog_manager.cc:5697] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:56.090159   883 catalog_manager.cc:5697] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:56.098182  1254 ts_tablet_manager.cc:1434] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.002s
I20260709 13:11:56.093254  1293 leader_election.cc:290] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:56.102926  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0110653f11d472797f30dc22496cb12" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:56.107720   984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0110653f11d472797f30dc22496cb12" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:56.114707   885 catalog_manager.cc:5697] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 0 to 1, leader changed from <none> to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:56.119922   984 raft_consensus.cc:2468] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 0.
I20260709 13:11:56.118796  1134 raft_consensus.cc:2468] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 0.
I20260709 13:11:56.125443  1175 leader_election.cc:304] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 
I20260709 13:11:56.126493  1349 raft_consensus.cc:2804] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:56.126870  1349 raft_consensus.cc:493] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:56.127202  1349 raft_consensus.cc:3060] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:56.130119  1255 tablet_bootstrap.cc:654] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:56.134263  1475 raft_consensus.cc:493] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:56.135121  1349 raft_consensus.cc:515] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:56.135000  1475 raft_consensus.cc:515] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:56.154220  1263 consensus_queue.cc:1048] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:56.159734  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7780e3bac1204186aefaa6f574717fdb" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:56.159852  1208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7780e3bac1204186aefaa6f574717fdb" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:56.156818  1475 leader_election.cc:290] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:56.165745  1061 raft_consensus.cc:2393] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 1: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
I20260709 13:11:56.168110  1254 tablet_bootstrap.cc:492] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:56.171994  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0110653f11d472797f30dc22496cb12" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:56.172870  1133 raft_consensus.cc:3060] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:56.175612  1349 leader_election.cc:290] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:56.176661   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0110653f11d472797f30dc22496cb12" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:56.177311   985 raft_consensus.cc:3060] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:56.178424  1263 consensus_queue.cc:1048] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:56.181495  1099 leader_election.cc:304] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:56.184216  1475 raft_consensus.cc:3060] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:56.192837  1256 tablet_bootstrap.cc:492] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:56.193317  1349 raft_consensus.cc:493] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:56.194092  1349 raft_consensus.cc:515] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:56.193390  1256 ts_tablet_manager.cc:1403] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.368s	user 0.011s	sys 0.004s
I20260709 13:11:56.195616  1257 tablet_bootstrap.cc:492] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:56.196165  1257 ts_tablet_manager.cc:1403] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.278s	user 0.013s	sys 0.000s
I20260709 13:11:56.199570   985 raft_consensus.cc:2468] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:11:56.199985  1257 raft_consensus.cc:359] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:56.201295  1349 leader_election.cc:290] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:56.202399  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f50253bf21654ca6937a357eb070beda" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:56.198760  1256 raft_consensus.cc:359] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:56.210631  1256 raft_consensus.cc:385] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:56.210126  1257 raft_consensus.cc:385] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:56.215389   983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f50253bf21654ca6937a357eb070beda" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:56.216995  1175 leader_election.cc:343] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:56.217602  1257 raft_consensus.cc:740] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
W20260709 13:11:56.218560  1173 leader_election.cc:343] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:56.218338  1257 consensus_queue.cc:260] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:56.210922  1256 raft_consensus.cc:740] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:56.225936  1254 tablet_bootstrap.cc:654] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:56.225045  1256 consensus_queue.cc:260] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:56.233036  1175 leader_election.cc:304] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 
I20260709 13:11:56.223348  1173 leader_election.cc:304] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:56.234071  1465 raft_consensus.cc:2804] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:56.234519  1465 raft_consensus.cc:697] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:11:56.237790  1293 raft_consensus.cc:2749] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:56.238090  1465 consensus_queue.cc:237] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:56.243646  1526 consensus_queue.cc:1048] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:56.250375  1256 ts_tablet_manager.cc:1434] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.055s	user 0.003s	sys 0.005s
I20260709 13:11:56.250766  1257 ts_tablet_manager.cc:1434] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.054s	user 0.006s	sys 0.000s
I20260709 13:11:56.248850  1475 raft_consensus.cc:2749] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:56.264117  1257 tablet_bootstrap.cc:492] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:56.264830  1256 tablet_bootstrap.cc:492] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:56.250160  1317 consensus_queue.cc:1048] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260709 13:11:56.276852  1255 tablet_bootstrap.cc:492] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:56.277379  1255 ts_tablet_manager.cc:1403] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.331s	user 0.034s	sys 0.036s
I20260709 13:11:56.278964  1465 consensus_queue.cc:1048] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:56.280113  1255 raft_consensus.cc:359] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:56.280833  1255 raft_consensus.cc:385] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:56.281090  1255 raft_consensus.cc:740] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:56.281764  1255 consensus_queue.cc:260] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:56.303421  1257 tablet_bootstrap.cc:654] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:56.310523  1255 ts_tablet_manager.cc:1434] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.033s	user 0.006s	sys 0.000s
I20260709 13:11:56.313279   985 raft_consensus.cc:3060] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:56.313674  1255 tablet_bootstrap.cc:492] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:56.322227   880 catalog_manager.cc:5697] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 0 to 1, leader changed from <none> to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 1 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:56.324729  1526 consensus_queue.cc:1048] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 13:11:56.329208  1256 tablet_bootstrap.cc:654] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:56.345525  1255 tablet_bootstrap.cc:654] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:56.356334  1263 raft_consensus.cc:493] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:56.357640  1348 consensus_queue.cc:1048] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:56.353024  1133 raft_consensus.cc:2468] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:11:56.356894  1263 raft_consensus.cc:515] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:56.370366  1263 leader_election.cc:290] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:56.370867  1263 consensus_queue.cc:1048] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:56.403584  1263 consensus_queue.cc:1048] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:56.402904  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:56.410815  1257 tablet_bootstrap.cc:492] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:56.411362  1257 ts_tablet_manager.cc:1403] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.154s	user 0.007s	sys 0.004s
I20260709 13:11:56.383809  1348 raft_consensus.cc:493] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:56.415035   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:56.415315  1348 raft_consensus.cc:515] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:56.416074  1293 consensus_queue.cc:1048] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:56.418815  1293 consensus_queue.cc:1048] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:11:56.427027  1027 leader_election.cc:343] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:56.450443  1257 raft_consensus.cc:359] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:56.451570  1257 raft_consensus.cc:385] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:56.451920  1257 raft_consensus.cc:740] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
W20260709 13:11:56.435386  1025 leader_election.cc:343] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:56.463622  1257 consensus_queue.cc:260] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:56.460897  1348 leader_election.cc:290] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:56.461463  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eec3e72036941a5b16c3a0157986a93" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:56.461164  1025 leader_election.cc:304] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:56.478538  1540 consensus_queue.cc:1048] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
I20260709 13:11:56.485471  1263 raft_consensus.cc:2749] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:11:56.485661  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 56309.7us
I20260709 13:11:56.488288   984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eec3e72036941a5b16c3a0157986a93" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:56.498392  1175 leader_election.cc:343] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:56.501086  1254 tablet_bootstrap.cc:492] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:56.501859  1254 ts_tablet_manager.cc:1403] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.334s	user 0.028s	sys 0.015s
I20260709 13:11:56.512251  1307 consensus_queue.cc:1048] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:56.518085  1254 raft_consensus.cc:359] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:56.518411  1293 raft_consensus.cc:493] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:56.521117  1293 raft_consensus.cc:515] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
W20260709 13:11:56.525031  1173 leader_election.cc:343] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:56.525225  1293 leader_election.cc:290] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:56.525533  1173 leader_election.cc:304] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:56.526971  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12df324e933f422c9a8ff5d37eb7d4ef" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:56.521795  1257 ts_tablet_manager.cc:1434] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.110s	user 0.007s	sys 0.002s
I20260709 13:11:56.530591  1254 raft_consensus.cc:385] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:56.531771  1254 raft_consensus.cc:740] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:56.533000  1254 consensus_queue.cc:260] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:56.540254  1256 tablet_bootstrap.cc:492] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:56.540809  1256 ts_tablet_manager.cc:1403] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.276s	user 0.015s	sys 0.061s
I20260709 13:11:56.540284  1254 ts_tablet_manager.cc:1434] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.038s	user 0.007s	sys 0.000s
I20260709 13:11:56.542125  1254 tablet_bootstrap.cc:492] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:56.543514  1256 raft_consensus.cc:359] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:56.544175  1256 raft_consensus.cc:385] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:56.544430  1256 raft_consensus.cc:740] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:56.545226  1256 consensus_queue.cc:260] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:56.549222  1254 tablet_bootstrap.cc:654] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:56.563428   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12df324e933f422c9a8ff5d37eb7d4ef" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:56.534636  1348 raft_consensus.cc:2749] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:56.556547  1528 consensus_queue.cc:1048] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
W20260709 13:11:56.570329  1175 leader_election.cc:343] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:56.576061  1257 tablet_bootstrap.cc:492] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:56.585788  1256 ts_tablet_manager.cc:1434] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.045s	user 0.014s	sys 0.002s
I20260709 13:11:56.605681  1255 tablet_bootstrap.cc:492] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:56.606447  1255 ts_tablet_manager.cc:1403] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.293s	user 0.020s	sys 0.018s
I20260709 13:11:56.616959  1256 tablet_bootstrap.cc:492] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:56.644409  1255 raft_consensus.cc:359] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:56.650560  1255 raft_consensus.cc:385] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:56.650911  1255 raft_consensus.cc:740] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:56.652501  1257 tablet_bootstrap.cc:654] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:56.656366  1255 consensus_queue.cc:260] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:56.658887  1255 ts_tablet_manager.cc:1434] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.052s	user 0.007s	sys 0.000s
I20260709 13:11:56.660872  1263 raft_consensus.cc:493] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:56.661648  1263 raft_consensus.cc:515] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:56.668231  1255 tablet_bootstrap.cc:492] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
W20260709 13:11:56.666373  1173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 68983.2us
W20260709 13:11:56.670948  1173 leader_election.cc:343] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:56.671399  1173 leader_election.cc:304] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:56.672190  1465 raft_consensus.cc:2749] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:56.677366  1255 tablet_bootstrap.cc:654] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:56.686196  1263 leader_election.cc:290] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:56.687320   983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f50253bf21654ca6937a357eb070beda" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:56.688802  1027 leader_election.cc:343] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:56.689671  1263 raft_consensus.cc:493] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:56.690672  1263 raft_consensus.cc:515] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:56.704347  1263 leader_election.cc:290] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:56.706033   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "483b49446c4e4a23b780921d7fabfbdf" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:56.711321  1208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f50253bf21654ca6937a357eb070beda" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:56.712101  1208 raft_consensus.cc:2468] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:56.713677  1025 leader_election.cc:304] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:11:56.714823  1579 raft_consensus.cc:2804] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:56.715205  1579 raft_consensus.cc:493] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:56.715517  1579 raft_consensus.cc:3060] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:56.716130  1256 tablet_bootstrap.cc:654] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
W20260709 13:11:56.721395  1027 leader_election.cc:343] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:56.721387  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "483b49446c4e4a23b780921d7fabfbdf" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:56.729271  1579 raft_consensus.cc:515] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:56.737360   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f50253bf21654ca6937a357eb070beda" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
W20260709 13:11:56.738982  1027 leader_election.cc:343] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:56.741482  1579 leader_election.cc:290] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
W20260709 13:11:56.750532  1025 leader_election.cc:343] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:56.751065  1025 leader_election.cc:304] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:56.741947  1208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f50253bf21654ca6937a357eb070beda" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:56.751892  1208 raft_consensus.cc:3060] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:56.752954  1254 tablet_bootstrap.cc:492] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:56.753726  1254 ts_tablet_manager.cc:1403] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.212s	user 0.022s	sys 0.027s
I20260709 13:11:56.759994  1538 raft_consensus.cc:493] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:56.760573  1538 raft_consensus.cc:515] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:56.761576  1208 raft_consensus.cc:2468] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:56.762782  1538 leader_election.cc:290] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:56.764324  1025 leader_election.cc:304] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:11:56.752807  1579 raft_consensus.cc:2749] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:56.765750  1578 raft_consensus.cc:2804] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:56.765389  1254 raft_consensus.cc:359] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:56.766260  1578 raft_consensus.cc:697] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:11:56.766420  1254 raft_consensus.cc:385] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:56.766866  1254 raft_consensus.cc:740] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:56.767216  1578 consensus_queue.cc:237] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:56.767889  1254 consensus_queue.cc:260] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:56.765419  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:56.770452  1062 raft_consensus.cc:2468] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:56.772284   950 leader_election.cc:304] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:56.777050  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:56.777767  1136 raft_consensus.cc:2468] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:56.780210  1528 raft_consensus.cc:2804] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:56.780665  1528 raft_consensus.cc:493] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:56.781021  1528 raft_consensus.cc:3060] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:56.790524  1254 ts_tablet_manager.cc:1434] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.036s	user 0.008s	sys 0.000s
I20260709 13:11:56.793180  1256 tablet_bootstrap.cc:492] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:56.793735  1256 ts_tablet_manager.cc:1403] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.201s	user 0.022s	sys 0.000s
I20260709 13:11:56.794943  1528 raft_consensus.cc:515] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:56.800181  1585 consensus_queue.cc:1048] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:56.802664  1528 leader_election.cc:290] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:56.796473  1256 raft_consensus.cc:359] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:56.804144  1256 raft_consensus.cc:385] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:56.804430  1256 raft_consensus.cc:740] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:56.805900  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:56.806491  1062 raft_consensus.cc:3060] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:56.806383  1256 consensus_queue.cc:260] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:56.807621  1254 tablet_bootstrap.cc:492] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:56.814272  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:56.814934  1134 raft_consensus.cc:3060] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:56.819568  1062 raft_consensus.cc:2468] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:56.830717   950 leader_election.cc:304] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:56.831670  1528 raft_consensus.cc:2804] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:56.832487  1528 raft_consensus.cc:697] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:11:56.833381  1528 consensus_queue.cc:237] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:56.823405  1585 consensus_queue.cc:1048] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260709 13:11:56.836144  1256 ts_tablet_manager.cc:1434] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.042s	user 0.007s	sys 0.000s
I20260709 13:11:56.837293  1256 tablet_bootstrap.cc:492] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:56.842881  1257 tablet_bootstrap.cc:492] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:56.843437  1257 ts_tablet_manager.cc:1403] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.285s	user 0.022s	sys 0.042s
I20260709 13:11:56.843794  1255 tablet_bootstrap.cc:492] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:56.844354  1255 ts_tablet_manager.cc:1403] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.177s	user 0.013s	sys 0.004s
I20260709 13:11:56.846257  1257 raft_consensus.cc:359] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:56.846971  1257 raft_consensus.cc:385] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:56.847245  1257 raft_consensus.cc:740] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:56.847172  1255 raft_consensus.cc:359] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:56.852861  1256 tablet_bootstrap.cc:654] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:56.853976  1255 raft_consensus.cc:385] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:56.854391  1255 raft_consensus.cc:740] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:56.855218  1255 consensus_queue.cc:260] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:56.847301  1317 raft_consensus.cc:493] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:56.859985  1317 raft_consensus.cc:515] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:56.864619  1317 leader_election.cc:290] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:56.864092  1134 raft_consensus.cc:2468] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:56.866148  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eec3e72036941a5b16c3a0157986a93" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:56.875196  1209 raft_consensus.cc:2468] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:56.869535   983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eec3e72036941a5b16c3a0157986a93" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:56.876209   983 raft_consensus.cc:2468] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:56.876092   883 catalog_manager.cc:5697] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:56.878119  1099 leader_election.cc:304] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:56.879535  1254 tablet_bootstrap.cc:654] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:56.880391  1255 ts_tablet_manager.cc:1434] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.036s	user 0.006s	sys 0.001s
I20260709 13:11:56.887534  1317 raft_consensus.cc:2804] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:56.888051  1255 tablet_bootstrap.cc:492] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:56.888621  1257 consensus_queue.cc:260] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:56.896596  1580 raft_consensus.cc:493] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:56.897367  1580 raft_consensus.cc:515] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:56.899344  1256 tablet_bootstrap.cc:492] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:56.899816  1580 leader_election.cc:290] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:56.900264  1256 ts_tablet_manager.cc:1403] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.063s	user 0.015s	sys 0.002s
I20260709 13:11:56.903077  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ddfc34a205c43ad81653b5000cbf394" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:56.903092  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ddfc34a205c43ad81653b5000cbf394" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:56.904367  1257 ts_tablet_manager.cc:1434] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.061s	user 0.004s	sys 0.004s
I20260709 13:11:56.907768  1257 tablet_bootstrap.cc:492] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:56.891513  1317 raft_consensus.cc:493] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:56.913017  1317 raft_consensus.cc:3060] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:56.918167  1257 tablet_bootstrap.cc:654] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
W20260709 13:11:56.921274  1025 leader_election.cc:343] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 862b41697c74421fbac82c10c245434d (127.23.140.68:34477): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:11:56.922099  1025 leader_election.cc:343] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:56.922448  1025 leader_election.cc:304] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:56.923616  1579 raft_consensus.cc:2749] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:56.928071  1256 raft_consensus.cc:359] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:56.928920  1256 raft_consensus.cc:385] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:56.929270  1256 raft_consensus.cc:740] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:56.930114  1256 consensus_queue.cc:260] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:56.932010  1255 tablet_bootstrap.cc:654] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:56.932482  1256 ts_tablet_manager.cc:1434] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.032s	user 0.005s	sys 0.002s
I20260709 13:11:56.933637  1256 tablet_bootstrap.cc:492] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:56.948114   885 catalog_manager.cc:5697] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 0 to 1, leader changed from <none> to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:56.979879  1255 tablet_bootstrap.cc:492] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:56.984009  1255 ts_tablet_manager.cc:1403] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.096s	user 0.010s	sys 0.004s
I20260709 13:11:56.995165  1256 tablet_bootstrap.cc:654] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:56.997782  1317 raft_consensus.cc:515] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.002457  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eec3e72036941a5b16c3a0157986a93" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:57.003240  1209 raft_consensus.cc:3060] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.009119  1256 tablet_bootstrap.cc:492] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:57.009723  1256 ts_tablet_manager.cc:1403] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.076s	user 0.013s	sys 0.007s
I20260709 13:11:57.008996  1255 raft_consensus.cc:359] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.010311  1255 raft_consensus.cc:385] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.010617  1255 raft_consensus.cc:740] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.011684  1255 consensus_queue.cc:260] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.013911  1256 raft_consensus.cc:359] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.014662  1255 ts_tablet_manager.cc:1434] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.030s	user 0.009s	sys 0.000s
I20260709 13:11:57.016785  1209 raft_consensus.cc:2468] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:57.018885  1317 leader_election.cc:290] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:57.018635  1257 tablet_bootstrap.cc:492] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:57.019654  1257 ts_tablet_manager.cc:1403] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.112s	user 0.014s	sys 0.044s
I20260709 13:11:57.020980  1099 leader_election.cc:304] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:57.022688  1257 raft_consensus.cc:359] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.023491  1257 raft_consensus.cc:385] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.023836  1257 raft_consensus.cc:740] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.023990  1583 raft_consensus.cc:2804] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:57.024570  1583 raft_consensus.cc:697] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:11:57.024678  1257 consensus_queue.cc:260] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.025418  1583 consensus_queue.cc:237] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.027172  1257 ts_tablet_manager.cc:1434] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:11:57.028136  1257 tablet_bootstrap.cc:492] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:57.028556  1254 tablet_bootstrap.cc:492] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:57.029107  1254 ts_tablet_manager.cc:1403] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.222s	user 0.023s	sys 0.015s
I20260709 13:11:57.034294  1257 tablet_bootstrap.cc:654] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.034961   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eec3e72036941a5b16c3a0157986a93" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:57.035580   987 raft_consensus.cc:3060] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.032605  1254 raft_consensus.cc:359] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:57.045990  1254 raft_consensus.cc:385] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.046322  1254 raft_consensus.cc:740] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.047017  1254 consensus_queue.cc:260] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:57.047681   987 raft_consensus.cc:2468] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:57.049099  1254 ts_tablet_manager.cc:1434] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.020s	user 0.002s	sys 0.002s
I20260709 13:11:57.050266  1254 tablet_bootstrap.cc:492] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:57.053574  1256 raft_consensus.cc:385] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.053987  1256 raft_consensus.cc:740] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.057082  1256 consensus_queue.cc:260] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.061434  1256 ts_tablet_manager.cc:1434] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.051s	user 0.006s	sys 0.002s
I20260709 13:11:57.062881  1256 tablet_bootstrap.cc:492] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:57.016016  1255 tablet_bootstrap.cc:492] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:57.079177   880 catalog_manager.cc:5697] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:57.082544  1254 tablet_bootstrap.cc:654] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.089650  1256 tablet_bootstrap.cc:654] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.089439  1584 raft_consensus.cc:493] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:57.103117  1584 raft_consensus.cc:515] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.103686  1257 tablet_bootstrap.cc:492] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:57.104202  1257 ts_tablet_manager.cc:1403] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.076s	user 0.007s	sys 0.011s
I20260709 13:11:57.106971  1257 raft_consensus.cc:359] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.107632  1257 raft_consensus.cc:385] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.107897  1257 raft_consensus.cc:740] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.108143  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8c8635d8e54625af9c95ecbcd5653b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:57.109565  1584 leader_election.cc:290] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:57.109289  1257 consensus_queue.cc:260] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
W20260709 13:11:57.111285  1173 leader_election.cc:343] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:57.111541   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8c8635d8e54625af9c95ecbcd5653b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:57.112452  1257 ts_tablet_manager.cc:1434] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
W20260709 13:11:57.113863  1175 leader_election.cc:343] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:57.114297  1175 leader_election.cc:304] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:57.114619  1257 tablet_bootstrap.cc:492] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
W20260709 13:11:57.098304   932 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:11:57.115185  1255 tablet_bootstrap.cc:654] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.114970  1584 raft_consensus.cc:2749] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:57.143159  1254 tablet_bootstrap.cc:492] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:57.152192  1256 tablet_bootstrap.cc:492] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:57.152348  1254 ts_tablet_manager.cc:1403] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.102s	user 0.010s	sys 0.009s
I20260709 13:11:57.152740  1256 ts_tablet_manager.cc:1403] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.090s	user 0.012s	sys 0.006s
I20260709 13:11:57.155025  1254 raft_consensus.cc:359] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.155579  1254 raft_consensus.cc:385] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.155869  1254 raft_consensus.cc:740] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.156519  1254 consensus_queue.cc:260] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.156215  1256 raft_consensus.cc:359] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.157303  1256 raft_consensus.cc:385] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.157671  1256 raft_consensus.cc:740] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.158541  1256 consensus_queue.cc:260] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.160071  1257 tablet_bootstrap.cc:654] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.160547  1255 tablet_bootstrap.cc:492] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:57.160987  1255 ts_tablet_manager.cc:1403] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.145s	user 0.015s	sys 0.010s
I20260709 13:11:57.161324  1501 raft_consensus.cc:493] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:57.161842  1501 raft_consensus.cc:515] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.163416  1255 raft_consensus.cc:359] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.163945  1501 leader_election.cc:290] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:57.165205  1208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12df324e933f422c9a8ff5d37eb7d4ef" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:57.165620   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12df324e933f422c9a8ff5d37eb7d4ef" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:57.165848  1208 raft_consensus.cc:2468] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:57.167131  1025 leader_election.cc:304] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:57.167935  1501 raft_consensus.cc:2804] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:57.168305  1501 raft_consensus.cc:493] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:57.168340  1256 ts_tablet_manager.cc:1434] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.000s
I20260709 13:11:57.168327  1257 tablet_bootstrap.cc:492] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:57.169301  1257 ts_tablet_manager.cc:1403] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.055s	user 0.007s	sys 0.004s
I20260709 13:11:57.169726  1256 tablet_bootstrap.cc:492] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:57.172520  1257 raft_consensus.cc:359] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.173362  1257 raft_consensus.cc:385] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.174116  1257 raft_consensus.cc:740] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.174840  1257 consensus_queue.cc:260] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.164052  1255 raft_consensus.cc:385] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.176406  1256 tablet_bootstrap.cc:654] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.176622  1255 raft_consensus.cc:740] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.163995  1254 ts_tablet_manager.cc:1434] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.003s
I20260709 13:11:57.177232  1257 ts_tablet_manager.cc:1434] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:11:57.178669  1257 tablet_bootstrap.cc:492] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:57.184815  1257 tablet_bootstrap.cc:654] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.199956  1257 tablet_bootstrap.cc:492] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:57.200526  1257 ts_tablet_manager.cc:1403] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.022s	user 0.011s	sys 0.005s
I20260709 13:11:57.203815  1257 raft_consensus.cc:359] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.204536  1257 raft_consensus.cc:385] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.204842  1257 raft_consensus.cc:740] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.205696  1257 consensus_queue.cc:260] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.207772  1257 ts_tablet_manager.cc:1434] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.003s
I20260709 13:11:57.208796  1257 tablet_bootstrap.cc:492] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:57.210455  1254 tablet_bootstrap.cc:492] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:57.214907  1257 tablet_bootstrap.cc:654] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.216806  1254 tablet_bootstrap.cc:654] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.219013  1255 consensus_queue.cc:260] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.220606   987 raft_consensus.cc:2468] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:57.223359  1501 raft_consensus.cc:3060] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.229919  1257 tablet_bootstrap.cc:492] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:57.230513  1257 ts_tablet_manager.cc:1403] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.022s	user 0.013s	sys 0.004s
I20260709 13:11:57.230630  1255 ts_tablet_manager.cc:1434] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.069s	user 0.006s	sys 0.000s
I20260709 13:11:57.231787  1255 tablet_bootstrap.cc:492] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:57.233507  1257 raft_consensus.cc:359] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.234277  1257 raft_consensus.cc:385] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.234606  1257 raft_consensus.cc:740] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.235352  1257 consensus_queue.cc:260] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.242622   987 raft_consensus.cc:3060] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.245488  1349 raft_consensus.cc:493] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:57.246196  1349 raft_consensus.cc:515] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.250020   983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b75f9192e7246d2a560fd091fb68606" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:57.257987  1175 leader_election.cc:343] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:57.258273  1580 consensus_queue.cc:1048] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:57.261765  1257 ts_tablet_manager.cc:1434] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.031s	user 0.006s	sys 0.000s
I20260709 13:11:57.263015  1257 tablet_bootstrap.cc:492] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:57.269752  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b75f9192e7246d2a560fd091fb68606" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:57.270414  1257 tablet_bootstrap.cc:654] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.270493  1061 raft_consensus.cc:2468] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 0.
I20260709 13:11:57.271908  1173 leader_election.cc:304] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:11:57.272886  1584 raft_consensus.cc:2804] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:57.273293  1584 raft_consensus.cc:493] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:57.275691  1584 raft_consensus.cc:3060] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.282279  1257 tablet_bootstrap.cc:492] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:57.282896  1257 ts_tablet_manager.cc:1403] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.020s	user 0.014s	sys 0.003s
I20260709 13:11:57.283650  1255 tablet_bootstrap.cc:654] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.285920  1257 raft_consensus.cc:359] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.286792  1257 raft_consensus.cc:385] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.287098  1257 raft_consensus.cc:740] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.287894  1257 consensus_queue.cc:260] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.290141  1257 ts_tablet_manager.cc:1434] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:11:57.291483  1257 tablet_bootstrap.cc:492] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:57.297299  1257 tablet_bootstrap.cc:654] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.304165  1579 consensus_queue.cc:1048] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:57.310732  1254 tablet_bootstrap.cc:492] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:57.310830  1257 tablet_bootstrap.cc:492] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:57.311493  1254 ts_tablet_manager.cc:1403] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.101s	user 0.010s	sys 0.000s
I20260709 13:11:57.311674  1257 ts_tablet_manager.cc:1403] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.020s	user 0.012s	sys 0.004s
I20260709 13:11:57.314325  1254 raft_consensus.cc:359] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.315057  1254 raft_consensus.cc:385] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.315346  1254 raft_consensus.cc:740] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.314697  1257 raft_consensus.cc:359] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.316612  1257 raft_consensus.cc:385] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.316934  1257 raft_consensus.cc:740] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.317628  1257 consensus_queue.cc:260] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.319682  1257 ts_tablet_manager.cc:1434] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.002s
I20260709 13:11:57.319912  1254 consensus_queue.cc:260] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.320844  1257 tablet_bootstrap.cc:492] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:57.323791  1254 ts_tablet_manager.cc:1434] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.001s
I20260709 13:11:57.327309  1257 tablet_bootstrap.cc:654] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.323664  1349 leader_election.cc:290] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:57.338711  1257 tablet_bootstrap.cc:492] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:57.342387  1257 ts_tablet_manager.cc:1403] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.022s	user 0.013s	sys 0.005s
I20260709 13:11:57.342890  1583 raft_consensus.cc:493] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:57.343413  1583 raft_consensus.cc:515] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.339051  1255 tablet_bootstrap.cc:492] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:57.337054  1254 tablet_bootstrap.cc:492] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:57.345917  1255 ts_tablet_manager.cc:1403] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.114s	user 0.008s	sys 0.003s
I20260709 13:11:57.345539  1583 leader_election.cc:290] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:57.348156  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ddfc34a205c43ad81653b5000cbf394" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:57.348768  1209 raft_consensus.cc:2468] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:57.349725  1099 leader_election.cc:304] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:57.349987  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ddfc34a205c43ad81653b5000cbf394" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:57.245200  1501 raft_consensus.cc:515] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.350649  1061 raft_consensus.cc:2468] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:57.345204  1257 raft_consensus.cc:359] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.351753  1257 raft_consensus.cc:385] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.351538  1473 raft_consensus.cc:2804] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:57.352167  1257 raft_consensus.cc:740] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.352486  1473 raft_consensus.cc:493] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:57.352926  1473 raft_consensus.cc:3060] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.353807   983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12df324e933f422c9a8ff5d37eb7d4ef" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:57.354436   983 raft_consensus.cc:3060] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.354560  1255 raft_consensus.cc:359] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:57.355958  1255 raft_consensus.cc:385] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.356305  1255 raft_consensus.cc:740] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.357049  1255 consensus_queue.cc:260] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:57.360680  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12df324e933f422c9a8ff5d37eb7d4ef" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:57.357925  1257 consensus_queue.cc:260] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.365554  1209 raft_consensus.cc:3060] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.364225  1501 leader_election.cc:290] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:57.367017  1257 ts_tablet_manager.cc:1434] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.004s
I20260709 13:11:57.368076  1257 tablet_bootstrap.cc:492] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:57.374614  1257 tablet_bootstrap.cc:654] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.375563  1254 tablet_bootstrap.cc:654] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.386204  1256 tablet_bootstrap.cc:492] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:57.386705  1256 ts_tablet_manager.cc:1403] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.217s	user 0.072s	sys 0.070s
I20260709 13:11:57.388777  1551 raft_consensus.cc:493] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:57.389317  1551 raft_consensus.cc:515] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.391328  1551 leader_election.cc:290] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:57.391718  1538 consensus_queue.cc:1048] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:57.394397  1206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:57.395115  1206 raft_consensus.cc:2468] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:57.395275  1584 raft_consensus.cc:515] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.395663  1254 tablet_bootstrap.cc:492] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:57.396495  1254 ts_tablet_manager.cc:1403] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.060s	user 0.007s	sys 0.011s
I20260709 13:11:57.396397  1099 leader_election.cc:304] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:57.397500  1579 raft_consensus.cc:493] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:57.397989  1579 raft_consensus.cc:515] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.398334  1551 raft_consensus.cc:2804] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:57.398734  1551 raft_consensus.cc:493] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:57.399080  1551 raft_consensus.cc:3060] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.400003  1579 leader_election.cc:290] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:57.402365  1473 raft_consensus.cc:515] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.405891  1206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:57.406337  1209 raft_consensus.cc:2468] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:57.406638  1206 raft_consensus.cc:2468] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:57.407814  1025 leader_election.cc:304] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:57.408550  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:57.415933  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ddfc34a205c43ad81653b5000cbf394" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:57.430441  1059 raft_consensus.cc:3060] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.409428  1254 raft_consensus.cc:359] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.432044  1254 raft_consensus.cc:385] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.433588  1254 raft_consensus.cc:740] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.434723  1580 raft_consensus.cc:2804] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:57.439954  1580 raft_consensus.cc:697] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:11:57.441238  1580 consensus_queue.cc:237] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.451076  1473 leader_election.cc:290] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:57.452586  1254 consensus_queue.cc:260] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.426254  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:57.457433  1061 raft_consensus.cc:2468] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:57.460268  1317 raft_consensus.cc:493] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:57.460755  1317 raft_consensus.cc:515] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.462400  1349 raft_consensus.cc:493] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:57.462838  1317 leader_election.cc:290] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:57.462958  1349 raft_consensus.cc:515] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.465408   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8c8635d8e54625af9c95ecbcd5653b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:57.465875  1349 leader_election.cc:290] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:57.467664  1254 ts_tablet_manager.cc:1434] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.070s	user 0.006s	sys 0.001s
I20260709 13:11:57.470727  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
W20260709 13:11:57.472147  1173 leader_election.cc:343] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:57.481580  1538 consensus_queue.cc:1048] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:57.480786  1254 tablet_bootstrap.cc:492] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:57.493719  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ddfc34a205c43ad81653b5000cbf394" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:57.494143  1317 raft_consensus.cc:493] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:57.494356  1209 raft_consensus.cc:3060] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.494706  1317 raft_consensus.cc:515] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:57.496917  1317 leader_election.cc:290] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:57.502624  1255 ts_tablet_manager.cc:1434] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.156s	user 0.007s	sys 0.000s
I20260709 13:11:57.510322  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "483b49446c4e4a23b780921d7fabfbdf" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:57.511066  1061 raft_consensus.cc:2468] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:57.445451  1256 raft_consensus.cc:359] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:57.435362  1025 leader_election.cc:304] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:57.467846  1208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8c8635d8e54625af9c95ecbcd5653b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:57.514655  1208 raft_consensus.cc:2468] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
W20260709 13:11:57.468276  1101 leader_election.cc:343] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:11:57.516206  1101 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 47970.7us
I20260709 13:11:57.469398  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:57.517972   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "483b49446c4e4a23b780921d7fabfbdf" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:57.518980  1578 raft_consensus.cc:2804] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:57.519488  1578 raft_consensus.cc:493] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:57.519850  1578 raft_consensus.cc:3060] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:11:57.524417  1173 leader_election.cc:343] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:57.525087  1173 leader_election.cc:304] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:57.529389  1349 raft_consensus.cc:2749] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:57.518868   987 raft_consensus.cc:2468] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:57.531723   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b75f9192e7246d2a560fd091fb68606" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
W20260709 13:11:57.534751  1175 leader_election.cc:343] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:57.537503  1349 raft_consensus.cc:493] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:57.538082  1349 raft_consensus.cc:515] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.540525  1349 leader_election.cc:290] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:57.541747   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:57.543246  1175 leader_election.cc:343] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:57.550143  1099 leader_election.cc:304] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:11:57.551134  1584 leader_election.cc:290] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:57.554263  1255 tablet_bootstrap.cc:492] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
W20260709 13:11:57.518880  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 83603.6us
I20260709 13:11:57.562878  1101 leader_election.cc:304] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:57.567329  1317 raft_consensus.cc:2804] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:57.567723  1317 raft_consensus.cc:493] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:57.569831  1317 raft_consensus.cc:3060] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.514106  1256 raft_consensus.cc:385] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.570974  1256 raft_consensus.cc:740] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
W20260709 13:11:57.571060  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 54181.7us
I20260709 13:11:57.572031  1583 raft_consensus.cc:2804] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:57.439005   983 raft_consensus.cc:2468] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:57.573441  1349 raft_consensus.cc:493] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:57.574365  1349 raft_consensus.cc:515] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.579492  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:11:57.580897  1173 leader_election.cc:343] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:57.574936  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b75f9192e7246d2a560fd091fb68606" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:57.572949  1583 raft_consensus.cc:493] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:57.582643  1062 raft_consensus.cc:3060] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.581230  1349 leader_election.cc:290] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:57.583329  1583 raft_consensus.cc:3060] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.571660  1256 consensus_queue.cc:260] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:57.592146  1256 ts_tablet_manager.cc:1434] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.205s	user 0.007s	sys 0.000s
I20260709 13:11:57.600507  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:57.552613  1578 raft_consensus.cc:515] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.607609  1254 tablet_bootstrap.cc:700] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485: Time spent opening tablet: real 0.101s	user 0.002s	sys 0.000s
I20260709 13:11:57.608165  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
W20260709 13:11:57.609822  1173 leader_election.cc:343] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:57.610337  1173 leader_election.cc:304] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:57.610105  1254 tablet_bootstrap.cc:654] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.616649  1256 tablet_bootstrap.cc:492] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:57.613996  1349 raft_consensus.cc:2749] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:57.615885  1208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:57.617504  1208 raft_consensus.cc:3060] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:11:57.619915  1101 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 57142.7us
I20260709 13:11:57.604854  1578 leader_election.cc:290] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:57.628425  1208 raft_consensus.cc:2468] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
W20260709 13:11:57.631194  1173 leader_election.cc:343] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:57.632946  1551 raft_consensus.cc:515] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.633800  1059 raft_consensus.cc:2468] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:57.631245  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:57.635891  1349 raft_consensus.cc:493] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:57.653326  1257 tablet_bootstrap.cc:492] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:57.646176  1629 consensus_queue.cc:1048] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260709 13:11:57.659142  1257 ts_tablet_manager.cc:1403] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.291s	user 0.123s	sys 0.122s
I20260709 13:11:57.658397  1349 raft_consensus.cc:515] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.661213  1136 raft_consensus.cc:2393] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 1: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:11:57.662350   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21783150bb0b4141be797e841c3f46fa" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:57.662955  1257 raft_consensus.cc:359] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
W20260709 13:11:57.663691  1175 leader_election.cc:343] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:57.665078  1349 leader_election.cc:290] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:57.635089  1317 raft_consensus.cc:515] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:57.663717  1257 raft_consensus.cc:385] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.666676  1257 raft_consensus.cc:740] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.666918  1317 leader_election.cc:290] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:57.667024  1630 consensus_queue.cc:1048] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:57.667914  1134 raft_consensus.cc:2393] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 1: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:11:57.667501  1257 consensus_queue.cc:260] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.645740  1209 raft_consensus.cc:2468] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:57.670123  1257 ts_tablet_manager.cc:1434] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.001s
I20260709 13:11:57.670727  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:57.671602  1059 raft_consensus.cc:2393] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 1: Already voted for candidate 9c45b6d917734835a8a37b0f4c789ae6 in this term.
I20260709 13:11:57.673606  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:57.670742  1173 leader_election.cc:304] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:57.640388  1255 tablet_bootstrap.cc:654] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.676524  1584 raft_consensus.cc:2749] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:57.678100  1257 tablet_bootstrap.cc:492] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d: Bootstrap starting.
I20260709 13:11:57.678907  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "483b49446c4e4a23b780921d7fabfbdf" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:57.679704  1059 raft_consensus.cc:3060] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.680914   983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "483b49446c4e4a23b780921d7fabfbdf" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:57.681607   983 raft_consensus.cc:3060] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.687644  1257 tablet_bootstrap.cc:654] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.689899  1099 leader_election.cc:304] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:57.689945   983 raft_consensus.cc:2468] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:57.680369  1062 raft_consensus.cc:2468] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:11:57.694481  1630 raft_consensus.cc:2804] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:57.695483  1630 raft_consensus.cc:697] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:11:57.663686  1551 leader_election.cc:290] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:57.697404  1630 consensus_queue.cc:237] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
W20260709 13:11:57.693732  1173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 62570.7us
I20260709 13:11:57.702194  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21783150bb0b4141be797e841c3f46fa" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:57.706317  1173 leader_election.cc:304] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:11:57.707286  1349 raft_consensus.cc:2804] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:57.707814  1349 raft_consensus.cc:697] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:11:57.708635  1349 consensus_queue.cc:237] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
W20260709 13:11:57.714576  1173 leader_election.cc:343] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:57.717275  1173 leader_election.cc:304] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:57.734644  1348 raft_consensus.cc:2749] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:57.733495   880 catalog_manager.cc:5697] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:57.735575  1348 raft_consensus.cc:493] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:57.719360  1025 leader_election.cc:304] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:57.736096  1348 raft_consensus.cc:515] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.715919  1255 tablet_bootstrap.cc:492] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:57.737948  1255 ts_tablet_manager.cc:1403] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.184s	user 0.013s	sys 0.000s
I20260709 13:11:57.738433  1348 leader_election.cc:290] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:57.739164  1254 tablet_bootstrap.cc:492] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:57.687639  1059 raft_consensus.cc:2468] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:57.739866   983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:57.741299  1175 leader_election.cc:343] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:57.723310  1101 leader_election.cc:304] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:57.742465  1551 raft_consensus.cc:2804] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:57.742996  1551 raft_consensus.cc:697] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:11:57.743233  1583 raft_consensus.cc:515] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.743949  1551 consensus_queue.cc:237] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:57.718019  1256 tablet_bootstrap.cc:654] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.746968  1099 leader_election.cc:304] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:57.747977  1631 raft_consensus.cc:2804] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:57.750793  1629 raft_consensus.cc:2749] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:11:57.754984  1255 raft_consensus.cc:359] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.755750  1255 raft_consensus.cc:385] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.756011  1255 raft_consensus.cc:740] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.757448  1631 raft_consensus.cc:697] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:11:57.766407  1256 tablet_bootstrap.cc:492] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:57.767050  1256 ts_tablet_manager.cc:1403] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.151s	user 0.008s	sys 0.004s
I20260709 13:11:57.739926  1254 ts_tablet_manager.cc:1403] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.259s	user 0.013s	sys 0.000s
I20260709 13:11:57.773830  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:57.775306  1257 tablet_bootstrap.cc:492] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d: No bootstrap required, opened a new log
I20260709 13:11:57.775821  1257 ts_tablet_manager.cc:1403] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d: Time spent bootstrapping tablet: real 0.098s	user 0.027s	sys 0.056s
W20260709 13:11:57.778967  1173 leader_election.cc:343] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:57.779529  1173 leader_election.cc:304] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:57.780664  1584 raft_consensus.cc:2749] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:57.779016  1256 raft_consensus.cc:359] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
W20260709 13:11:57.781812   950 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 53900.9us
I20260709 13:11:57.781965  1256 raft_consensus.cc:385] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.783586  1256 raft_consensus.cc:740] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.758477  1631 consensus_queue.cc:237] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.784353  1256 consensus_queue.cc:260] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.787189  1208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8c8635d8e54625af9c95ecbcd5653b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:57.788026  1208 raft_consensus.cc:3060] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.788609  1254 raft_consensus.cc:359] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.789374  1254 raft_consensus.cc:385] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.789661  1254 raft_consensus.cc:740] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.789752  1257 raft_consensus.cc:359] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.790552  1257 raft_consensus.cc:385] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.790477  1254 consensus_queue.cc:260] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.790926  1257 raft_consensus.cc:740] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.792822  1257 consensus_queue.cc:260] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.768227   885 catalog_manager.cc:5697] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 0 to 1, leader changed from <none> to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 1 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:57.799701  1208 raft_consensus.cc:2468] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:57.756672  1255 consensus_queue.cc:260] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.806605  1099 leader_election.cc:304] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:57.807590  1551 raft_consensus.cc:2804] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:57.808058  1551 raft_consensus.cc:697] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:11:57.808849  1551 consensus_queue.cc:237] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.820351  1255 ts_tablet_manager.cc:1434] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.082s	user 0.006s	sys 0.000s
I20260709 13:11:57.825703  1254 ts_tablet_manager.cc:1434] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.055s	user 0.006s	sys 0.001s
I20260709 13:11:57.804170  1257 ts_tablet_manager.cc:1434] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d: Time spent starting tablet: real 0.028s	user 0.006s	sys 0.000s
I20260709 13:11:57.804957   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8c8635d8e54625af9c95ecbcd5653b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:57.786578  1583 leader_election.cc:290] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:57.826661   987 raft_consensus.cc:3060] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.825770  1255 tablet_bootstrap.cc:492] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:57.833042  1254 tablet_bootstrap.cc:492] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:57.803184  1256 ts_tablet_manager.cc:1434] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.036s	user 0.006s	sys 0.001s
I20260709 13:11:57.834756  1256 tablet_bootstrap.cc:492] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:57.837080   987 raft_consensus.cc:2468] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:57.840104  1254 tablet_bootstrap.cc:654] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.843595  1255 tablet_bootstrap.cc:654] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.843595  1256 tablet_bootstrap.cc:654] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.847918  1254 tablet_bootstrap.cc:492] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:57.848453  1254 ts_tablet_manager.cc:1403] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.016s	user 0.009s	sys 0.003s
I20260709 13:11:57.858371  1254 raft_consensus.cc:359] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:57.859483  1254 raft_consensus.cc:385] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.859839  1254 raft_consensus.cc:740] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.860638  1254 consensus_queue.cc:260] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:57.862987  1254 ts_tablet_manager.cc:1434] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.002s
I20260709 13:11:57.864039  1254 tablet_bootstrap.cc:492] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:57.864368  1256 tablet_bootstrap.cc:492] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:57.870276  1254 tablet_bootstrap.cc:654] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.880266  1256 ts_tablet_manager.cc:1403] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.046s	user 0.012s	sys 0.003s
I20260709 13:11:57.891677  1255 tablet_bootstrap.cc:492] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:57.892261  1255 ts_tablet_manager.cc:1403] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.067s	user 0.015s	sys 0.033s
I20260709 13:11:57.895267  1255 raft_consensus.cc:359] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:57.895978  1255 raft_consensus.cc:385] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.896245  1255 raft_consensus.cc:740] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.896991  1255 consensus_queue.cc:260] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:57.899158  1255 ts_tablet_manager.cc:1434] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:11:57.887676   879 catalog_manager.cc:5697] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:57.903163  1255 tablet_bootstrap.cc:492] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:57.911099  1255 tablet_bootstrap.cc:654] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.884928  1256 raft_consensus.cc:359] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.918059  1551 raft_consensus.cc:493] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:57.919324  1551 raft_consensus.cc:515] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:57.922156  1551 leader_election.cc:290] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:57.924207  1256 raft_consensus.cc:385] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.924563  1256 raft_consensus.cc:740] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.924403  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867d95d9806478ba50d324c565446f6" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:57.925221  1060 raft_consensus.cc:2468] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:57.926258   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867d95d9806478ba50d324c565446f6" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:57.926463  1099 leader_election.cc:304] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:57.927001   987 raft_consensus.cc:2468] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:57.927348  1551 raft_consensus.cc:2804] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:57.927769  1551 raft_consensus.cc:493] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:57.926226  1255 tablet_bootstrap.cc:492] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:57.928141  1551 raft_consensus.cc:3060] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.928725  1255 ts_tablet_manager.cc:1403] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.026s	user 0.005s	sys 0.013s
I20260709 13:11:57.931676  1255 raft_consensus.cc:359] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.932605  1255 raft_consensus.cc:385] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:57.932916  1255 raft_consensus.cc:740] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:57.933714  1255 consensus_queue.cc:260] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:57.937069   883 catalog_manager.cc:5697] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:57.940495  1255 ts_tablet_manager.cc:1434] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 13:11:57.956218  1255 tablet_bootstrap.cc:492] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:57.943917  1551 raft_consensus.cc:515] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:57.925376  1256 consensus_queue.cc:260] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:57.965044  1256 ts_tablet_manager.cc:1434] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.084s	user 0.005s	sys 0.003s
I20260709 13:11:57.966529  1551 leader_election.cc:290] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:57.966805  1256 tablet_bootstrap.cc:492] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:57.970638  1255 tablet_bootstrap.cc:654] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:57.972659  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867d95d9806478ba50d324c565446f6" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:57.973331  1060 raft_consensus.cc:3060] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.976384  1348 raft_consensus.cc:493] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:57.977093  1348 raft_consensus.cc:515] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:57.982457  1501 consensus_queue.cc:1048] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:57.987015   983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867d95d9806478ba50d324c565446f6" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:57.987960   983 raft_consensus.cc:3060] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:57.997354   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:58.003243  1175 leader_election.cc:343] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:58.012593  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:11:58.014935  1173 leader_election.cc:343] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:58.015614  1173 leader_election.cc:304] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:58.016877  1584 raft_consensus.cc:2749] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:58.018065  1628 consensus_queue.cc:1048] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:58.019232  1348 leader_election.cc:290] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:58.027122  1631 raft_consensus.cc:493] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:58.027729  1631 raft_consensus.cc:515] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:58.030544  1631 leader_election.cc:290] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:58.033502  1208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:58.034260  1208 raft_consensus.cc:2468] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:58.032668  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:58.035691  1136 raft_consensus.cc:2468] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:58.036986  1025 leader_election.cc:304] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:58.037990   883 catalog_manager.cc:5697] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:58.044063  1631 raft_consensus.cc:2804] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:58.045333  1631 raft_consensus.cc:493] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:58.049337  1631 raft_consensus.cc:3060] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:58.040690   883 catalog_manager.cc:5697] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:58.077385  1256 tablet_bootstrap.cc:654] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:58.093003  1254 tablet_bootstrap.cc:492] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:58.093676  1254 ts_tablet_manager.cc:1403] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.230s	user 0.071s	sys 0.105s
I20260709 13:11:58.096845  1254 raft_consensus.cc:359] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:58.097654  1254 raft_consensus.cc:385] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:58.097987  1254 raft_consensus.cc:740] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:58.098802  1254 consensus_queue.cc:260] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:58.101042  1254 ts_tablet_manager.cc:1434] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:11:58.102516  1254 tablet_bootstrap.cc:492] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:58.109313  1254 tablet_bootstrap.cc:654] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:58.109675   983 raft_consensus.cc:2468] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:58.112003  1101 leader_election.cc:304] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:58.083209  1060 raft_consensus.cc:2468] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:58.115995  1551 raft_consensus.cc:2804] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:58.116772  1551 raft_consensus.cc:697] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:11:58.117417  1348 raft_consensus.cc:493] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:58.117956  1348 raft_consensus.cc:515] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:58.117864  1551 consensus_queue.cc:237] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:58.123446  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff504aa514be43da8b1aac12ab550daf" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
W20260709 13:11:58.128042  1173 leader_election.cc:343] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:58.130520  1348 leader_election.cc:290] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
W20260709 13:11:58.136492  1175 consensus_peers.cc:597] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:11:58.137531   983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff504aa514be43da8b1aac12ab550daf" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:58.148579  1175 leader_election.cc:343] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:58.149286  1175 leader_election.cc:304] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:58.152101  1584 raft_consensus.cc:2749] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:58.163669  1501 raft_consensus.cc:493] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:58.164474  1501 raft_consensus.cc:515] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:58.170780  1473 consensus_queue.cc:1048] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:58.172410  1256 tablet_bootstrap.cc:492] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:58.173559  1256 ts_tablet_manager.cc:1403] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.208s	user 0.007s	sys 0.007s
I20260709 13:11:58.174484  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:58.175130  1501 leader_election.cc:290] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
W20260709 13:11:58.175668  1025 leader_election.cc:343] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:58.144593  1631 raft_consensus.cc:515] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:58.187206  1208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:58.188110  1208 raft_consensus.cc:2468] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:58.188328  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:58.188964  1134 raft_consensus.cc:3060] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:58.190389  1255 tablet_bootstrap.cc:492] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:58.190903  1255 ts_tablet_manager.cc:1403] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.235s	user 0.097s	sys 0.072s
I20260709 13:11:58.193689  1255 raft_consensus.cc:359] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:58.194387  1255 raft_consensus.cc:385] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:58.194686  1255 raft_consensus.cc:740] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:58.179996  1256 raft_consensus.cc:359] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:58.181209  1473 consensus_queue.cc:1048] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:58.181437  1538 raft_consensus.cc:493] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:58.195796  1538 raft_consensus.cc:515] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:58.196905  1134 raft_consensus.cc:2468] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:58.196877  1256 raft_consensus.cc:385] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:58.197947  1256 raft_consensus.cc:740] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:58.198081  1538 leader_election.cc:290] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:58.198160  1255 consensus_queue.cc:260] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:58.198693  1256 consensus_queue.cc:260] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:58.200034  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:58.200181  1025 leader_election.cc:304] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:58.200659  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:58.201639  1501 raft_consensus.cc:2804] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:58.202140  1501 raft_consensus.cc:697] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:11:58.202967  1501 consensus_queue.cc:237] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:58.201443  1134 raft_consensus.cc:2468] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:58.200721  1207 raft_consensus.cc:2468] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:58.207582   950 leader_election.cc:304] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 
I20260709 13:11:58.199590  1254 tablet_bootstrap.cc:492] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:58.208294  1538 raft_consensus.cc:2804] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:58.210675  1538 raft_consensus.cc:493] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:58.211081  1538 raft_consensus.cc:3060] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:58.211333  1255 ts_tablet_manager.cc:1434] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.002s
I20260709 13:11:58.214673  1025 leader_election.cc:304] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:11:58.215801  1628 raft_consensus.cc:2804] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:58.216205  1628 raft_consensus.cc:493] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:58.216722  1256 ts_tablet_manager.cc:1434] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.043s	user 0.000s	sys 0.008s
I20260709 13:11:58.218014  1256 tablet_bootstrap.cc:492] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:58.210897  1254 ts_tablet_manager.cc:1403] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.109s	user 0.028s	sys 0.051s
I20260709 13:11:58.216567  1628 raft_consensus.cc:3060] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:58.221360   885 catalog_manager.cc:5697] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:58.229768  1538 raft_consensus.cc:515] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:58.221181  1254 raft_consensus.cc:359] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:58.232925  1254 raft_consensus.cc:385] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:58.233314  1254 raft_consensus.cc:740] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:58.234742  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:58.235325  1207 raft_consensus.cc:3060] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:58.235430  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:58.236322  1134 raft_consensus.cc:3060] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:58.231750   879 catalog_manager.cc:5697] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:58.234128  1254 consensus_queue.cc:260] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:58.238070  1255 tablet_bootstrap.cc:492] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:58.239812  1254 ts_tablet_manager.cc:1434] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.000s
I20260709 13:11:58.240870  1254 tablet_bootstrap.cc:492] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:58.245618  1255 tablet_bootstrap.cc:654] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:58.246479  1256 tablet_bootstrap.cc:654] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:58.251134  1254 tablet_bootstrap.cc:654] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:58.232142  1538 leader_election.cc:290] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:58.259109  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:58.259713  1209 raft_consensus.cc:3060] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:58.263650  1134 raft_consensus.cc:2468] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:58.265455   950 leader_election.cc:304] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:58.266489  1209 raft_consensus.cc:2468] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:58.252172  1207 raft_consensus.cc:2468] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:58.269341  1528 raft_consensus.cc:2804] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:58.272948  1528 raft_consensus.cc:697] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:11:58.277022  1528 consensus_queue.cc:237] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:58.281015  1584 consensus_queue.cc:1048] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:58.290736  1630 raft_consensus.cc:493] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:58.292469  1630 raft_consensus.cc:515] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:58.291227  1317 consensus_queue.cc:1048] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:58.295073  1628 raft_consensus.cc:515] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:58.297878   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:58.299536  1101 leader_election.cc:343] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:58.300696  1630 leader_election.cc:290] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:58.187237  1631 leader_election.cc:290] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:58.319284  1255 tablet_bootstrap.cc:492] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:58.319976  1255 ts_tablet_manager.cc:1403] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.082s	user 0.009s	sys 0.009s
I20260709 13:11:58.323930  1254 tablet_bootstrap.cc:492] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:58.324466  1254 ts_tablet_manager.cc:1403] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.084s	user 0.015s	sys 0.001s
I20260709 13:11:58.324977  1206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:58.325729  1206 raft_consensus.cc:2468] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:58.325527  1583 consensus_queue.cc:1048] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:58.327837  1254 raft_consensus.cc:359] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:58.328428  1099 leader_election.cc:304] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:11:58.328608  1254 raft_consensus.cc:385] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:58.330453  1254 raft_consensus.cc:740] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:58.333241  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:58.333374  1254 consensus_queue.cc:260] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
W20260709 13:11:58.335047  1025 leader_election.cc:343] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:58.350015  1255 raft_consensus.cc:359] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:58.350842  1255 raft_consensus.cc:385] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:58.351184  1255 raft_consensus.cc:740] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
W20260709 13:11:58.354358   932 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:11:58.358547  1255 consensus_queue.cc:260] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:58.330229  1473 raft_consensus.cc:2804] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:58.362324  1473 raft_consensus.cc:493] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:58.366211  1473 raft_consensus.cc:3060] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:58.385962  1254 ts_tablet_manager.cc:1434] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.061s	user 0.007s	sys 0.001s
I20260709 13:11:58.386965  1628 leader_election.cc:290] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:58.401413  1254 tablet_bootstrap.cc:492] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:58.415447  1256 tablet_bootstrap.cc:492] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:58.416019  1256 ts_tablet_manager.cc:1403] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.198s	user 0.018s	sys 0.018s
I20260709 13:11:58.422899  1254 tablet_bootstrap.cc:654] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:58.428933  1473 raft_consensus.cc:515] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:58.432335  1473 leader_election.cc:290] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:58.435716  1317 consensus_queue.cc:1048] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:58.437628   986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:58.437654  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:58.439247  1207 raft_consensus.cc:3060] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:11:58.451678  1101 leader_election.cc:343] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:58.453085  1256 raft_consensus.cc:359] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:58.453874  1256 raft_consensus.cc:385] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:58.454144  1256 raft_consensus.cc:740] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:58.454840  1256 consensus_queue.cc:260] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:58.459008  1256 ts_tablet_manager.cc:1434] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.043s	user 0.004s	sys 0.002s
I20260709 13:11:58.462668  1256 tablet_bootstrap.cc:492] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:58.470661  1473 consensus_queue.cc:1048] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:58.471596  1256 tablet_bootstrap.cc:654] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
W20260709 13:11:58.478839  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 121043us
I20260709 13:11:58.485831  1628 consensus_queue.cc:1048] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260709 13:11:58.511502  1255 ts_tablet_manager.cc:1434] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.190s	user 0.006s	sys 0.013s
I20260709 13:11:58.504499  1538 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:58.534705  1674 consensus_queue.cc:1048] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.045s
I20260709 13:11:58.556588  1538 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:58.560956  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:58.561914  1210 raft_consensus.cc:3060] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:58.566529  1255 tablet_bootstrap.cc:492] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:58.585850  1254 tablet_bootstrap.cc:492] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:58.586282  1254 ts_tablet_manager.cc:1403] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.189s	user 0.027s	sys 0.024s
W20260709 13:11:58.584831  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 85879.5us
W20260709 13:11:58.587149  1672 log.cc:927] Time spent T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 13:11:58.658061  1254 raft_consensus.cc:359] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:58.658829  1254 raft_consensus.cc:385] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:58.659134  1254 raft_consensus.cc:740] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:58.666894  1254 consensus_queue.cc:260] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:58.695039  1254 ts_tablet_manager.cc:1434] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.109s	user 0.008s	sys 0.000s
I20260709 13:11:58.653574  1207 raft_consensus.cc:2468] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:11:58.690786  1681 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:58.716430  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:58.719295  1254 tablet_bootstrap.cc:492] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:58.738665  1099 leader_election.cc:304] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:11:58.739688  1317 raft_consensus.cc:2804] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:58.740298  1317 raft_consensus.cc:697] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:11:58.741425  1317 consensus_queue.cc:237] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:58.738353  1631 raft_consensus.cc:493] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:58.722932  1681 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:58.791222  1062 raft_consensus.cc:2468] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:58.771872  1631 raft_consensus.cc:515] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:58.794734  1631 leader_election.cc:290] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
W20260709 13:11:58.841310   950 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 83224.3us
W20260709 13:11:58.839200  1688 log.cc:927] Time spent T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 13:11:58.847015  1583 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:58.851713  1695 raft_consensus.cc:493] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:58.865896  1255 tablet_bootstrap.cc:654] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:58.889072  1254 tablet_bootstrap.cc:654] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:58.868186  1695 raft_consensus.cc:515] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:58.891880  1695 leader_election.cc:290] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:58.896793   879 catalog_manager.cc:5697] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 0 to 1, leader changed from <none> to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } }
I20260709 13:11:58.901273  1204 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b75f9192e7246d2a560fd091fb68606" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:58.892781  1210 raft_consensus.cc:2468] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:58.864013  1583 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:58.908888  1583 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:58.916000  1538 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:58.941076  1695 consensus_queue.cc:1048] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:58.946851  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:58.947777  1255 tablet_bootstrap.cc:492] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:58.948504  1255 ts_tablet_manager.cc:1403] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.402s	user 0.017s	sys 0.003s
I20260709 13:11:58.952248  1255 raft_consensus.cc:359] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:58.952929  1255 raft_consensus.cc:385] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:58.954056  1255 raft_consensus.cc:740] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:58.955657  1255 consensus_queue.cc:260] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:58.959208  1206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21783150bb0b4141be797e841c3f46fa" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:58.959879  1206 raft_consensus.cc:2468] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:58.961192  1256 tablet_bootstrap.cc:492] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:58.961429  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b75f9192e7246d2a560fd091fb68606" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:58.962266  1256 ts_tablet_manager.cc:1403] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.500s	user 0.016s	sys 0.016s
W20260709 13:11:58.925567  1101 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 65935.1us
I20260709 13:11:58.964496  1473 consensus_queue.cc:1048] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:58.969853  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:58.970117  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:58.970719  1061 raft_consensus.cc:2468] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:58.974015  1099 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:58.986291   950 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:58.947840  1131 raft_consensus.cc:2468] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 0.
I20260709 13:11:58.987071  1307 raft_consensus.cc:2804] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:58.988409  1307 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:58.988767  1307 raft_consensus.cc:3060] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:58.990286  1700 consensus_queue.cc:1048] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:58.992333  1256 raft_consensus.cc:359] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:58.993088  1256 raft_consensus.cc:385] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:58.993393  1256 raft_consensus.cc:740] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:58.994063  1256 consensus_queue.cc:260] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:58.996459  1256 ts_tablet_manager.cc:1434] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.034s	user 0.005s	sys 0.001s
I20260709 13:11:58.997427  1256 tablet_bootstrap.cc:492] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6: Bootstrap starting.
I20260709 13:11:58.998485  1255 ts_tablet_manager.cc:1434] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.050s	user 0.003s	sys 0.008s
I20260709 13:11:58.999384  1255 tablet_bootstrap.cc:492] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:58.997622  1254 tablet_bootstrap.cc:492] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:59.001451  1254 ts_tablet_manager.cc:1403] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.304s	user 0.032s	sys 0.021s
I20260709 13:11:58.997323  1701 raft_consensus.cc:2804] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:59.008016   950 leader_election.cc:304] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:59.017944  1676 raft_consensus.cc:2749] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:59.018519   983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21783150bb0b4141be797e841c3f46fa" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:59.020905  1307 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:59.022814  1668 consensus_queue.cc:1048] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 13:11:59.026185  1307 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:59.032071  1701 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:59.032572  1701 raft_consensus.cc:3060] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:59.038355   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:59.038570  1025 leader_election.cc:304] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:11:59.039340   985 raft_consensus.cc:2393] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 1: Already voted for candidate 294145900dd3496cb4a1d722613e1558 in this term.
I20260709 13:11:59.042985   980 raft_consensus.cc:3060] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:59.043731  1699 raft_consensus.cc:2804] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:59.044430  1699 raft_consensus.cc:697] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
W20260709 13:11:59.053489  1027 leader_election.cc:343] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:59.054502  1027 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:59.056047  1631 raft_consensus.cc:3060] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:59.065869  1025 leader_election.cc:304] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:11:59.062111  1681 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:11:59.076131  1699 consensus_queue.cc:237] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:59.072028  1501 consensus_queue.cc:1048] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 13:11:59.076177  1693 raft_consensus.cc:2804] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:59.078783  1693 raft_consensus.cc:493] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:59.078639  1701 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:59.079320  1693 raft_consensus.cc:3060] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:59.080715  1701 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:11:59.092629   950 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 77702.8us
W20260709 13:11:59.097357  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 61596.3us
I20260709 13:11:59.102577  1256 tablet_bootstrap.cc:654] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:59.103600  1255 tablet_bootstrap.cc:654] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:59.109998  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:59.122530  1254 raft_consensus.cc:359] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:59.123190  1254 raft_consensus.cc:385] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:59.123557  1254 raft_consensus.cc:740] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:59.104447  1580 consensus_queue.cc:1048] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260709 13:11:59.133428  1254 consensus_queue.cc:260] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:59.137565   981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:59.138751   981 raft_consensus.cc:2393] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 1: Already voted for candidate 294145900dd3496cb4a1d722613e1558 in this term.
I20260709 13:11:59.142313  1254 ts_tablet_manager.cc:1434] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.141s	user 0.005s	sys 0.003s
I20260709 13:11:59.119637  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:59.158646  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:59.155428  1631 raft_consensus.cc:2749] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:59.208173  1133 raft_consensus.cc:2393] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 in current term 1: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:11:59.248119  1062 raft_consensus.cc:2468] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:11:59.248986  1254 tablet_bootstrap.cc:492] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485: Bootstrap starting.
I20260709 13:11:59.256122  1254 tablet_bootstrap.cc:654] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:59.274581  1716 raft_consensus.cc:493] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:59.275156  1716 raft_consensus.cc:515] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:11:59.280100   983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:59.280999   983 raft_consensus.cc:2393] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 1: Already voted for candidate 294145900dd3496cb4a1d722613e1558 in this term.
I20260709 13:11:59.277475  1716 leader_election.cc:290] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
W20260709 13:11:59.284834  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 173762us
I20260709 13:11:59.289543  1693 raft_consensus.cc:515] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:59.313482   950 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:11:59.314266  1099 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:59.332788  1720 raft_consensus.cc:2749] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:11:59.321094  1676 raft_consensus.cc:2804] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:59.339349  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:59.340358  1255 tablet_bootstrap.cc:492] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:59.340879  1255 ts_tablet_manager.cc:1403] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.342s	user 0.011s	sys 0.007s
I20260709 13:11:59.340246  1201 raft_consensus.cc:2393] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 1: Already voted for candidate 9c45b6d917734835a8a37b0f4c789ae6 in this term.
I20260709 13:11:59.343828  1676 raft_consensus.cc:697] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:11:59.346421  1676 consensus_queue.cc:237] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:59.349730   883 catalog_manager.cc:5697] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:59.409722  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21783150bb0b4141be797e841c3f46fa" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:59.410436  1207 raft_consensus.cc:3060] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:59.411605  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:59.408699  1255 raft_consensus.cc:359] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:59.415534  1255 raft_consensus.cc:385] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:59.415930  1255 raft_consensus.cc:740] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:59.416760  1255 consensus_queue.cc:260] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:59.420250  1256 tablet_bootstrap.cc:492] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6: No bootstrap required, opened a new log
I20260709 13:11:59.420995  1256 ts_tablet_manager.cc:1403] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6: Time spent bootstrapping tablet: real 0.424s	user 0.024s	sys 0.038s
I20260709 13:11:59.429173  1551 consensus_queue.cc:1048] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:11:59.457343  1307 raft_consensus.cc:493] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:59.457960  1255 ts_tablet_manager.cc:1434] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.117s	user 0.007s	sys 0.000s
I20260709 13:11:59.459898  1255 tablet_bootstrap.cc:492] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:59.460063  1693 leader_election.cc:290] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:59.458024  1307 raft_consensus.cc:515] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:59.465066  1307 leader_election.cc:290] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:11:59.489472  1256 raft_consensus.cc:359] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:59.492683  1256 raft_consensus.cc:385] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:59.494751   984 raft_consensus.cc:3060] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:59.493112  1256 raft_consensus.cc:740] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Initialized, Role: FOLLOWER
I20260709 13:11:59.505532  1256 consensus_queue.cc:260] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:59.511510  1255 tablet_bootstrap.cc:654] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:59.516124  1254 tablet_bootstrap.cc:492] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485: No bootstrap required, opened a new log
I20260709 13:11:59.516686  1254 ts_tablet_manager.cc:1403] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485: Time spent bootstrapping tablet: real 0.268s	user 0.050s	sys 0.064s
I20260709 13:11:59.524204  1256 ts_tablet_manager.cc:1434] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6: Time spent starting tablet: real 0.103s	user 0.008s	sys 0.000s
W20260709 13:11:59.524384   932 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:11:59.528573  1099 leader_election.cc:304] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:59.516923  1716 raft_consensus.cc:493] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:59.535785  1700 raft_consensus.cc:3060] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:59.530107  1681 raft_consensus.cc:493] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:59.554823  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:59.540939  1681 raft_consensus.cc:515] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:59.558315  1681 leader_election.cc:290] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:59.539727  1716 raft_consensus.cc:515] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:59.565171  1716 leader_election.cc:290] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:59.567076  1254 raft_consensus.cc:359] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:59.567909  1254 raft_consensus.cc:385] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:59.568239  1254 raft_consensus.cc:740] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Initialized, Role: FOLLOWER
I20260709 13:11:59.569114  1254 consensus_queue.cc:260] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:59.591557  1204 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:59.598616   983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21783150bb0b4141be797e841c3f46fa" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:59.601749   985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:59.691793   950 leader_election.cc:304] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:11:59.692380  1254 ts_tablet_manager.cc:1434] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485: Time spent starting tablet: real 0.175s	user 0.011s	sys 0.000s
I20260709 13:11:59.677402   983 raft_consensus.cc:3060] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:59.677100   980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:59.712152  1686 raft_consensus.cc:493] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:59.712813  1686 raft_consensus.cc:515] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:59.715155  1686 leader_election.cc:290] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:59.722905  1207 raft_consensus.cc:2468] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:59.727861  1528 raft_consensus.cc:2749] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:59.679828  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
W20260709 13:11:59.748045  1027 leader_election.cc:343] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:59.768179  1700 raft_consensus.cc:2749] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:59.756727  1201 raft_consensus.cc:2468] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 0.
I20260709 13:11:59.798090  1025 leader_election.cc:304] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:59.798750  1629 raft_consensus.cc:493] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:59.804502  1671 raft_consensus.cc:493] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:11:59.805035  1671 raft_consensus.cc:515] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
W20260709 13:11:59.806993   950 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 56107.4us
I20260709 13:11:59.807126  1025 leader_election.cc:304] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:11:59.803354  1629 raft_consensus.cc:515] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
W20260709 13:11:59.789012  1101 leader_election.cc:343] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:59.812575  1255 tablet_bootstrap.cc:492] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:59.814409  1255 ts_tablet_manager.cc:1403] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.356s	user 0.027s	sys 0.004s
I20260709 13:11:59.803309  1501 raft_consensus.cc:2804] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:59.828073  1629 leader_election.cc:290] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:59.831190  1671 leader_election.cc:290] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:59.831354  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff504aa514be43da8b1aac12ab550daf" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:11:59.832161  1132 raft_consensus.cc:2468] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 0.
I20260709 13:11:59.836251   981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:59.839624  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:59.840286  1062 raft_consensus.cc:2468] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 0.
I20260709 13:11:59.841325  1501 raft_consensus.cc:697] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:11:59.821945  1697 raft_consensus.cc:2804] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:59.844465  1501 consensus_queue.cc:237] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:59.844928  1697 raft_consensus.cc:493] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:59.845320  1697 raft_consensus.cc:3060] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:59.855890  1173 leader_election.cc:304] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:59.849960  1473 consensus_queue.cc:1048] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 13:11:59.857625  1683 raft_consensus.cc:2804] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:59.858129  1683 raft_consensus.cc:493] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:59.858464  1683 raft_consensus.cc:3060] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:59.860405   983 raft_consensus.cc:2468] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:59.863056  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:11:59.870651  1683 raft_consensus.cc:515] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:59.884029  1683 leader_election.cc:290] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:59.885373  1099 leader_election.cc:304] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:11:59.886371  1700 raft_consensus.cc:2804] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:11:59.886767  1700 raft_consensus.cc:493] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:11:59.887091  1700 raft_consensus.cc:3060] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:59.884785   980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff504aa514be43da8b1aac12ab550daf" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:11:59.871793  1175 leader_election.cc:343] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:59.856185  1255 raft_consensus.cc:359] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:59.892817  1255 raft_consensus.cc:385] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:11:59.893168  1255 raft_consensus.cc:740] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:11:59.894080  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff504aa514be43da8b1aac12ab550daf" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:11:59.894758  1135 raft_consensus.cc:3060] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:11:59.896354  1101 leader_election.cc:343] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:59.899869  1255 consensus_queue.cc:260] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:59.907445  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff504aa514be43da8b1aac12ab550daf" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:11:59.908365   983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff504aa514be43da8b1aac12ab550daf" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:11:59.908334  1202 raft_consensus.cc:2393] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 1: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
I20260709 13:11:59.913363   885 catalog_manager.cc:5697] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:59.914681  1697 raft_consensus.cc:515] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:59.915956  1062 raft_consensus.cc:2393] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 1: Already voted for candidate 9c45b6d917734835a8a37b0f4c789ae6 in this term.
I20260709 13:11:59.918599  1173 leader_election.cc:304] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:11:59.919261  1700 raft_consensus.cc:515] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:59.919939  1584 raft_consensus.cc:3060] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:59.925866  1255 ts_tablet_manager.cc:1434] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.111s	user 0.007s	sys 0.000s
I20260709 13:11:59.926981  1099 leader_election.cc:304] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:11:59.927405  1700 leader_election.cc:290] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:11:59.928349  1255 tablet_bootstrap.cc:492] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:11:59.930591  1584 raft_consensus.cc:2749] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:59.935160  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:11:59.939002  1135 raft_consensus.cc:2468] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:11:59.946746  1062 raft_consensus.cc:3060] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:59.947132  1717 raft_consensus.cc:2749] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:11:59.950724   980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff504aa514be43da8b1aac12ab550daf" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:59.940320  1697 leader_election.cc:290] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:11:59.942204  1255 tablet_bootstrap.cc:654] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:11:59.954347  1173 leader_election.cc:304] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:59.956151  1584 raft_consensus.cc:2804] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:59.958249  1584 raft_consensus.cc:697] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:11:59.956822   981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:59.959089  1584 consensus_queue.cc:237] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:59.959353   981 raft_consensus.cc:3060] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:11:59.963891  1695 consensus_queue.cc:1048] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:11:59.957144  1062 raft_consensus.cc:2468] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
W20260709 13:11:59.967474  1175 leader_election.cc:343] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:59.967555  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:11:59.969462   883 catalog_manager.cc:5697] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 0 to 1, leader changed from <none> to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:11:59.970955  1099 leader_election.cc:304] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:11:59.971311   987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:11:59.971839  1717 raft_consensus.cc:2804] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:59.972348  1717 raft_consensus.cc:697] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:11:59.973289  1717 consensus_queue.cc:237] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:11:59.980882  1255 tablet_bootstrap.cc:492] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:11:59.981457  1255 ts_tablet_manager.cc:1403] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.055s	user 0.010s	sys 0.001s
W20260709 13:11:59.981724  1175 leader_election.cc:343] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:59.975337  1207 raft_consensus.cc:2468] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:59.987262  1631 consensus_queue.cc:1048] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:11:59.987972  1101 leader_election.cc:343] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:11:59.990837  1255 raft_consensus.cc:359] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:11:59.991575   981 raft_consensus.cc:2468] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:11:59.992869  1027 leader_election.cc:304] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:11:59.998031  1697 raft_consensus.cc:2804] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:11:59.998735  1697 raft_consensus.cc:697] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:11:59.999604  1697 consensus_queue.cc:237] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:11:59.991602  1255 raft_consensus.cc:385] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:00.004981  1255 raft_consensus.cc:740] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:00.006628  1538 consensus_queue.cc:1048] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:00.008009  1255 consensus_queue.cc:260] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:00.011111  1501 consensus_queue.cc:1048] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:00.014075  1255 ts_tablet_manager.cc:1434] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.032s	user 0.006s	sys 0.001s
I20260709 13:12:00.015151  1255 tablet_bootstrap.cc:492] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:00.023206  1255 tablet_bootstrap.cc:654] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:00.066823  1255 tablet_bootstrap.cc:492] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558: No bootstrap required, opened a new log
I20260709 13:12:00.067582  1255 ts_tablet_manager.cc:1403] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.053s	user 0.022s	sys 0.011s
I20260709 13:12:00.069520  1255 raft_consensus.cc:359] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:00.069983  1255 raft_consensus.cc:385] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:00.070148  1255 raft_consensus.cc:740] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:00.070704  1255 consensus_queue.cc:260] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:00.074312  1255 ts_tablet_manager.cc:1434] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:12:00.102782   883 catalog_manager.cc:5697] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 0 to 1, leader changed from <none> to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 1 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:00.123672   885 catalog_manager.cc:5697] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:00.125855   885 catalog_manager.cc:5697] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:00.147567   880 catalog_manager.cc:5697] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:00.264341 24113 tablet_server.cc:179] TabletServer@127.23.140.65:0 shutting down...
I20260709 13:12:00.311512 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260709 13:12:00.311582  1101 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:12:00.315263 24113 tablet_replica.cc:333] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.316412 24113 raft_consensus.cc:2243] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.317096 24113 raft_consensus.cc:2272] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:00.317173  1101 consensus_peers.cc:597] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:00.318857  1027 consensus_peers.cc:597] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:00.319818 24113 tablet_replica.cc:333] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.320449 24113 raft_consensus.cc:2243] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.320837 24113 raft_consensus.cc:2272] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.322887 24113 tablet_replica.cc:333] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.323654 24113 raft_consensus.cc:2243] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.324087 24113 raft_consensus.cc:2272] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.326681 24113 tablet_replica.cc:333] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.327250 24113 raft_consensus.cc:2243] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:00.328075 24113 raft_consensus.cc:2272] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.330636 24113 tablet_replica.cc:333] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.331274 24113 raft_consensus.cc:2243] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.331670 24113 raft_consensus.cc:2272] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.333437 24113 tablet_replica.cc:333] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.333966 24113 raft_consensus.cc:2243] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:00.334757 24113 raft_consensus.cc:2272] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.336421 24113 tablet_replica.cc:333] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.337073 24113 raft_consensus.cc:2243] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.337522 24113 raft_consensus.cc:2272] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.339681 24113 tablet_replica.cc:333] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.340270 24113 raft_consensus.cc:2243] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.341586 24113 raft_consensus.cc:2272] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.343905 24113 tablet_replica.cc:333] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
W20260709 13:12:00.344323  1027 consensus_peers.cc:597] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:00.344535 24113 raft_consensus.cc:2243] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.345126 24113 raft_consensus.cc:2272] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.346946 24113 tablet_replica.cc:333] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.347543 24113 raft_consensus.cc:2243] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.347946 24113 raft_consensus.cc:2272] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.350327 24113 tablet_replica.cc:333] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.351169 24113 raft_consensus.cc:2243] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.351634 24113 raft_consensus.cc:2272] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:00.353237  1101 consensus_peers.cc:597] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:00.353904 24113 tablet_replica.cc:333] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.354779 24113 raft_consensus.cc:2243] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.355247 24113 raft_consensus.cc:2272] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.357583 24113 tablet_replica.cc:333] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.358353 24113 raft_consensus.cc:2243] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.358914 24113 raft_consensus.cc:2272] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:00.360332  1027 consensus_peers.cc:597] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:00.361239 24113 tablet_replica.cc:333] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.362097 24113 raft_consensus.cc:2243] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:00.363018 24113 raft_consensus.cc:2272] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.365361 24113 tablet_replica.cc:333] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.366108 24113 raft_consensus.cc:2243] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.366549  1738 raft_consensus.cc:493] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:00.367041  1738 raft_consensus.cc:515] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:00.369095  1738 leader_election.cc:290] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:00.370486 24113 raft_consensus.cc:2272] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.370594  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff504aa514be43da8b1aac12ab550daf" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:00.371814  1135 raft_consensus.cc:2393] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 in current term 1: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
I20260709 13:12:00.373083 24113 tablet_replica.cc:333] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.374154 24113 raft_consensus.cc:2243] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:00.375165 24113 raft_consensus.cc:2272] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.377684 24113 tablet_replica.cc:333] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.378464 24113 raft_consensus.cc:2243] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:00.379365 24113 raft_consensus.cc:2272] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:00.380854  1175 consensus_peers.cc:597] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:00.381847 24113 tablet_replica.cc:333] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.382447  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff504aa514be43da8b1aac12ab550daf" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:00.382735 24113 raft_consensus.cc:2243] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.383263 24113 raft_consensus.cc:2272] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.384011   950 leader_election.cc:304] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:00.385072  1738 raft_consensus.cc:3060] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:00.385622 24113 tablet_replica.cc:333] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.386502 24113 raft_consensus.cc:2243] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.387112 24113 raft_consensus.cc:2272] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.389752 24113 tablet_replica.cc:333] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.390568 24113 raft_consensus.cc:2243] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.391083 24113 raft_consensus.cc:2272] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.393414 24113 tablet_replica.cc:333] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.394145 24113 raft_consensus.cc:2243] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:00.394644  1738 raft_consensus.cc:2749] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:12:00.395069 24113 raft_consensus.cc:2272] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.397574 24113 tablet_replica.cc:333] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.398377 24113 raft_consensus.cc:2243] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.398837 24113 raft_consensus.cc:2272] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.399219  1631 consensus_queue.cc:1048] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:00.401075 24113 tablet_replica.cc:333] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
W20260709 13:12:00.401149  1175 consensus_peers.cc:597] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:00.401221  1101 consensus_peers.cc:597] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:00.402091 24113 raft_consensus.cc:2243] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.402645 24113 raft_consensus.cc:2272] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.405074 24113 tablet_replica.cc:333] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.405906 24113 raft_consensus.cc:2243] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:00.406862 24113 raft_consensus.cc:2272] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:00.407153  1101 consensus_peers.cc:597] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:00.408524  1101 consensus_peers.cc:597] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:00.409304  1101 consensus_peers.cc:597] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:00.409467 24113 tablet_replica.cc:333] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.410481 24113 raft_consensus.cc:2243] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:00.411437 24113 raft_consensus.cc:2272] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:00.412184  1027 consensus_peers.cc:597] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:00.414165 24113 tablet_replica.cc:333] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.414930 24113 raft_consensus.cc:2243] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:00.415823 24113 raft_consensus.cc:2272] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:00.417878  1175 consensus_peers.cc:597] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:00.418164 24113 tablet_replica.cc:333] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.419018 24113 raft_consensus.cc:2243] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:00.419961 24113 raft_consensus.cc:2272] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.422251 24113 tablet_replica.cc:333] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.423014 24113 raft_consensus.cc:2243] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 13:12:00.423218  1175 consensus_peers.cc:597] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:00.423482 24113 raft_consensus.cc:2272] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.425958 24113 tablet_replica.cc:333] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.426792 24113 raft_consensus.cc:2243] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:00.427845 24113 raft_consensus.cc:2272] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.427942  1317 consensus_queue.cc:1048] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:00.430416 24113 tablet_replica.cc:333] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.431059 24113 raft_consensus.cc:2243] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.431634 24113 raft_consensus.cc:2272] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.433985 24113 tablet_replica.cc:333] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.434706 24113 raft_consensus.cc:2243] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.435153 24113 raft_consensus.cc:2272] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.437615 24113 tablet_replica.cc:333] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.438670 24113 raft_consensus.cc:2243] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.439414 24113 raft_consensus.cc:2272] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.441998 24113 tablet_replica.cc:333] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.442736 24113 raft_consensus.cc:2243] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.443192 24113 raft_consensus.cc:2272] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:00.443444  1175 consensus_peers.cc:597] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:00.446738  1027 consensus_peers.cc:597] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:00.446967 24113 tablet_replica.cc:333] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.447940 24113 raft_consensus.cc:2243] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.448433 24113 raft_consensus.cc:2272] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.451506 24113 tablet_replica.cc:333] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.452275 24113 raft_consensus.cc:2243] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.452807 24113 raft_consensus.cc:2272] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.457252 24113 tablet_replica.cc:333] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.458019 24113 raft_consensus.cc:2243] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.458555 24113 raft_consensus.cc:2272] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.461319 24113 tablet_replica.cc:333] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.462073 24113 raft_consensus.cc:2243] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:00.463019 24113 raft_consensus.cc:2272] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.465595 24113 tablet_replica.cc:333] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.466311 24113 raft_consensus.cc:2243] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.466830 24113 raft_consensus.cc:2272] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:00.468220  1101 consensus_peers.cc:597] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:00.469831 24113 tablet_replica.cc:333] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.469841  1348 consensus_queue.cc:1048] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:12:00.471185 24113 raft_consensus.cc:2243] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:00.472074 24113 raft_consensus.cc:2272] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.474895 24113 tablet_replica.cc:333] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.475682 24113 raft_consensus.cc:2243] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.476068 24113 raft_consensus.cc:2272] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.478497 24113 tablet_replica.cc:333] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.479084 24113 raft_consensus.cc:2243] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 13:12:00.479329  1101 consensus_peers.cc:597] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:00.479493 24113 raft_consensus.cc:2272] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.481841 24113 tablet_replica.cc:333] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.482532 24113 raft_consensus.cc:2243] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.483006 24113 raft_consensus.cc:2272] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.486192 24113 tablet_replica.cc:333] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.486884 24113 raft_consensus.cc:2243] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.487387 24113 raft_consensus.cc:2272] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.490139 24113 tablet_replica.cc:333] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.490876 24113 raft_consensus.cc:2243] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.491369 24113 raft_consensus.cc:2272] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.493786 24113 tablet_replica.cc:333] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558: stopping tablet replica
I20260709 13:12:00.494436 24113 raft_consensus.cc:2243] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:00.494858 24113 raft_consensus.cc:2272] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:00.521229  1580 consensus_queue.cc:1048] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:12:00.528139  1101 consensus_peers.cc:597] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:00.532244  1175 consensus_peers.cc:597] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:00.551609  1027 consensus_peers.cc:597] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:00.559060  1027 consensus_peers.cc:597] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:00.577931 24113 tablet_server.cc:196] TabletServer@127.23.140.65:0 shutdown complete.
W20260709 13:12:00.578388  1175 consensus_peers.cc:597] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:00.606271  1101 consensus_peers.cc:597] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:00.620566  1101 consensus_peers.cc:597] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:00.641480  1027 consensus_peers.cc:597] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:00.642004  1027 consensus_peers.cc:597] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:00.646838  1027 consensus_peers.cc:597] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:00.648850 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:12:00.651712  1101 consensus_peers.cc:597] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:00.655645  1761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:00.656288  1762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:00.657713  1764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:12:00.658294 24113 server_base.cc:1061] running on GCE node
I20260709 13:12:00.666076 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:12:00.666422 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:12:00.666666 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602720666647 us; error 0 us; skew 500 ppm
I20260709 13:12:00.667541 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:12:00.678910 24113 webserver.cc:533] Webserver started at http://127.23.140.65:40743/ using document root <none> and password file <none>
I20260709 13:12:00.679772 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:12:00.680027 24113 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 13:12:00.685469 24113 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
I20260709 13:12:00.693495  1769 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:00.694454 24113 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 13:12:00.694824 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-0-root
uuid: "294145900dd3496cb4a1d722613e1558"
format_stamp: "Formatted at 2026-07-09 13:11:49 on dist-test-slave-k73n"
I20260709 13:12:00.695173 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602412999163-24113-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260709 13:12:00.715286  1101 consensus_peers.cc:597] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:00.717943 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:12:00.719342 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:12:00.720896 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:12:00.726049  1776 ts_tablet_manager.cc:548] Loading tablet metadata (0/45 complete)
W20260709 13:12:00.763056  1027 consensus_peers.cc:597] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:00.803858  1027 consensus_peers.cc:597] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:00.991869 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (45 total tablets, 45 live tablets)
I20260709 13:12:00.992332 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.267s	user 0.005s	sys 0.004s
I20260709 13:12:00.992695 24113 ts_tablet_manager.cc:600] Registering tablets (0/45 complete)
I20260709 13:12:01.000574  1776 tablet_bootstrap.cc:492] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:01.027089  1776 tablet_bootstrap.cc:492] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:01.028090  1776 tablet_bootstrap.cc:492] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:01.028743  1776 ts_tablet_manager.cc:1403] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.028s	user 0.015s	sys 0.004s
I20260709 13:12:01.032315  1776 raft_consensus.cc:359] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.032972  1776 raft_consensus.cc:740] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:01.033855  1776 consensus_queue.cc:260] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.039143  1776 ts_tablet_manager.cc:1434] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.004s
I20260709 13:12:01.039834  1776 tablet_bootstrap.cc:492] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:01.056352  1776 tablet_bootstrap.cc:492] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:01.057256  1776 tablet_bootstrap.cc:492] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:01.057793  1776 ts_tablet_manager.cc:1403] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.018s	user 0.009s	sys 0.005s
I20260709 13:12:01.059880  1776 raft_consensus.cc:359] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.060393  1776 raft_consensus.cc:740] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:01.060935  1776 consensus_queue.cc:260] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.062381  1776 ts_tablet_manager.cc:1434] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.002s
I20260709 13:12:01.063014  1776 tablet_bootstrap.cc:492] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:01.074947  1776 tablet_bootstrap.cc:492] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:01.075683  1776 tablet_bootstrap.cc:492] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:01.076200  1776 ts_tablet_manager.cc:1403] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.013s	user 0.010s	sys 0.000s
I20260709 13:12:01.078593  1776 raft_consensus.cc:359] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.079249  1776 raft_consensus.cc:740] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:01.079878  1776 consensus_queue.cc:260] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.081993  1776 ts_tablet_manager.cc:1434] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
I20260709 13:12:01.082755  1776 tablet_bootstrap.cc:492] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:01.097478  1776 tablet_bootstrap.cc:492] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:01.098197  1776 tablet_bootstrap.cc:492] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:01.098701  1776 ts_tablet_manager.cc:1403] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.016s	user 0.011s	sys 0.004s
I20260709 13:12:01.100625  1776 raft_consensus.cc:359] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.101329  1776 raft_consensus.cc:740] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:01.101964  1776 consensus_queue.cc:260] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.104068  1776 ts_tablet_manager.cc:1434] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20260709 13:12:01.104808  1776 tablet_bootstrap.cc:492] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:01.117030  1776 tablet_bootstrap.cc:492] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:01.117833  1776 tablet_bootstrap.cc:492] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:01.118345  1776 ts_tablet_manager.cc:1403] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.014s	user 0.012s	sys 0.000s
I20260709 13:12:01.120127  1776 raft_consensus.cc:359] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.120616  1776 raft_consensus.cc:740] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:01.121225  1776 consensus_queue.cc:260] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.122540  1776 ts_tablet_manager.cc:1434] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20260709 13:12:01.123212  1776 tablet_bootstrap.cc:492] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:01.150233  1776 tablet_bootstrap.cc:492] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:01.151122  1776 tablet_bootstrap.cc:492] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:01.151748  1776 ts_tablet_manager.cc:1403] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.029s	user 0.012s	sys 0.012s
I20260709 13:12:01.153674  1776 raft_consensus.cc:359] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.154173  1776 raft_consensus.cc:740] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:01.154752  1776 consensus_queue.cc:260] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.156129  1776 ts_tablet_manager.cc:1434] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20260709 13:12:01.156733  1776 tablet_bootstrap.cc:492] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:01.169453  1776 tablet_bootstrap.cc:492] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:01.170485  1776 tablet_bootstrap.cc:492] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:01.171124  1776 ts_tablet_manager.cc:1403] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.015s	user 0.011s	sys 0.000s
I20260709 13:12:01.173611  1776 raft_consensus.cc:359] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.174307  1776 raft_consensus.cc:740] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:01.175064  1776 consensus_queue.cc:260] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.176971  1776 ts_tablet_manager.cc:1434] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:12:01.177732  1776 tablet_bootstrap.cc:492] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:01.199421  1776 tablet_bootstrap.cc:492] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:01.201444  1776 tablet_bootstrap.cc:492] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:01.202080  1776 ts_tablet_manager.cc:1403] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.025s	user 0.012s	sys 0.003s
I20260709 13:12:01.204651  1776 raft_consensus.cc:359] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.205386  1776 raft_consensus.cc:740] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:01.206046  1776 consensus_queue.cc:260] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.208480  1776 ts_tablet_manager.cc:1434] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:12:01.209254  1776 tablet_bootstrap.cc:492] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:01.222111 24113 ts_tablet_manager.cc:616] Registered 45 tablets
I20260709 13:12:01.222512 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.230s	user 0.219s	sys 0.008s
I20260709 13:12:01.261667  1776 tablet_bootstrap.cc:492] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:01.262859  1776 tablet_bootstrap.cc:492] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:01.263586  1776 ts_tablet_manager.cc:1403] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.055s	user 0.018s	sys 0.020s
I20260709 13:12:01.266928  1776 raft_consensus.cc:359] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.267875  1776 raft_consensus.cc:740] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:01.268975  1776 consensus_queue.cc:260] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.271780  1776 ts_tablet_manager.cc:1434] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 13:12:01.272869  1776 tablet_bootstrap.cc:492] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:01.289935  1776 tablet_bootstrap.cc:492] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:01.290683  1776 tablet_bootstrap.cc:492] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:01.291227  1776 ts_tablet_manager.cc:1403] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.019s	user 0.016s	sys 0.000s
I20260709 13:12:01.293157  1776 raft_consensus.cc:359] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.293689  1776 raft_consensus.cc:740] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:01.294306  1776 consensus_queue.cc:260] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.298074  1776 ts_tablet_manager.cc:1434] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:12:01.299072  1776 tablet_bootstrap.cc:492] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:01.319427  1776 tablet_bootstrap.cc:492] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:01.320691  1776 tablet_bootstrap.cc:492] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:01.321475  1776 ts_tablet_manager.cc:1403] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.023s	user 0.019s	sys 0.000s
I20260709 13:12:01.324540  1776 raft_consensus.cc:359] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.325368  1776 raft_consensus.cc:740] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:01.326421  1776 consensus_queue.cc:260] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.329682  1776 ts_tablet_manager.cc:1434] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:12:01.330737  1776 tablet_bootstrap.cc:492] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:01.339768 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.65:43721
I20260709 13:12:01.339874  1844 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.65:43721 every 8 connection(s)
I20260709 13:12:01.355931  1776 tablet_bootstrap.cc:492] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:01.357158  1776 tablet_bootstrap.cc:492] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:01.357806  1776 ts_tablet_manager.cc:1403] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.027s	user 0.020s	sys 0.000s
I20260709 13:12:01.360473  1776 raft_consensus.cc:359] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.361181  1776 raft_consensus.cc:740] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:01.361979  1776 consensus_queue.cc:260] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.363760  1776 ts_tablet_manager.cc:1434] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 13:12:01.364578  1776 tablet_bootstrap.cc:492] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:01.365103  1845 heartbeater.cc:344] Connected to a master server at 127.23.140.126:33289
I20260709 13:12:01.365657  1845 heartbeater.cc:461] Registering TS with master...
I20260709 13:12:01.366612  1845 heartbeater.cc:507] Master 127.23.140.126:33289 requested a full tablet report, sending...
I20260709 13:12:01.384321   883 ts_manager.cc:194] Re-registered known tserver with Master: 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:01.396623  1776 tablet_bootstrap.cc:492] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:01.398255  1776 tablet_bootstrap.cc:492] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:01.399256  1776 ts_tablet_manager.cc:1403] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.035s	user 0.029s	sys 0.003s
I20260709 13:12:01.401813  1776 raft_consensus.cc:359] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.402570  1776 raft_consensus.cc:740] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:01.403225  1683 raft_consensus.cc:493] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.403795  1683 raft_consensus.cc:515] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.406445  1683 leader_election.cc:290] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:01.407943  1776 consensus_queue.cc:260] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.408144  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:01.410224  1776 ts_tablet_manager.cc:1434] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.000s
I20260709 13:12:01.411306  1776 tablet_bootstrap.cc:492] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:01.442976   883 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53818
I20260709 13:12:01.448025  1776 tablet_bootstrap.cc:492] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:01.448807  1845 heartbeater.cc:499] Master 127.23.140.126:33289 was elected leader, sending a full tablet report...
I20260709 13:12:01.449215  1776 tablet_bootstrap.cc:492] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:01.450030  1776 ts_tablet_manager.cc:1403] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.039s	user 0.012s	sys 0.024s
I20260709 13:12:01.451387  1819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:12:01.452958  1175 leader_election.cc:343] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:01.452852  1776 raft_consensus.cc:359] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.453439  1175 leader_election.cc:304] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:01.453603  1776 raft_consensus.cc:740] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:01.454288  1776 consensus_queue.cc:260] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.454761  1683 raft_consensus.cc:2749] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:01.456341  1776 ts_tablet_manager.cc:1434] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20260709 13:12:01.457484  1776 tablet_bootstrap.cc:492] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:01.466269  1580 raft_consensus.cc:493] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.466825  1580 raft_consensus.cc:515] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.470965  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:12:01.472543  1027 leader_election.cc:343] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:01.468916  1580 leader_election.cc:290] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:01.476241  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:01.480780  1025 leader_election.cc:304] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:01.482612  1580 raft_consensus.cc:2749] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:01.484520  1776 tablet_bootstrap.cc:492] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:01.485540  1776 tablet_bootstrap.cc:492] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:01.486388  1776 ts_tablet_manager.cc:1403] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.029s	user 0.007s	sys 0.015s
I20260709 13:12:01.489953  1776 raft_consensus.cc:359] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.490756  1776 raft_consensus.cc:740] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:01.491508  1776 consensus_queue.cc:260] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.493472  1776 ts_tablet_manager.cc:1434] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.003s
I20260709 13:12:01.494233  1776 tablet_bootstrap.cc:492] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:01.504662  1853 raft_consensus.cc:493] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.505272  1853 raft_consensus.cc:515] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.508412  1206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:01.511114  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:01.510102  1853 leader_election.cc:290] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
W20260709 13:12:01.513034  1101 leader_election.cc:343] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:01.513504  1101 leader_election.cc:304] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:01.515620  1853 raft_consensus.cc:2749] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:01.521078  1776 tablet_bootstrap.cc:492] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:01.522030  1776 tablet_bootstrap.cc:492] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:01.522711  1776 ts_tablet_manager.cc:1403] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.029s	user 0.011s	sys 0.004s
I20260709 13:12:01.524207  1580 raft_consensus.cc:493] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.525239  1776 raft_consensus.cc:359] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.525934  1776 raft_consensus.cc:740] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:01.525058  1580 raft_consensus.cc:515] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.526700  1776 consensus_queue.cc:260] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.529553  1776 ts_tablet_manager.cc:1434] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:12:01.529388  1206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:01.530444  1776 tablet_bootstrap.cc:492] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:01.535548  1853 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.536134  1853 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.550216  1819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:01.550648  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:01.551011  1819 raft_consensus.cc:2468] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:12:01.552397  1853 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:01.552592  1027 leader_election.cc:304] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 862b41697c74421fbac82c10c245434d
I20260709 13:12:01.551339  1817 raft_consensus.cc:2468] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:12:01.553423  1679 raft_consensus.cc:493] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.554087  1101 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:01.553918  1679 raft_consensus.cc:515] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.556140  1317 raft_consensus.cc:2804] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:01.556622  1317 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.556952  1317 raft_consensus.cc:3060] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.558286  1584 raft_consensus.cc:493] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.558799  1584 raft_consensus.cc:515] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.562167  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:01.562455  1717 raft_consensus.cc:493] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.562848  1819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:01.562994  1717 raft_consensus.cc:515] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.563541  1501 raft_consensus.cc:2804] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:01.565590  1501 raft_consensus.cc:493] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.566128  1501 raft_consensus.cc:3060] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.570606  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:01.563522  1819 raft_consensus.cc:2468] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:12:01.571597  1817 raft_consensus.cc:2468] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:12:01.571801  1584 leader_election.cc:290] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:01.556180  1679 leader_election.cc:290] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:01.572585  1175 leader_election.cc:304] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 
I20260709 13:12:01.572599  1679 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.573376  1348 raft_consensus.cc:493] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.573855  1679 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.574193  1348 raft_consensus.cc:515] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.576105  1679 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:01.577090  1348 leader_election.cc:290] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:01.578205  1348 raft_consensus.cc:2804] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:01.578639  1348 raft_consensus.cc:493] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.578677  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b89f56a91e004f408b36c36d7aee0076" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:01.580299  1819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b89f56a91e004f408b36c36d7aee0076" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:01.580662  1717 leader_election.cc:290] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:01.580991  1819 raft_consensus.cc:2468] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:12:01.583150  1819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:12:01.584404  1101 leader_election.cc:343] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:01.579170  1348 raft_consensus.cc:3060] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.585417  1175 leader_election.cc:304] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:01.586807  1584 raft_consensus.cc:2804] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:01.587189  1584 raft_consensus.cc:493] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.587570  1584 raft_consensus.cc:3060] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.589375  1580 leader_election.cc:290] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:01.595723  1206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:01.595580  1348 raft_consensus.cc:515] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.597913  1099 leader_election.cc:304] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:01.598685  1717 raft_consensus.cc:2749] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:01.596195  1317 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.599547  1348 leader_election.cc:290] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:01.599666  1819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:01.604313  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:01.604780  1819 raft_consensus.cc:3060] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.604965  1817 raft_consensus.cc:2468] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:12:01.601450  1717 raft_consensus.cc:493] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.608095  1348 raft_consensus.cc:493] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.608598  1348 raft_consensus.cc:515] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.609506  1027 leader_election.cc:304] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 862b41697c74421fbac82c10c245434d
I20260709 13:12:01.609586  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:01.610616  1135 raft_consensus.cc:2393] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 2: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:01.610677  1027 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:01.611234  1348 leader_election.cc:290] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:01.611759  1861 raft_consensus.cc:2804] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:01.611984  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:01.612219  1861 raft_consensus.cc:493] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.612617  1861 raft_consensus.cc:3060] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.612679  1132 raft_consensus.cc:2468] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:12:01.613984  1173 leader_election.cc:304] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:01.614143  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:01.614759  1817 raft_consensus.cc:3060] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.615969  1348 raft_consensus.cc:2804] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:01.616384  1348 raft_consensus.cc:493] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.616719  1348 raft_consensus.cc:3060] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.616854  1580 raft_consensus.cc:2804] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:01.618414  1317 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:01.618417  1580 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.619366  1580 raft_consensus.cc:3060] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.621879  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:01.623407  1584 raft_consensus.cc:515] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.624951  1317 raft_consensus.cc:493] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.626042  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b89f56a91e004f408b36c36d7aee0076" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:01.626196  1317 raft_consensus.cc:515] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
W20260709 13:12:01.627468  1175 leader_election.cc:343] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:01.629182  1348 raft_consensus.cc:515] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.630280  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:01.631131  1348 leader_election.cc:290] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
W20260709 13:12:01.632228  1101 leader_election.cc:343] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:01.632836  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:01.633105  1861 raft_consensus.cc:515] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.633512  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:01.606220  1717 raft_consensus.cc:515] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.640632  1132 raft_consensus.cc:3060] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.633725  1317 leader_election.cc:290] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:01.602756  1501 raft_consensus.cc:515] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.602769  1776 tablet_bootstrap.cc:492] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:01.642884  1776 tablet_bootstrap.cc:492] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:01.643567  1776 ts_tablet_manager.cc:1403] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.113s	user 0.032s	sys 0.023s
I20260709 13:12:01.644119  1817 raft_consensus.cc:2468] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:01.645190  1101 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:01.645572  1501 leader_election.cc:290] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:01.646224  1853 raft_consensus.cc:2804] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:01.646064  1776 raft_consensus.cc:359] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.646773  1853 raft_consensus.cc:697] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 2 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:01.646898  1861 leader_election.cc:290] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:01.646786  1776 raft_consensus.cc:740] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
W20260709 13:12:01.635761  1175 leader_election.cc:343] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:01.648619  1776 consensus_queue.cc:260] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.649104  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:01.648658  1720 raft_consensus.cc:493] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.652948  1720 raft_consensus.cc:515] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.651450  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:01.655036  1720 leader_election.cc:290] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:01.655633  1207 raft_consensus.cc:2393] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 2: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
I20260709 13:12:01.656889  1853 consensus_queue.cc:237] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.658620  1776 ts_tablet_manager.cc:1434] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.001s
I20260709 13:12:01.659400  1206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:01.660604  1776 tablet_bootstrap.cc:492] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:01.661506  1348 raft_consensus.cc:493] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.664304  1348 raft_consensus.cc:515] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.655615  1819 raft_consensus.cc:2468] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:01.666234  1348 leader_election.cc:290] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:01.663237  1025 leader_election.cc:304] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:01.646605  1717 leader_election.cc:290] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:01.678051  1819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b89f56a91e004f408b36c36d7aee0076" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:01.670135  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:01.672472  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:01.673048  1861 raft_consensus.cc:2749] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:01.673337  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:01.681102  1818 raft_consensus.cc:3060] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.681420  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b89f56a91e004f408b36c36d7aee0076" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:01.674614  1175 leader_election.cc:304] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 
I20260709 13:12:01.667027  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b89f56a91e004f408b36c36d7aee0076" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:01.673661  1584 leader_election.cc:290] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:01.677963  1132 raft_consensus.cc:2468] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:01.679981  1819 raft_consensus.cc:3060] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.685148  1202 raft_consensus.cc:2393] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 2: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
I20260709 13:12:01.685878  1580 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.687968  1580 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:01.690251  1861 raft_consensus.cc:493] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.690785  1861 raft_consensus.cc:515] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.691233  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:01.692108  1056 raft_consensus.cc:2393] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 2: Already voted for candidate 9c45b6d917734835a8a37b0f4c789ae6 in this term.
I20260709 13:12:01.696228  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:01.683074  1348 raft_consensus.cc:2804] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:01.700953  1348 raft_consensus.cc:697] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
W20260709 13:12:01.700999  1175 leader_election.cc:343] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:12:01.697047  1101 leader_election.cc:343] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:01.678388  1501 raft_consensus.cc:493] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.703248  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:01.698096   883 catalog_manager.cc:5697] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 1 to 2, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 2 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:01.701821  1348 consensus_queue.cc:237] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.704316  1173 leader_election.cc:304] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:01.704380  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
W20260709 13:12:01.704576  1027 leader_election.cc:343] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:01.705052  1131 raft_consensus.cc:2468] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:12:01.705502  1683 raft_consensus.cc:2804] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:01.703617  1501 raft_consensus.cc:515] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.709822  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:01.710088  1025 leader_election.cc:304] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:01.710781  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:01.711851  1056 raft_consensus.cc:2393] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 2: Already voted for candidate 9c45b6d917734835a8a37b0f4c789ae6 in this term.
W20260709 13:12:01.711738  1027 leader_election.cc:343] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:01.692919  1861 leader_election.cc:290] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:01.712898  1679 raft_consensus.cc:2804] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:01.713848  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:01.714320  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:01.714248  1679 raft_consensus.cc:493] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.715281  1056 raft_consensus.cc:2393] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 2: Already voted for candidate 9c45b6d917734835a8a37b0f4c789ae6 in this term.
I20260709 13:12:01.715438  1679 raft_consensus.cc:3060] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.716157  1584 raft_consensus.cc:493] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.716665  1584 raft_consensus.cc:515] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.718791  1584 leader_election.cc:290] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:01.720278  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:01.714210  1501 leader_election.cc:290] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:01.722209  1101 leader_election.cc:304] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:01.731097  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:01.734634  1501 raft_consensus.cc:493] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.735848  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:01.735245  1501 raft_consensus.cc:515] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.736521  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:01.737265  1816 raft_consensus.cc:2468] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:12:01.738647  1175 leader_election.cc:304] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 
I20260709 13:12:01.738829  1853 raft_consensus.cc:3060] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.739234  1819 raft_consensus.cc:2468] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:01.739768  1057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:01.741245  1584 raft_consensus.cc:2804] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:01.741654  1584 raft_consensus.cc:493] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.742038  1584 raft_consensus.cc:3060] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.736697  1059 raft_consensus.cc:2393] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 2: Already voted for candidate 9c45b6d917734835a8a37b0f4c789ae6 in this term.
I20260709 13:12:01.745765  1025 leader_election.cc:304] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:01.748629  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:01.747149  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:01.749332  1062 raft_consensus.cc:2468] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:12:01.749837  1817 raft_consensus.cc:2393] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 2: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:01.750216  1818 raft_consensus.cc:2468] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
I20260709 13:12:01.750866  1027 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:01.752054  1580 raft_consensus.cc:2749] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:01.757714  1027 leader_election.cc:304] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 862b41697c74421fbac82c10c245434d
I20260709 13:12:01.746608  1859 raft_consensus.cc:2749] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:01.760228  1175 leader_election.cc:304] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:01.731248  1683 raft_consensus.cc:697] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:01.763978  1099 leader_election.cc:304] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:01.768332  1317 raft_consensus.cc:2804] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:01.768847  1317 raft_consensus.cc:493] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.769311  1317 raft_consensus.cc:3060] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.748750  1679 raft_consensus.cc:515] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.783028  1317 raft_consensus.cc:515] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.789716  1173 leader_election.cc:304] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:01.763149  1861 raft_consensus.cc:2804] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:01.790678  1501 leader_election.cc:290] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:01.790889  1861 raft_consensus.cc:697] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:01.791886  1861 consensus_queue.cc:237] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.789338  1859 raft_consensus.cc:493] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.793367  1864 raft_consensus.cc:493] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.793542  1859 raft_consensus.cc:515] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.793931  1864 raft_consensus.cc:515] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.794570  1679 leader_election.cc:290] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:01.797817  1348 raft_consensus.cc:2804] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:01.798476  1348 raft_consensus.cc:697] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:01.801298  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:01.816495  1317 leader_election.cc:290] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:01.800886  1859 leader_election.cc:290] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:01.801450  1683 consensus_queue.cc:237] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.819819  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf962b21fa14634af6e810ca29f1bc7" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
W20260709 13:12:01.820996  1027 leader_election.cc:343] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:12:01.825269  1027 leader_election.cc:343] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:01.799482  1348 consensus_queue.cc:237] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.810099  1868 raft_consensus.cc:493] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.826136  1057 raft_consensus.cc:2393] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 2: Already voted for candidate 9c45b6d917734835a8a37b0f4c789ae6 in this term.
I20260709 13:12:01.841109  1317 raft_consensus.cc:493] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.846343  1317 raft_consensus.cc:515] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.842063  1859 raft_consensus.cc:493] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.850327  1871 raft_consensus.cc:2749] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:01.850121  1859 raft_consensus.cc:515] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.852411  1859 leader_election.cc:290] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:01.826699  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:01.821061  1584 raft_consensus.cc:515] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.857264  1317 leader_election.cc:290] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:01.845211  1868 raft_consensus.cc:515] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.868743  1868 leader_election.cc:290] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
W20260709 13:12:01.870410  1175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 110243us
I20260709 13:12:01.853021  1853 raft_consensus.cc:2749] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:01.855165  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf962b21fa14634af6e810ca29f1bc7" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:01.874641  1874 raft_consensus.cc:493] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.874747  1132 raft_consensus.cc:2468] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:12:01.875429  1874 raft_consensus.cc:515] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:01.880183  1776 tablet_bootstrap.cc:492] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:01.884455  1776 tablet_bootstrap.cc:492] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
W20260709 13:12:01.886281  1027 leader_election.cc:343] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:01.889199  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:01.889794  1776 ts_tablet_manager.cc:1403] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.229s	user 0.016s	sys 0.008s
I20260709 13:12:01.883543  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:01.891815  1876 raft_consensus.cc:493] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.889837  1207 raft_consensus.cc:2468] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:12:01.897357  1776 raft_consensus.cc:359] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.897900  1025 leader_election.cc:304] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:01.898201  1776 raft_consensus.cc:385] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:01.896586  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:01.887307  1864 leader_election.cc:290] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:01.899114  1776 raft_consensus.cc:740] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:01.901762  1099 leader_election.cc:304] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:01.896710  1876 raft_consensus.cc:515] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:01.906040  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:01.908380  1025 leader_election.cc:304] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:01.910303  1501 raft_consensus.cc:2804] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:01.913748  1059 raft_consensus.cc:2468] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:12:01.916299  1131 raft_consensus.cc:3060] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.931411  1876 leader_election.cc:290] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:01.913744  1860 raft_consensus.cc:2804] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:12:01.916215  1027 leader_election.cc:343] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:01.907562  1206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:01.917770  1205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:01.918630  1874 leader_election.cc:290] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:01.913427  1776 consensus_queue.cc:260] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:01.940191  1205 raft_consensus.cc:2393] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 2: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
I20260709 13:12:01.934805  1860 raft_consensus.cc:493] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.960465  1860 raft_consensus.cc:3060] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.952984  1206 raft_consensus.cc:2468] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:12:01.926657  1501 raft_consensus.cc:493] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:01.962308  1501 raft_consensus.cc:3060] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.962980  1776 ts_tablet_manager.cc:1434] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.070s	user 0.007s	sys 0.000s
W20260709 13:12:01.965394  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 51753us
I20260709 13:12:01.971053  1025 leader_election.cc:304] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:01.983137  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:01.984071  1136 raft_consensus.cc:3060] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:01.997493  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:01.999895  1131 raft_consensus.cc:2468] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
I20260709 13:12:02.000192  1811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:01.999912  1057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf962b21fa14634af6e810ca29f1bc7" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:02.002204  1584 leader_election.cc:290] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:02.003690  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:02.005537  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:02.006182  1059 raft_consensus.cc:3060] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:02.009258  1853 raft_consensus.cc:2749] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:02.013017  1501 raft_consensus.cc:515] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
W20260709 13:12:02.014935  1175 leader_election.cc:343] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:02.018446  1129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:01.988034  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf962b21fa14634af6e810ca29f1bc7" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:02.019158  1129 raft_consensus.cc:2468] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:12:02.016211  1136 raft_consensus.cc:2468] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
W20260709 13:12:02.017423  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 46459.7us
I20260709 13:12:02.028872  1025 leader_election.cc:304] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:02.001397  1811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:02.014528  1867 raft_consensus.cc:2804] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:02.039600  1816 raft_consensus.cc:3060] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:02.040452  1059 raft_consensus.cc:2468] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:02.044308  1173 leader_election.cc:304] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:02.045668  1886 raft_consensus.cc:2804] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 13:12:02.060531  1175 leader_election.cc:343] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:02.055981  1860 raft_consensus.cc:515] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:02.067137  1173 leader_election.cc:304] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:02.068292  1871 raft_consensus.cc:2804] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:02.068795  1871 raft_consensus.cc:493] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:02.069325  1871 raft_consensus.cc:3060] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:02.073367  1584 raft_consensus.cc:2804] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:02.074018  1584 raft_consensus.cc:493] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:02.074509  1584 raft_consensus.cc:3060] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:02.078321  1099 leader_election.cc:304] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:02.079622  1869 raft_consensus.cc:2804] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:02.080106  1869 raft_consensus.cc:697] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 2 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:02.081173  1869 consensus_queue.cc:237] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:02.089249  1173 leader_election.cc:304] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:02.103205  1683 raft_consensus.cc:2804] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:02.103988  1683 raft_consensus.cc:697] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:02.105159  1683 consensus_queue.cc:237] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
W20260709 13:12:02.121706  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 86715.6us
I20260709 13:12:02.050777  1867 raft_consensus.cc:493] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:02.054651  1811 raft_consensus.cc:2468] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:12:02.066023  1814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:02.144178  1776 tablet_bootstrap.cc:492] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:02.147325  1061 raft_consensus.cc:1275] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:12:02.148878  1175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 94451.8us
I20260709 13:12:02.154948  1886 raft_consensus.cc:697] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:02.156142  1886 consensus_queue.cc:237] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:02.156589  1062 raft_consensus.cc:1275] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:02.127331  1867 raft_consensus.cc:3060] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:02.169495  1866 consensus_queue.cc:1048] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 13:12:02.159735  1057 raft_consensus.cc:2393] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 2: Already voted for candidate 9c45b6d917734835a8a37b0f4c789ae6 in this term.
I20260709 13:12:02.192727  1811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:02.193565  1811 raft_consensus.cc:3060] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:02.187268  1584 raft_consensus.cc:515] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:02.187690  1860 leader_election.cc:290] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:02.197054  1867 raft_consensus.cc:515] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:02.215112  1871 raft_consensus.cc:515] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:02.198088  1813 raft_consensus.cc:1275] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:02.189399   880 catalog_manager.cc:5697] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 1 to 2, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 2 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:02.213599  1584 leader_election.cc:290] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:02.215960  1867 leader_election.cc:290] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:02.218866  1871 leader_election.cc:290] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:02.191830  1501 leader_election.cc:290] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:02.174007  1814 raft_consensus.cc:2393] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 2: Already voted for candidate 9c45b6d917734835a8a37b0f4c789ae6 in this term.
W20260709 13:12:02.220415  1175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 69002.1us
W20260709 13:12:02.201020  1101 leader_election.cc:343] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:02.225739  1101 leader_election.cc:304] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d
I20260709 13:12:02.227746  1868 raft_consensus.cc:2764] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260709 13:12:02.230027  1868 consensus_queue.cc:1048] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:02.231026  1812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:02.232292  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:02.233023  1810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
W20260709 13:12:02.237922  1101 leader_election.cc:343] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:12:02.244733  1175 leader_election.cc:343] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:02.238763   880 catalog_manager.cc:5697] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 1 to 2, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 2 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
W20260709 13:12:02.248013  1175 leader_election.cc:343] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:02.249007  1175 leader_election.cc:304] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:02.250831  1348 raft_consensus.cc:3060] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:02.260634  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:02.261297  1061 raft_consensus.cc:3060] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:02.264806  1348 raft_consensus.cc:2749] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:02.250839  1814 raft_consensus.cc:1275] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:02.273165  1317 consensus_queue.cc:1048] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:02.282049  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:02.283233  1814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf962b21fa14634af6e810ca29f1bc7" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:02.285034  1810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:02.270874   879 catalog_manager.cc:5697] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 1 to 2, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:02.274955  1205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:02.275988  1206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:02.274394  1099 leader_election.cc:304] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:02.288545  1205 raft_consensus.cc:3060] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:02.312604  1811 raft_consensus.cc:2468] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:02.307726  1348 consensus_queue.cc:1048] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 13:12:02.323763  1061 raft_consensus.cc:2468] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:02.337683  1173 leader_election.cc:304] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:02.305711  1206 raft_consensus.cc:3060] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:12:02.344245  1027 leader_election.cc:343] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:02.352993  1896 raft_consensus.cc:3060] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:02.353780  1205 raft_consensus.cc:2468] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
I20260709 13:12:02.355880  1025 leader_election.cc:304] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
W20260709 13:12:02.368719  1027 leader_election.cc:343] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:02.375495  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf962b21fa14634af6e810ca29f1bc7" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:02.347146  1866 raft_consensus.cc:2804] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:02.373481  1580 raft_consensus.cc:2804] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:02.406776   880 catalog_manager.cc:5697] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 1 to 2, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 2 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:02.409940  1776 tablet_bootstrap.cc:492] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:02.435591  1776 tablet_bootstrap.cc:492] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:02.436592  1776 ts_tablet_manager.cc:1403] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.293s	user 0.016s	sys 0.015s
I20260709 13:12:02.415908  1866 raft_consensus.cc:697] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
W20260709 13:12:02.441011  1027 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 51008.4us
I20260709 13:12:02.442698  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:02.419538  1580 raft_consensus.cc:697] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:02.454833  1580 consensus_queue.cc:237] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:02.453953  1135 raft_consensus.cc:3060] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:12:02.457217  1027 leader_election.cc:343] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:02.458011  1866 consensus_queue.cc:237] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:02.465425  1776 raft_consensus.cc:359] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:02.466892  1776 raft_consensus.cc:740] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:02.467558  1776 consensus_queue.cc:260] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
W20260709 13:12:02.469009  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 73910.3us
I20260709 13:12:02.473922  1776 ts_tablet_manager.cc:1434] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.037s	user 0.006s	sys 0.001s
I20260709 13:12:02.474869  1896 raft_consensus.cc:2749] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:02.477658  1811 raft_consensus.cc:1275] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:02.481050   879 catalog_manager.cc:5697] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 1 to 2, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 2 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:02.483059   879 catalog_manager.cc:5697] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 1 to 2, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 2 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:02.490219  1136 raft_consensus.cc:2468] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
I20260709 13:12:02.476820  1135 raft_consensus.cc:2468] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:02.479465  1206 raft_consensus.cc:2468] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
I20260709 13:12:02.509938  1879 consensus_queue.cc:1048] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:02.510856  1202 raft_consensus.cc:1275] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:02.512287  1894 consensus_queue.cc:1048] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:02.520323  1776 tablet_bootstrap.cc:492] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
W20260709 13:12:02.528592  1175 consensus_peers.cc:597] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:02.533556  1025 leader_election.cc:304] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:02.535163  1132 raft_consensus.cc:1275] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:02.536677  1867 raft_consensus.cc:2804] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:02.537267  1867 raft_consensus.cc:697] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:02.538138  1867 consensus_queue.cc:237] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:02.540920  1819 raft_consensus.cc:1275] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:02.541648  1131 raft_consensus.cc:1275] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:02.545874  1173 leader_election.cc:304] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:02.547128  1348 raft_consensus.cc:2804] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:02.547657  1348 raft_consensus.cc:697] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:02.548413  1885 consensus_queue.cc:1048] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:02.548410  1348 consensus_queue.cc:237] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:02.567603  1584 consensus_queue.cc:1048] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260709 13:12:02.625466  1025 leader_election.cc:304] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:02.574151  1871 consensus_queue.cc:1048] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260709 13:12:02.636508  1501 raft_consensus.cc:2804] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:02.673357  1501 raft_consensus.cc:697] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:02.675223  1501 consensus_queue.cc:237] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:02.678150  1812 raft_consensus.cc:1275] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:12:02.699765  1027 consensus_peers.cc:597] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:02.713548  1132 raft_consensus.cc:1275] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:02.715464  1871 consensus_queue.cc:1048] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:02.757194  1133 raft_consensus.cc:1275] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:12:02.763468  1101 consensus_peers.cc:597] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:02.766901  1892 consensus_queue.cc:1048] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:12:02.769438  1027 consensus_peers.cc:597] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:02.779289  1879 consensus_queue.cc:1048] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:02.782951   879 catalog_manager.cc:5697] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 1 to 2, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 2 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:02.784909   879 catalog_manager.cc:5697] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 1 to 2, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 2 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
W20260709 13:12:02.781316  1918 log.cc:927] Time spent T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 13:12:02.793533  1776 tablet_bootstrap.cc:492] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:02.802834  1776 tablet_bootstrap.cc:492] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:02.803560  1776 ts_tablet_manager.cc:1403] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.283s	user 0.038s	sys 0.039s
I20260709 13:12:02.806396  1776 raft_consensus.cc:359] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:02.807062  1776 raft_consensus.cc:740] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:02.807866  1776 consensus_queue.cc:260] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:02.810267  1776 ts_tablet_manager.cc:1434] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:12:02.811163  1776 tablet_bootstrap.cc:492] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
W20260709 13:12:02.838435  1175 consensus_peers.cc:597] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:02.842149  1776 tablet_bootstrap.cc:492] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:02.843048  1776 tablet_bootstrap.cc:492] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:02.917500  1776 ts_tablet_manager.cc:1403] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.106s	user 0.021s	sys 0.012s
I20260709 13:12:02.923476  1776 raft_consensus.cc:359] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:02.932973  1776 raft_consensus.cc:740] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:02.945607   883 catalog_manager.cc:5697] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 1 to 2, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:02.949770   883 catalog_manager.cc:5697] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 1 to 2, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } }
W20260709 13:12:02.967236  1027 consensus_peers.cc:597] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:02.978343  1027 consensus_peers.cc:597] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:02.995066  1060 raft_consensus.cc:1275] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:12:03.015386  1027 consensus_peers.cc:597] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:03.019789  1027 consensus_peers.cc:597] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:03.027683  1202 raft_consensus.cc:1275] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:03.030409  1210 raft_consensus.cc:1275] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:03.035815  1580 consensus_queue.cc:1048] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:03.033702   879 catalog_manager.cc:5697] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 1 to 2, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:03.056921   879 catalog_manager.cc:5697] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 1 to 2, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:03.058290  1136 raft_consensus.cc:1275] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:02.933974  1776 consensus_queue.cc:260] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
W20260709 13:12:03.050374  1027 consensus_peers.cc:597] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:03.051537  1060 raft_consensus.cc:1275] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:03.055069  1883 consensus_queue.cc:1048] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:03.073359  1584 consensus_queue.cc:1048] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:03.086532  1776 ts_tablet_manager.cc:1434] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.168s	user 0.011s	sys 0.000s
I20260709 13:12:03.090886  1776 tablet_bootstrap.cc:492] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
W20260709 13:12:03.107956  1027 consensus_peers.cc:597] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:03.119406  1175 consensus_peers.cc:597] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:03.126258  1879 consensus_queue.cc:1048] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 13:12:03.143479  1853 consensus_queue.cc:1048] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
W20260709 13:12:03.169456  1173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 47133.8us
W20260709 13:12:03.178766  1175 consensus_peers.cc:597] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:03.178936  1101 consensus_peers.cc:597] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:03.179328  1175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 53354.4us
W20260709 13:12:03.190635  1175 consensus_peers.cc:597] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:03.225898  1136 raft_consensus.cc:1275] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:03.228860  1882 consensus_queue.cc:1048] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:12:03.244838  1027 consensus_peers.cc:597] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:03.267443  1027 consensus_peers.cc:597] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:03.269343  1776 tablet_bootstrap.cc:492] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.270464  1776 tablet_bootstrap.cc:492] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.271140  1776 ts_tablet_manager.cc:1403] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.181s	user 0.025s	sys 0.013s
I20260709 13:12:03.274333  1776 raft_consensus.cc:359] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.275080  1776 raft_consensus.cc:740] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.275938  1776 consensus_queue.cc:260] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
W20260709 13:12:03.285356  1101 consensus_peers.cc:597] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:03.286136  1101 consensus_peers.cc:597] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:03.289885  1776 ts_tablet_manager.cc:1434] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.018s	user 0.004s	sys 0.004s
I20260709 13:12:03.291188  1776 tablet_bootstrap.cc:492] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
W20260709 13:12:03.312942  1101 consensus_peers.cc:597] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:03.326174  1776 tablet_bootstrap.cc:492] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.327162  1776 tablet_bootstrap.cc:492] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.327775  1776 ts_tablet_manager.cc:1403] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.037s	user 0.008s	sys 0.007s
I20260709 13:12:03.330943  1776 raft_consensus.cc:359] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:03.331621  1776 raft_consensus.cc:740] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.332276  1776 consensus_queue.cc:260] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:03.334249  1776 ts_tablet_manager.cc:1434] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 13:12:03.349298  1776 tablet_bootstrap.cc:492] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.367319  1776 tablet_bootstrap.cc:492] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.367980  1776 tablet_bootstrap.cc:492] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.368454  1776 ts_tablet_manager.cc:1403] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.022s	user 0.012s	sys 0.000s
I20260709 13:12:03.370410  1776 raft_consensus.cc:359] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.370846  1776 raft_consensus.cc:740] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.371304  1776 consensus_queue.cc:260] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.373179  1776 ts_tablet_manager.cc:1434] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
I20260709 13:12:03.374363  1776 tablet_bootstrap.cc:492] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.390695  1776 tablet_bootstrap.cc:492] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.391528  1776 tablet_bootstrap.cc:492] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.392134  1776 ts_tablet_manager.cc:1403] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.018s	user 0.013s	sys 0.000s
W20260709 13:12:03.393187  1027 consensus_peers.cc:597] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:03.394661  1776 raft_consensus.cc:359] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:03.395251  1776 raft_consensus.cc:740] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.395854  1776 consensus_queue.cc:260] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:03.397387  1776 ts_tablet_manager.cc:1434] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20260709 13:12:03.398281  1776 tablet_bootstrap.cc:492] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.416471  1776 tablet_bootstrap.cc:492] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.417469  1776 tablet_bootstrap.cc:492] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.418190  1776 ts_tablet_manager.cc:1403] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.020s	user 0.018s	sys 0.000s
I20260709 13:12:03.420852  1776 raft_consensus.cc:359] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:03.421396  1776 raft_consensus.cc:740] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.421990  1776 consensus_queue.cc:260] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:03.423527  1776 ts_tablet_manager.cc:1434] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:12:03.424743  1776 tablet_bootstrap.cc:492] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.441812  1776 tablet_bootstrap.cc:492] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.442708  1776 tablet_bootstrap.cc:492] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.443369  1776 ts_tablet_manager.cc:1403] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.019s	user 0.016s	sys 0.000s
I20260709 13:12:03.445734  1776 raft_consensus.cc:359] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:03.446206  1776 raft_consensus.cc:740] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.446758  1776 consensus_queue.cc:260] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:03.448566  1776 ts_tablet_manager.cc:1434] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:12:03.449498  1776 tablet_bootstrap.cc:492] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
W20260709 13:12:03.457357  1027 consensus_peers.cc:597] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:03.464151  1776 tablet_bootstrap.cc:492] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.464953  1776 tablet_bootstrap.cc:492] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.465607  1776 ts_tablet_manager.cc:1403] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.016s	user 0.015s	sys 0.000s
I20260709 13:12:03.467967  1776 raft_consensus.cc:359] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.468587  1776 raft_consensus.cc:740] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.469247  1776 consensus_queue.cc:260] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.470809  1776 ts_tablet_manager.cc:1434] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:12:03.471890  1776 tablet_bootstrap.cc:492] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.485863  1776 tablet_bootstrap.cc:492] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.486629  1776 tablet_bootstrap.cc:492] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.487174  1776 ts_tablet_manager.cc:1403] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.016s	user 0.005s	sys 0.008s
I20260709 13:12:03.488878  1776 raft_consensus.cc:359] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.489454  1776 raft_consensus.cc:740] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.490056  1776 consensus_queue.cc:260] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.491641  1776 ts_tablet_manager.cc:1434] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20260709 13:12:03.492476  1776 tablet_bootstrap.cc:492] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.507055  1776 tablet_bootstrap.cc:492] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.508139  1776 tablet_bootstrap.cc:492] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.508913  1776 ts_tablet_manager.cc:1403] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.017s	user 0.012s	sys 0.000s
I20260709 13:12:03.511900  1776 raft_consensus.cc:359] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.512562  1776 raft_consensus.cc:740] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.513352  1776 consensus_queue.cc:260] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.515785  1776 ts_tablet_manager.cc:1434] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.003s
I20260709 13:12:03.517618  1776 tablet_bootstrap.cc:492] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.535856  1776 tablet_bootstrap.cc:492] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.536597  1776 tablet_bootstrap.cc:492] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.537250  1776 ts_tablet_manager.cc:1403] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.020s	user 0.015s	sys 0.004s
I20260709 13:12:03.539192  1776 raft_consensus.cc:359] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.539697  1776 raft_consensus.cc:740] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.540297  1776 consensus_queue.cc:260] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.542245  1776 ts_tablet_manager.cc:1434] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:12:03.543617  1776 tablet_bootstrap.cc:492] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.575255  1776 tablet_bootstrap.cc:492] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.576151  1776 tablet_bootstrap.cc:492] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.576828  1776 ts_tablet_manager.cc:1403] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.034s	user 0.013s	sys 0.012s
I20260709 13:12:03.579329  1776 raft_consensus.cc:359] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:03.579995  1776 raft_consensus.cc:740] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.580765  1776 consensus_queue.cc:260] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:03.582906  1776 ts_tablet_manager.cc:1434] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 13:12:03.583771  1776 tablet_bootstrap.cc:492] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.599032  1776 tablet_bootstrap.cc:492] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.599983  1776 tablet_bootstrap.cc:492] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.600678  1776 ts_tablet_manager.cc:1403] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.017s	user 0.013s	sys 0.000s
I20260709 13:12:03.602963  1776 raft_consensus.cc:359] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:03.603654  1776 raft_consensus.cc:740] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.604341  1776 consensus_queue.cc:260] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:03.606423  1776 ts_tablet_manager.cc:1434] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.003s
I20260709 13:12:03.607277  1776 tablet_bootstrap.cc:492] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.623090  1776 tablet_bootstrap.cc:492] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.623842  1776 tablet_bootstrap.cc:492] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.624390  1776 ts_tablet_manager.cc:1403] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.017s	user 0.013s	sys 0.000s
I20260709 13:12:03.626757  1776 raft_consensus.cc:359] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.627625  1776 raft_consensus.cc:740] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.628433  1776 consensus_queue.cc:260] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.630716  1776 ts_tablet_manager.cc:1434] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 13:12:03.631851  1776 tablet_bootstrap.cc:492] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.646507  1776 tablet_bootstrap.cc:492] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.647150  1776 tablet_bootstrap.cc:492] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.647753  1776 ts_tablet_manager.cc:1403] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.016s	user 0.008s	sys 0.005s
I20260709 13:12:03.649672  1776 raft_consensus.cc:359] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:03.650120  1776 raft_consensus.cc:740] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.650578  1776 consensus_queue.cc:260] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:03.652138  1776 ts_tablet_manager.cc:1434] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.004s
I20260709 13:12:03.652958  1776 tablet_bootstrap.cc:492] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.672755  1776 tablet_bootstrap.cc:492] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.673712  1776 tablet_bootstrap.cc:492] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.674360  1776 ts_tablet_manager.cc:1403] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.022s	user 0.015s	sys 0.000s
I20260709 13:12:03.676854  1776 raft_consensus.cc:359] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:03.677569  1776 raft_consensus.cc:740] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.678229  1776 consensus_queue.cc:260] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:03.679793  1776 ts_tablet_manager.cc:1434] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:12:03.680598  1776 tablet_bootstrap.cc:492] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.681188  1815 raft_consensus.cc:3060] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:03.700842  1776 tablet_bootstrap.cc:492] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.701965  1776 tablet_bootstrap.cc:492] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.702658  1776 ts_tablet_manager.cc:1403] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.022s	user 0.017s	sys 0.000s
I20260709 13:12:03.705257  1776 raft_consensus.cc:359] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:03.705981  1776 raft_consensus.cc:740] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.706687  1776 consensus_queue.cc:260] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:03.708841  1776 ts_tablet_manager.cc:1434] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20260709 13:12:03.710029  1776 tablet_bootstrap.cc:492] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.721729  1776 tablet_bootstrap.cc:492] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.722445  1776 tablet_bootstrap.cc:492] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.722950  1776 ts_tablet_manager.cc:1403] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.013s	user 0.011s	sys 0.000s
I20260709 13:12:03.724867  1776 raft_consensus.cc:359] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:03.725430  1776 raft_consensus.cc:740] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.725983  1776 consensus_queue.cc:260] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:03.727996  1776 ts_tablet_manager.cc:1434] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20260709 13:12:03.729314  1776 tablet_bootstrap.cc:492] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.737236  1815 raft_consensus.cc:3060] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:03.746225  1776 tablet_bootstrap.cc:492] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.747205  1776 tablet_bootstrap.cc:492] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.747876  1776 ts_tablet_manager.cc:1403] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.019s	user 0.014s	sys 0.000s
I20260709 13:12:03.750447  1776 raft_consensus.cc:359] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.751206  1776 raft_consensus.cc:740] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.751996  1776 consensus_queue.cc:260] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.754145  1776 ts_tablet_manager.cc:1434] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.003s
I20260709 13:12:03.755007  1776 tablet_bootstrap.cc:492] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.773005  1776 tablet_bootstrap.cc:492] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.773854  1776 tablet_bootstrap.cc:492] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.774864  1776 ts_tablet_manager.cc:1403] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.020s	user 0.014s	sys 0.005s
I20260709 13:12:03.778995  1776 raft_consensus.cc:359] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:03.779702  1776 raft_consensus.cc:740] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.780364  1776 consensus_queue.cc:260] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:03.784345  1776 ts_tablet_manager.cc:1434] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.003s
I20260709 13:12:03.789531  1776 tablet_bootstrap.cc:492] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.798105  1815 raft_consensus.cc:3060] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:03.807035  1776 tablet_bootstrap.cc:492] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.808211  1776 tablet_bootstrap.cc:492] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.808915  1776 ts_tablet_manager.cc:1403] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.024s	user 0.008s	sys 0.011s
I20260709 13:12:03.812304  1776 raft_consensus.cc:359] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:03.812831  1776 raft_consensus.cc:740] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.813464  1776 consensus_queue.cc:260] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:03.817413  1776 ts_tablet_manager.cc:1434] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 13:12:03.818298  1776 tablet_bootstrap.cc:492] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.837602  1776 tablet_bootstrap.cc:492] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.838335  1776 tablet_bootstrap.cc:492] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.838997  1776 ts_tablet_manager.cc:1403] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.021s	user 0.007s	sys 0.007s
I20260709 13:12:03.840652  1815 raft_consensus.cc:3060] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:03.841300  1776 raft_consensus.cc:359] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.842036  1776 raft_consensus.cc:740] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.842698  1776 consensus_queue.cc:260] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.844702  1776 ts_tablet_manager.cc:1434] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:12:03.845572  1776 tablet_bootstrap.cc:492] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.859980  1776 tablet_bootstrap.cc:492] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.860812  1776 tablet_bootstrap.cc:492] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.861516  1776 ts_tablet_manager.cc:1403] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.016s	user 0.014s	sys 0.000s
I20260709 13:12:03.864814  1776 raft_consensus.cc:359] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.865482  1776 raft_consensus.cc:740] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.866089  1776 consensus_queue.cc:260] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.868094  1776 ts_tablet_manager.cc:1434] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20260709 13:12:03.869270  1776 tablet_bootstrap.cc:492] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.883030  1776 tablet_bootstrap.cc:492] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.883838  1776 tablet_bootstrap.cc:492] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.884388  1776 ts_tablet_manager.cc:1403] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.015s	user 0.012s	sys 0.000s
I20260709 13:12:03.886232  1776 raft_consensus.cc:359] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.886734  1776 raft_consensus.cc:740] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.887339  1776 consensus_queue.cc:260] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:03.889510  1776 ts_tablet_manager.cc:1434] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
I20260709 13:12:03.890648  1776 tablet_bootstrap.cc:492] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558: Bootstrap starting.
I20260709 13:12:03.904443  1776 tablet_bootstrap.cc:492] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:03.905246  1776 tablet_bootstrap.cc:492] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558: Bootstrap complete.
I20260709 13:12:03.905900  1776 ts_tablet_manager.cc:1403] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558: Time spent bootstrapping tablet: real 0.015s	user 0.012s	sys 0.003s
I20260709 13:12:03.908280  1776 raft_consensus.cc:359] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:03.908926  1776 raft_consensus.cc:740] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Initialized, Role: FOLLOWER
I20260709 13:12:03.909596  1776 consensus_queue.cc:260] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:03.911681  1776 ts_tablet_manager.cc:1434] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 13:12:04.019742  1810 raft_consensus.cc:3060] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:04.115756  1815 raft_consensus.cc:3060] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:04.206357  1815 raft_consensus.cc:3060] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:04.306929  1810 raft_consensus.cc:3060] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:12:08.581229  1027 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.005s	sys 0.012s
W20260709 13:12:08.581787  1027 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.005s	sys 0.012s
I20260709 13:12:08.886263  1986 raft_consensus.cc:493] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:08.887017  1986 raft_consensus.cc:515] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:08.891381  1986 leader_election.cc:290] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:08.949673  1986 raft_consensus.cc:493] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:08.950551  1986 raft_consensus.cc:515] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:08.953982  1986 leader_election.cc:290] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.017433  1970 raft_consensus.cc:493] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.018069  1970 raft_consensus.cc:515] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.020303  1970 leader_election.cc:290] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.030267  1989 raft_consensus.cc:493] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.030975  1986 raft_consensus.cc:493] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.031157  1970 raft_consensus.cc:493] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.034274  1970 raft_consensus.cc:515] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.047839  1993 raft_consensus.cc:493] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.049577  1993 raft_consensus.cc:515] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:09.054086  1970 leader_election.cc:290] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.056663  1970 raft_consensus.cc:493] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.033722  1986 raft_consensus.cc:515] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:09.060665  1970 raft_consensus.cc:515] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.037287  1989 raft_consensus.cc:515] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:09.080865  1970 leader_election.cc:290] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.088619  1997 raft_consensus.cc:493] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.089296  1999 raft_consensus.cc:493] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.090945  1995 raft_consensus.cc:493] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.091378  1999 raft_consensus.cc:515] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:09.091498  1995 raft_consensus.cc:515] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:09.093740  1999 leader_election.cc:290] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.096465  1989 leader_election.cc:290] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.097370  1993 leader_election.cc:290] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.101718  1986 leader_election.cc:290] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.095781  1995 leader_election.cc:290] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.102567  1986 raft_consensus.cc:493] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.089344  1997 raft_consensus.cc:515] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.107976  1986 raft_consensus.cc:515] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:09.110186  1986 leader_election.cc:290] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:09.110110  1997 leader_election.cc:290] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.111609  1986 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.112221  1986 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:09.112526  1998 raft_consensus.cc:493] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.102814  1992 raft_consensus.cc:493] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.114110  1998 raft_consensus.cc:515] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:09.114709  1992 raft_consensus.cc:515] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:09.116560  1998 leader_election.cc:290] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:09.113219  1996 raft_consensus.cc:493] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.113109  1990 raft_consensus.cc:493] T 161b9f09b3fb47febfabacf867781986 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.117352  1996 raft_consensus.cc:515] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.120489  1990 raft_consensus.cc:515] T 161b9f09b3fb47febfabacf867781986 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:09.123766  1988 raft_consensus.cc:493] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.124035  1996 leader_election.cc:290] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.124290  1990 leader_election.cc:290] T 161b9f09b3fb47febfabacf867781986 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.124614  1998 raft_consensus.cc:493] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.094712  1999 raft_consensus.cc:493] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.125111  1998 raft_consensus.cc:515] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:09.124310  1988 raft_consensus.cc:515] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.150101  1990 raft_consensus.cc:493] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.151654  1999 raft_consensus.cc:515] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:09.152581  1993 raft_consensus.cc:493] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.211607  1996 raft_consensus.cc:493] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.212558  1996 raft_consensus.cc:515] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.215435  1996 leader_election.cc:290] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.151166  1995 raft_consensus.cc:493] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.222041  1995 raft_consensus.cc:515] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:09.227203  1986 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.228520  1992 leader_election.cc:290] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.233280  1990 raft_consensus.cc:515] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.236204  1990 leader_election.cc:290] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.239952  1993 raft_consensus.cc:515] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:09.245361  1989 raft_consensus.cc:493] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.246357  1989 raft_consensus.cc:515] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:09.249560  2001 raft_consensus.cc:493] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.250339  2001 raft_consensus.cc:515] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:09.266451  1999 leader_election.cc:290] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.307025  1992 raft_consensus.cc:493] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.307834  1992 raft_consensus.cc:515] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:09.156903  1988 leader_election.cc:290] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.341643  1988 raft_consensus.cc:493] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.347018  1993 leader_election.cc:290] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.347999  1993 raft_consensus.cc:493] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.348721  1993 raft_consensus.cc:515] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:09.356608  1986 raft_consensus.cc:493] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.357416  1986 raft_consensus.cc:515] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:09.360244  1986 leader_election.cc:290] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:09.249007  1989 leader_election.cc:290] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.366852  1989 raft_consensus.cc:493] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.367552  1989 raft_consensus.cc:515] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:09.370401  1989 leader_election.cc:290] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.224789  1995 leader_election.cc:290] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.419155  2001 leader_election.cc:290] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.420192  1992 leader_election.cc:290] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.425874  1994 raft_consensus.cc:493] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.583400  1994 raft_consensus.cc:515] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:09.586133  1994 leader_election.cc:290] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:09.426177  1997 raft_consensus.cc:493] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.426385  1987 raft_consensus.cc:493] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.587150  1987 raft_consensus.cc:515] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.589334  1997 raft_consensus.cc:515] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:09.591573  1997 leader_election.cc:290] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.592434  1987 leader_election.cc:290] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.426777  2002 raft_consensus.cc:493] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.593247  2002 raft_consensus.cc:515] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:09.595844  2002 leader_election.cc:290] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.426218  2000 raft_consensus.cc:493] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.160956  1998 leader_election.cc:290] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.596601  2000 raft_consensus.cc:515] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:09.599669  2000 leader_election.cc:290] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:09.342374  1988 raft_consensus.cc:515] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:09.602555  1988 leader_election.cc:290] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:09.517745  1986 raft_consensus.cc:493] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.604480  1986 raft_consensus.cc:515] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:09.607113  1986 leader_election.cc:290] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:09.621505  2007 raft_consensus.cc:493] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.536434  1993 leader_election.cc:290] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.466610  2003 raft_consensus.cc:493] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.631944  2003 raft_consensus.cc:515] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:09.635164  2003 leader_election.cc:290] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.640580  1999 raft_consensus.cc:493] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.641440  1999 raft_consensus.cc:515] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:09.644030  1999 leader_election.cc:290] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.644622  1995 raft_consensus.cc:493] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.645468  1995 raft_consensus.cc:515] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:09.647917  1995 leader_election.cc:290] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.648578  1995 raft_consensus.cc:493] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.653155  1995 raft_consensus.cc:515] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:09.658834  2001 raft_consensus.cc:493] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.672479  2001 raft_consensus.cc:515] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:09.675225  2001 leader_election.cc:290] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.660465  1992 raft_consensus.cc:493] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.684943  1992 raft_consensus.cc:515] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:09.687660  1992 leader_election.cc:290] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.692184  1992 raft_consensus.cc:493] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.735277  1994 raft_consensus.cc:493] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.735532  1991 raft_consensus.cc:493] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.729799  1999 raft_consensus.cc:493] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.735879  1994 raft_consensus.cc:515] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:09.736159  1991 raft_consensus.cc:515] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.736720  1999 raft_consensus.cc:515] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:09.738090  1994 leader_election.cc:290] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:09.738396  1991 leader_election.cc:290] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.738787  1999 leader_election.cc:290] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.731227  2003 raft_consensus.cc:493] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.745617  2000 raft_consensus.cc:493] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.733412  1987 raft_consensus.cc:493] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.746027  2003 raft_consensus.cc:515] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.734035  1995 leader_election.cc:290] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.746501  1987 raft_consensus.cc:515] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:09.622445  2007 raft_consensus.cc:515] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.749552  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:12:09.739429  1173 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.001s	sys 0.013s
W20260709 13:12:09.750325  1173 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.002s	sys 0.014s
I20260709 13:12:09.740031  1997 raft_consensus.cc:493] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.745473  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:09.751510  1987 leader_election.cc:290] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.751686  1061 raft_consensus.cc:2468] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:12:09.752158  2000 raft_consensus.cc:515] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.752606  1997 raft_consensus.cc:515] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.754657  1997 leader_election.cc:290] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.754870  2000 leader_election.cc:290] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
W20260709 13:12:09.755465  1025 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.675s	user 0.010s	sys 0.001s
W20260709 13:12:09.755784  1025 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.010s	sys 0.001s
I20260709 13:12:09.731189  1989 raft_consensus.cc:493] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.762243  1989 raft_consensus.cc:515] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:09.766163  1994 raft_consensus.cc:493] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.732231  2008 raft_consensus.cc:493] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.749699  2007 leader_election.cc:290] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.764678  1991 raft_consensus.cc:493] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.733644  2009 raft_consensus.cc:493] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.734709  1992 raft_consensus.cc:515] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:09.760805  2004 raft_consensus.cc:493] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.776100  1995 raft_consensus.cc:493] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.776345  1999 raft_consensus.cc:493] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.767179  1989 leader_election.cc:290] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.748883  2003 leader_election.cc:290] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.781301  1853 raft_consensus.cc:493] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.776983  2001 raft_consensus.cc:493] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.844699  2001 raft_consensus.cc:515] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:09.788080  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:09.843467  1999 raft_consensus.cc:515] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:09.811429  1853 raft_consensus.cc:515] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:09.857820  1987 raft_consensus.cc:493] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.858275  2001 leader_election.cc:290] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.859511  1999 leader_election.cc:290] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.860918  1853 leader_election.cc:290] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:09.779610  2008 raft_consensus.cc:515] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:09.861496  1853 raft_consensus.cc:493] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.862084  1853 raft_consensus.cc:515] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:09.863508  2008 leader_election.cc:290] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.787377  2009 raft_consensus.cc:515] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.864931  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:09.865645  1056 raft_consensus.cc:2468] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:12:09.866302  2009 leader_election.cc:290] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.868347  1057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:09.858599  1987 raft_consensus.cc:515] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.874433  1987 leader_election.cc:290] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.870903  2011 raft_consensus.cc:493] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.900370  2011 raft_consensus.cc:515] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:09.864321  1853 leader_election.cc:290] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.903231  2011 leader_election.cc:290] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.778007  1994 raft_consensus.cc:515] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.905870  1994 leader_election.cc:290] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.783782  1991 raft_consensus.cc:515] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:09.908469  1991 leader_election.cc:290] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:09.911253  1992 leader_election.cc:290] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.849669  1995 raft_consensus.cc:515] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:09.919627  1995 leader_election.cc:290] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.883811  2004 raft_consensus.cc:515] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:09.923485  2004 leader_election.cc:290] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:09.961220  2009 raft_consensus.cc:493] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.961836  2009 raft_consensus.cc:515] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.962342  2004 raft_consensus.cc:493] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.963004  2004 raft_consensus.cc:515] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.963963  2009 leader_election.cc:290] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.964691  2009 raft_consensus.cc:493] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.965361  2004 leader_election.cc:290] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.965190  2009 raft_consensus.cc:515] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.967180  2009 leader_election.cc:290] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.970615  2013 raft_consensus.cc:493] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.971349  2013 raft_consensus.cc:515] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:09.974092  2013 leader_election.cc:290] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.977108  2014 raft_consensus.cc:493] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.977870  2014 raft_consensus.cc:515] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:09.980578  2014 leader_election.cc:290] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.981271  2014 raft_consensus.cc:493] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:09.981823  2014 raft_consensus.cc:515] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:09.984268  2014 leader_election.cc:290] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.988303  1987 raft_consensus.cc:493] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:09.989737  2003 raft_consensus.cc:493] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.990221  2012 raft_consensus.cc:493] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.990482  2003 raft_consensus.cc:515] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:09.990844  2012 raft_consensus.cc:515] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.992466  2012 leader_election.cc:290] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.993040  2003 leader_election.cc:290] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:09.993952  1987 raft_consensus.cc:515] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:09.994448  2003 raft_consensus.cc:493] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:09.994905  2008 raft_consensus.cc:493] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
W20260709 13:12:10.000617  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 147112us
I20260709 13:12:10.000994  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "240eb8c24e0142c8a23a0afdd5dccda6" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:09.995425  2012 raft_consensus.cc:493] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.002254  2012 raft_consensus.cc:515] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:10.004372  2012 leader_election.cc:290] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:10.004868  2008 raft_consensus.cc:515] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:10.006522  1099 leader_election.cc:304] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:10.007131  2008 leader_election.cc:290] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:09.997153  2016 raft_consensus.cc:493] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:10.007740  2016 raft_consensus.cc:515] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:10.008157  2002 raft_consensus.cc:2804] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:10.008565  2002 raft_consensus.cc:493] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:10.008926  2002 raft_consensus.cc:3060] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:10.009989  2016 leader_election.cc:290] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:10.010892  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:09.997666  1987 leader_election.cc:290] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:09.999420  2004 raft_consensus.cc:493] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.013023  2004 raft_consensus.cc:515] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:10.015121  2004 leader_election.cc:290] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:09.999872  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:09.995277  2003 raft_consensus.cc:515] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:10.016438  2009 raft_consensus.cc:493] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:10.016945  2009 raft_consensus.cc:515] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:10.017735  2003 leader_election.cc:290] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:10.017505  2002 raft_consensus.cc:515] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:10.019584  2009 leader_election.cc:290] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:10.021607  2012 raft_consensus.cc:493] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.022187  2012 raft_consensus.cc:515] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:10.030601  2004 raft_consensus.cc:493] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:10.031312  2004 raft_consensus.cc:515] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:10.033371  2004 leader_election.cc:290] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:10.034271  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f50253bf21654ca6937a357eb070beda" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:10.036048  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.036653  1815 raft_consensus.cc:2468] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:10.038183  2003 raft_consensus.cc:493] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:10.038779  2003 raft_consensus.cc:515] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:10.040743  2003 leader_election.cc:290] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:10.041260  2008 raft_consensus.cc:493] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.041688  2008 raft_consensus.cc:515] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:10.043494  2008 leader_election.cc:290] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:10.043993  2008 raft_consensus.cc:493] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:10.044474  2008 raft_consensus.cc:515] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:10.028309  2012 leader_election.cc:290] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:10.045276  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3fbed7be9014cf5984a19768b6269d4" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:10.046428  2008 leader_election.cc:290] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:10.047679  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8aead26aca7435abf7d4af1ab2321af" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:10.048401  1995 raft_consensus.cc:493] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.048692  2001 raft_consensus.cc:493] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.049068  1992 raft_consensus.cc:493] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:10.048722  2013 raft_consensus.cc:493] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:10.048945  1995 raft_consensus.cc:515] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:10.049821  1992 raft_consensus.cc:515] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:10.049928  2013 raft_consensus.cc:515] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:10.050931  1999 raft_consensus.cc:493] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:10.051374  1995 leader_election.cc:290] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:10.051427  1999 raft_consensus.cc:515] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:10.051795  1992 leader_election.cc:290] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:10.052135  1995 raft_consensus.cc:493] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:10.052520  1992 raft_consensus.cc:493] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.051818  2013 leader_election.cc:290] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:10.052659  1995 raft_consensus.cc:515] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:10.053090  1992 raft_consensus.cc:515] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:10.049263  2001 raft_consensus.cc:515] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:10.055109  1992 leader_election.cc:290] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:10.056547  2001 leader_election.cc:290] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:10.048926  2008 raft_consensus.cc:493] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.057145  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:10.057605  2008 raft_consensus.cc:515] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:10.058041  1059 raft_consensus.cc:2468] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:12:10.053390  2013 raft_consensus.cc:493] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.059620  2008 leader_election.cc:290] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:10.054833  1995 leader_election.cc:290] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:10.052574  1056 raft_consensus.cc:2468] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:12:10.047785  2003 raft_consensus.cc:493] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.059160  2013 raft_consensus.cc:515] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:10.060699  2003 raft_consensus.cc:515] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:10.062937  2003 leader_election.cc:290] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:10.063431  2013 leader_election.cc:290] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:10.053256  1989 raft_consensus.cc:493] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.064350  2010 raft_consensus.cc:493] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.064239  1989 raft_consensus.cc:515] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:10.064669  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.064914  2010 raft_consensus.cc:515] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:10.065382  1992 raft_consensus.cc:493] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:10.066416  1989 leader_election.cc:290] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:10.067464  2010 leader_election.cc:290] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:10.065968  1992 raft_consensus.cc:515] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:10.068339  1993 raft_consensus.cc:493] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.068871  1993 raft_consensus.cc:515] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:10.069819  1992 leader_election.cc:290] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:10.070372  1992 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:10.070755  1993 leader_election.cc:290] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:10.070870  1992 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:10.071360  1993 raft_consensus.cc:493] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:10.071889  1993 raft_consensus.cc:515] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:10.073414  2006 raft_consensus.cc:493] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.073621  1999 leader_election.cc:290] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:10.073951  2006 raft_consensus.cc:515] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:10.074994  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.075649  1815 raft_consensus.cc:2468] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:12:10.076093  2006 leader_election.cc:290] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:10.073014  1992 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:10.077979  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.080050  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0110653f11d472797f30dc22496cb12" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.080682  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.081382  1816 raft_consensus.cc:2468] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:10.082235  2010 raft_consensus.cc:493] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:10.074239  1999 raft_consensus.cc:493] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.080659  1815 raft_consensus.cc:2468] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:12:10.086962  2010 raft_consensus.cc:515] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:10.090794  2010 leader_election.cc:290] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:10.091713  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.096426  1810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.092669  1816 raft_consensus.cc:2468] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:12:10.097216  1810 raft_consensus.cc:2468] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:12:10.108469  1999 raft_consensus.cc:515] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:10.111400  1999 leader_election.cc:290] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:10.126240  1993 leader_election.cc:290] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:10.205291  2017 raft_consensus.cc:493] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.206168  2017 raft_consensus.cc:515] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:10.203866  2004 raft_consensus.cc:493] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.207302  2004 raft_consensus.cc:515] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:10.214910  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d3595711f1e49dfb415fb2f6720e66d" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:10.216152  1813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.217293  2004 leader_election.cc:290] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:10.217176  1813 raft_consensus.cc:2468] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:10.226056  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ddfc34a205c43ad81653b5000cbf394" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.227046  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:10.234079  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f50253bf21654ca6937a357eb070beda" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.234967  1815 raft_consensus.cc:2468] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:12:10.271301  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d3595711f1e49dfb415fb2f6720e66d" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.271504  1101 leader_election.cc:304] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:10.346923  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.269575  1993 raft_consensus.cc:493] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.412523  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.397568  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:10.415082  1129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.421159  2017 leader_election.cc:290] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:10.422875  1101 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 151463us
I20260709 13:12:10.423822  1101 leader_election.cc:304] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:10.413352  1815 raft_consensus.cc:2468] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:10.586567  1987 raft_consensus.cc:2804] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:10.587270  1994 raft_consensus.cc:2804] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:10.412982  1999 raft_consensus.cc:493] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:10.636803  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "240eb8c24e0142c8a23a0afdd5dccda6" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.637192  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8aead26aca7435abf7d4af1ab2321af" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.637759  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.638026  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.602830  2002 leader_election.cc:290] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:10.638860  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.585645  1993 raft_consensus.cc:515] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:10.679188  1130 raft_consensus.cc:2468] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:12:10.656558  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:10.621362  1987 raft_consensus.cc:493] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:10.697222  1812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219d22690fa145adb5b04192c1fc16bd" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.704434  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0110653f11d472797f30dc22496cb12" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:10.697014  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:10.655459  1994 raft_consensus.cc:493] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.687090  1129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "161b9f09b3fb47febfabacf867781986" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.704838  1987 raft_consensus.cc:3060] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:12:10.721647  1101 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 296485us
I20260709 13:12:10.722685  1101 leader_election.cc:304] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:10.723436  1997 raft_consensus.cc:2804] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:10.707778  1994 raft_consensus.cc:3060] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:10.726742  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867d95d9806478ba50d324c565446f6" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.727767  1101 leader_election.cc:304] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:10.729689  1101 leader_election.cc:304] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:10.731630  1101 leader_election.cc:304] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:10.719048  1993 leader_election.cc:290] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:10.739298  1810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.685565  1999 raft_consensus.cc:515] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:10.814405  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:10.815261  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
W20260709 13:12:10.815762  1101 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 84245.2us
I20260709 13:12:10.721168  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.815054  1210 raft_consensus.cc:2468] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:12:10.815130  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7780e3bac1204186aefaa6f574717fdb" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.726167  1997 raft_consensus.cc:493] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.815683  2000 raft_consensus.cc:493] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:10.819150  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35cd7c6206834f5dabfca17b22a0605a" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.819443  2000 raft_consensus.cc:515] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:10.818308  1133 raft_consensus.cc:2468] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:12:10.810469  2002 raft_consensus.cc:2804] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:10.741113  2011 raft_consensus.cc:2804] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:10.813635  1810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.826912  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:10.824694  2011 raft_consensus.cc:493] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:10.816833  2016 raft_consensus.cc:493] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:10.828950  1819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6f69ac309b6488b81b409d785a2d465" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.829910  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "545027a75bf749969ab288105031caff" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.824396  2002 raft_consensus.cc:493] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:10.834070  1210 raft_consensus.cc:2468] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:12:10.836259  1810 raft_consensus.cc:2468] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 1.
I20260709 13:12:10.851087  2002 raft_consensus.cc:3060] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:10.821373  2021 raft_consensus.cc:2804] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:10.842748  2011 raft_consensus.cc:3060] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:10.853776  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3fbed7be9014cf5984a19768b6269d4" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:10.821897  1997 raft_consensus.cc:3060] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:10.856918  1210 raft_consensus.cc:2468] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:12:10.859867  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "240eb8c24e0142c8a23a0afdd5dccda6" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.861553  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.863057  1134 raft_consensus.cc:1240] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Rejecting Update request from peer 862b41697c74421fbac82c10c245434d for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:10.864432  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.864668  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8aead26aca7435abf7d4af1ab2321af" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.865525  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219d22690fa145adb5b04192c1fc16bd" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.865852  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ddfc34a205c43ad81653b5000cbf394" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.866568  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dde48ef51c4666a28b97181ec2ee21" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.866968  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.867648  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "545027a75bf749969ab288105031caff" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.868095  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8c8635d8e54625af9c95ecbcd5653b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.868752  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.869549  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d3595711f1e49dfb415fb2f6720e66d" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.870329  1099 leader_election.cc:304] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:10.872087  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eec3e72036941a5b16c3a0157986a93" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.874307  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.875142  1987 raft_consensus.cc:515] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:10.876720  1999 leader_election.cc:290] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:10.878862  1129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eec3e72036941a5b16c3a0157986a93" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.890609  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21783150bb0b4141be797e841c3f46fa" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.907133  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b917eb6128a4341b3e3288130a65db1" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.907305  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb722a3043814db7a188e49c9861ceca" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.907904  1816 raft_consensus.cc:2468] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:12:10.908536  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.892158  1812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.901849  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f95cbeb7360c4c61b442e75e35d93e73" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
W20260709 13:12:10.902436  1101 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 86010.3us
I20260709 13:12:10.852762  2000 leader_election.cc:290] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
W20260709 13:12:10.905087  1173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 56158.9us
I20260709 13:12:10.856251  2021 raft_consensus.cc:493] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:10.915513  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.869576  1127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.918740  1813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8c8635d8e54625af9c95ecbcd5653b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.852617  2016 raft_consensus.cc:515] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:10.921351  2029 raft_consensus.cc:2804] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:10.921813  2029 raft_consensus.cc:493] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:10.922156  2029 raft_consensus.cc:3060] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:10.922624  2016 leader_election.cc:290] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:10.926189  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.926847  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3fbed7be9014cf5984a19768b6269d4" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:10.936025  2002 raft_consensus.cc:515] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:10.937883  1987 leader_election.cc:290] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:10.916708  2021 raft_consensus.cc:3060] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:10.904667  1997 raft_consensus.cc:515] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:10.947460  1130 raft_consensus.cc:1240] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Rejecting Update request from peer 862b41697c74421fbac82c10c245434d for earlier term 2. Current term is 3. Ops: []
I20260709 13:12:10.947943  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "160ad44adbce4b7f97094b248fd05415" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.949055  1813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf962b21fa14634af6e810ca29f1bc7" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.950232  1812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b89f56a91e004f408b36c36d7aee0076" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.951351  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e038f78f3e9540399aeda1f38cbea3fd" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:10.966615  1997 leader_election.cc:290] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:10.966269  1994 raft_consensus.cc:515] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
W20260709 13:12:10.974623  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 122264us
I20260709 13:12:10.988917  2029 raft_consensus.cc:515] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:10.993806  2029 leader_election.cc:290] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:10.994577  2021 raft_consensus.cc:515] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:10.997048  2021 leader_election.cc:290] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:10.988901  2011 raft_consensus.cc:515] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:10.999728  2011 leader_election.cc:290] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
W20260709 13:12:11.008692  1027 thread.cc:641] server-negotiator [worker] (thread pool) Time spent creating pthread: real 0.798s	user 0.018s	sys 0.016s
W20260709 13:12:11.009158  1027 thread.cc:608] server-negotiator [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.018s	sys 0.016s
I20260709 13:12:11.132802  1128 raft_consensus.cc:1240] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Rejecting Update request from peer 862b41697c74421fbac82c10c245434d for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:11.153021  1134 raft_consensus.cc:1240] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Rejecting Update request from peer 862b41697c74421fbac82c10c245434d for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:11.156054  1173 leader_election.cc:304] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:11.156908  2030 raft_consensus.cc:2804] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:11.157018  1173 leader_election.cc:304] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:11.157346  2030 raft_consensus.cc:493] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:11.157735  2030 raft_consensus.cc:3060] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.157846  2026 raft_consensus.cc:2804] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:11.158298  2026 raft_consensus.cc:493] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:11.158664  2026 raft_consensus.cc:3060] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.162560  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dde48ef51c4666a28b97181ec2ee21" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.167708  2026 raft_consensus.cc:515] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
W20260709 13:12:11.169376  1027 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.140.65:43721 timed out after 1.590s (SENT) [suppressed 69 similar messages]
I20260709 13:12:11.171082  2026 leader_election.cc:290] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:11.173666  2030 raft_consensus.cc:515] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:11.180608  2030 leader_election.cc:290] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:11.189751  1994 leader_election.cc:290] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:11.211398  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.211673  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb722a3043814db7a188e49c9861ceca" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.212116  1061 raft_consensus.cc:2468] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:12:11.213493  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:12:11.213610  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 343349us
I20260709 13:12:11.214967  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.215009  1025 leader_election.cc:304] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:11.216383  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867d95d9806478ba50d324c565446f6" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.216949  1060 raft_consensus.cc:2468] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:12:11.218367  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.218696  2037 raft_consensus.cc:2804] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:11.218897  1060 raft_consensus.cc:2468] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:11.219169  2037 raft_consensus.cc:493] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:11.219615  2037 raft_consensus.cc:3060] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.220182  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "483b49446c4e4a23b780921d7fabfbdf" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.217108  1025 leader_election.cc:304] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:11.221796  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11adc51543f7421cb3140972d9f0a424" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.223235  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35cd7c6206834f5dabfca17b22a0605a" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.229465  2035 raft_consensus.cc:2804] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:11.229974  2035 raft_consensus.cc:493] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:11.230373  2035 raft_consensus.cc:3060] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.244169  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff504aa514be43da8b1aac12ab550daf" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.245718  1814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867d95d9806478ba50d324c565446f6" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.273605  1814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:11.276957  1812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:11.277899  1812 raft_consensus.cc:3060] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:11.278911  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.280982  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.281497  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.282222  1202 raft_consensus.cc:2468] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:12:11.284075  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.289109  1057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3fbed7be9014cf5984a19768b6269d4" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.290012  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b75f9192e7246d2a560fd091fb68606" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.291477  1819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "161b9f09b3fb47febfabacf867781986" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.292052  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11cd8a46e76f4bc685aa01885b27cc57" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.292307  1810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.292794  1819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11adc51543f7421cb3140972d9f0a424" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.294173  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.294730  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.294811  1202 raft_consensus.cc:2468] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:12:11.295866  2036 raft_consensus.cc:493] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:11.296073  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.296383  2036 raft_consensus.cc:515] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:11.297690  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.298606  2036 leader_election.cc:290] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:11.299144  2036 raft_consensus.cc:493] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:11.299562  2036 raft_consensus.cc:515] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:11.303079  1205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7780e3bac1204186aefaa6f574717fdb" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.304360  1811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.304733  1205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b89f56a91e004f408b36c36d7aee0076" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.310609  1814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.311280  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.311815  1814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.312341  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b75f9192e7246d2a560fd091fb68606" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.313494  1810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867d95d9806478ba50d324c565446f6" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.314393  1057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b917eb6128a4341b3e3288130a65db1" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.316919  2036 leader_election.cc:290] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:11.313578  1986 raft_consensus.cc:493] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:11.312925  1814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.318605  1205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:11.411723  1204 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.361505  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:11.413611  1205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d3595711f1e49dfb415fb2f6720e66d" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.413877  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.411072  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ddfc34a205c43ad81653b5000cbf394" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.414896  1986 raft_consensus.cc:515] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:11.319620  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e038f78f3e9540399aeda1f38cbea3fd" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.415884  2002 leader_election.cc:290] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:11.393846  1812 raft_consensus.cc:2468] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 3.
W20260709 13:12:11.345086  1173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 178857us
I20260709 13:12:11.417533  1811 raft_consensus.cc:1240] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Rejecting Update request from peer 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 2. Current term is 3. Ops: []
I20260709 13:12:11.337308  1206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:11.417115  1986 leader_election.cc:290] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:11.344376  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0110653f11d472797f30dc22496cb12" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:11.411990  1208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.410266  2037 raft_consensus.cc:515] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:11.423600  2037 leader_election.cc:290] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:11.431560  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff504aa514be43da8b1aac12ab550daf" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.435159  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "160ad44adbce4b7f97094b248fd05415" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.437008  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b75f9192e7246d2a560fd091fb68606" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.437721  1056 raft_consensus.cc:2468] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:12:11.347579  1101 leader_election.cc:304] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:11.336951  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:11.446936  2014 raft_consensus.cc:2804] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:11.447630  2014 raft_consensus.cc:493] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:11.448169  2014 raft_consensus.cc:3060] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.471608  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7780e3bac1204186aefaa6f574717fdb" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
W20260709 13:12:11.477036  1101 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 129534us
I20260709 13:12:11.482542  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b75f9192e7246d2a560fd091fb68606" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.483943  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf962b21fa14634af6e810ca29f1bc7" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.485626  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11cd8a46e76f4bc685aa01885b27cc57" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.486896  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.488137  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12df324e933f422c9a8ff5d37eb7d4ef" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.490098  1129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:11.488636  1127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "483b49446c4e4a23b780921d7fabfbdf" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.489387  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f95cbeb7360c4c61b442e75e35d93e73" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.369643  2035 raft_consensus.cc:515] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:11.491410  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f50253bf21654ca6937a357eb070beda" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.498529  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "240eb8c24e0142c8a23a0afdd5dccda6" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.496801  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.497599  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb722a3043814db7a188e49c9861ceca" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:12:11.502938  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 69191.9us
I20260709 13:12:11.506547  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.507263  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35cd7c6206834f5dabfca17b22a0605a" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.508185  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.505381  1173 leader_election.cc:304] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.510049  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.510764  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21783150bb0b4141be797e841c3f46fa" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.511020  1173 leader_election.cc:304] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.492800  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:11.493377  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90e13f7e16774e68bca79e3f141ed6bc" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.514096  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2312d768cbf4b3495d5b1574b0b6eb5" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.515357  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.493582  1204 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8aead26aca7435abf7d4af1ab2321af" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.517117  2017 raft_consensus.cc:2749] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.520166  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:11.517509  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:11.490903  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:11.511926  2026 raft_consensus.cc:2749] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.505371  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6f69ac309b6488b81b409d785a2d465" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.526177  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2312d768cbf4b3495d5b1574b0b6eb5" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.536729  1059 raft_consensus.cc:2393] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 2: Already voted for candidate 9c45b6d917734835a8a37b0f4c789ae6 in this term.
I20260709 13:12:11.525089  1812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:11.513217  1054 raft_consensus.cc:3060] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.546239  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8aead26aca7435abf7d4af1ab2321af" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:11.547299  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.549440  1101 leader_election.cc:304] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:11.549988  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.545086  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.553108  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.551327  1173 leader_election.cc:304] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.545250  1814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:11.553757  1991 raft_consensus.cc:2804] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:11.611603  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b75f9192e7246d2a560fd091fb68606" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.610880  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:11.626235  2035 leader_election.cc:290] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:11.587431  1055 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6f69ac309b6488b81b409d785a2d465" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.608229  2045 raft_consensus.cc:493] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
W20260709 13:12:11.553253  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 62911.2us
I20260709 13:12:11.674757  1055 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "160ad44adbce4b7f97094b248fd05415" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.675722  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8aead26aca7435abf7d4af1ab2321af" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:11.676028  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:11.674569  2045 raft_consensus.cc:515] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:11.677719  1055 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b917eb6128a4341b3e3288130a65db1" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.581604  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.637156  1819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "483b49446c4e4a23b780921d7fabfbdf" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.680771  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.681391  1062 raft_consensus.cc:2468] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:11.682266  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:11.683367  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.683946  1062 raft_consensus.cc:2468] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:11.673483  1991 raft_consensus.cc:697] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:11.589465  1053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.613590  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ddfc34a205c43ad81653b5000cbf394" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.685012  1991 consensus_queue.cc:237] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:11.686007  1053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21783150bb0b4141be797e841c3f46fa" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.687448  1053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.592129  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11adc51543f7421cb3140972d9f0a424" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.688516  2040 raft_consensus.cc:2749] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:12:11.688663  1173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 137394us
I20260709 13:12:11.585564  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:11.611423  1206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7780e3bac1204186aefaa6f574717fdb" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.688717  1053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12df324e933f422c9a8ff5d37eb7d4ef" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:12:11.633241  1101 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 75055.6us
I20260709 13:12:11.669456  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.592259  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f95cbeb7360c4c61b442e75e35d93e73" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.545030  1204 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11cd8a46e76f4bc685aa01885b27cc57" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.586052  1057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7780e3bac1204186aefaa6f574717fdb" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.582093  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ddfc34a205c43ad81653b5000cbf394" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.646329  2014 raft_consensus.cc:515] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:11.610481  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f95cbeb7360c4c61b442e75e35d93e73" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.607517  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf962b21fa14634af6e810ca29f1bc7" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.610994  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "240eb8c24e0142c8a23a0afdd5dccda6" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.678854  2045 leader_election.cc:290] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:11.636634  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90e13f7e16774e68bca79e3f141ed6bc" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.609340  1812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "161b9f09b3fb47febfabacf867781986" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.657181  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12df324e933f422c9a8ff5d37eb7d4ef" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.552609  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0110653f11d472797f30dc22496cb12" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:11.545606  2017 consensus_queue.cc:1059] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.129s
I20260709 13:12:11.649796  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2312d768cbf4b3495d5b1574b0b6eb5" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.695710  1057 raft_consensus.cc:2393] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 2: Already voted for candidate 9c45b6d917734835a8a37b0f4c789ae6 in this term.
I20260709 13:12:11.708716  1062 raft_consensus.cc:1240] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Rejecting Update request from peer 862b41697c74421fbac82c10c245434d for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:11.692303  1173 leader_election.cc:304] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.711086  1128 raft_consensus.cc:1240] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Rejecting Update request from peer 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 2. Current term is 3. Ops: []
I20260709 13:12:11.696893  1060 raft_consensus.cc:1240] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Rejecting Update request from peer 862b41697c74421fbac82c10c245434d for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:11.713959  2017 raft_consensus.cc:2749] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.672837  1054 raft_consensus.cc:2468] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:11.715720  2017 consensus_queue.cc:1059] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:11.716864  2026 raft_consensus.cc:3055] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:11.717227  2026 raft_consensus.cc:740] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:11.717638   879 catalog_manager.cc:5697] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 2 to 3, leader changed from 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66) to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:11.718060  2026 consensus_queue.cc:260] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:11.719730  2026 raft_consensus.cc:3060] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:11.720963  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b917eb6128a4341b3e3288130a65db1" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:11.716018  1056 raft_consensus.cc:1240] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Rejecting Update request from peer 862b41697c74421fbac82c10c245434d for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:11.724349  2037 raft_consensus.cc:493] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:11.724885  2037 raft_consensus.cc:515] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:11.690289  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:11.704139  1099 leader_election.cc:304] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.706353  2040 raft_consensus.cc:3055] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:11.727723  2040 raft_consensus.cc:740] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:11.728859  2037 leader_election.cc:290] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:11.728616  2040 consensus_queue.cc:260] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:11.729919  2040 raft_consensus.cc:3060] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.700538  2014 leader_election.cc:290] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:11.733075  2017 consensus_queue.cc:1059] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:11.734760  2038 raft_consensus.cc:3055] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:11.735281  2038 raft_consensus.cc:740] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:11.736344  2038 consensus_queue.cc:260] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:11.737609  2026 consensus_queue.cc:1059] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:11.738514  1175 leader_election.cc:304] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.738555  1770 leader_election.cc:304] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:11.743436  2014 raft_consensus.cc:2749] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.745347  2045 raft_consensus.cc:2804] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:12:11.742419  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 44477.4us
W20260709 13:12:11.743122  1174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:12:11.743404  1099 leader_election.cc:304] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.745738  2045 raft_consensus.cc:493] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:11.746426  2045 raft_consensus.cc:3060] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.746863  1099 leader_election.cc:304] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.748198  1770 leader_election.cc:304] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:11.749377  1853 raft_consensus.cc:2749] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.749760  2026 raft_consensus.cc:3055] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:11.750154  2026 raft_consensus.cc:740] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:11.750378  1099 leader_election.cc:304] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.751089  2055 raft_consensus.cc:2804] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:11.750991  2026 consensus_queue.cc:260] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:11.751545  2055 raft_consensus.cc:493] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:11.751955  2055 raft_consensus.cc:3060] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:11.752208  1991 raft_consensus.cc:2749] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.756620  1173 leader_election.cc:304] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.757457  2052 raft_consensus.cc:2749] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.758324  1173 leader_election.cc:304] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.759061  2052 raft_consensus.cc:2749] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.759337  1025 leader_election.cc:304] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.759701  1173 leader_election.cc:304] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.760289  2035 raft_consensus.cc:2749] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.760450  2052 raft_consensus.cc:2749] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.760838  1099 leader_election.cc:304] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:11.761878  2055 raft_consensus.cc:515] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:11.762508  2046 raft_consensus.cc:2804] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:11.763120  2046 raft_consensus.cc:697] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 2 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:11.763777  2038 raft_consensus.cc:3060] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.764153  2055 leader_election.cc:290] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:11.763967  2046 consensus_queue.cc:237] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:11.766530  1025 leader_election.cc:304] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.746860  2014 raft_consensus.cc:2749] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:11.767870  2048 raft_consensus.cc:2749] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.768386  1099 leader_election.cc:304] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.769848  2015 raft_consensus.cc:2749] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.771974  2045 raft_consensus.cc:515] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:11.772506  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb722a3043814db7a188e49c9861ceca" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.773499  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.773806  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.774084  2045 leader_election.cc:290] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:11.774669  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219d22690fa145adb5b04192c1fc16bd" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.774758  2052 raft_consensus.cc:2804] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:11.774956  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.775300  2052 raft_consensus.cc:493] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:11.775676  2052 raft_consensus.cc:3060] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:11.775868  1127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff504aa514be43da8b1aac12ab550daf" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.776026  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eec3e72036941a5b16c3a0157986a93" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.776135  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b89f56a91e004f408b36c36d7aee0076" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.777081  1129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "545027a75bf749969ab288105031caff" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.777489  1027 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.779126  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "483b49446c4e4a23b780921d7fabfbdf" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.777209  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.778573  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.778898  1129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.778600  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.761025  1173 leader_election.cc:304] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.798251  1025 leader_election.cc:304] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.825153  1135 raft_consensus.cc:2393] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 in current term 3: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:11.801932  1770 leader_election.cc:304] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
W20260709 13:12:11.825636  1173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 64677.4us
I20260709 13:12:11.786640  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dde48ef51c4666a28b97181ec2ee21" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.826759  1173 leader_election.cc:304] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.825487  2017 raft_consensus.cc:2749] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.827529  2040 raft_consensus.cc:2749] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.783991  2052 raft_consensus.cc:515] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:11.827829  2045 raft_consensus.cc:2749] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.828456  2040 consensus_queue.cc:1059] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:11.789696  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.829538  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.829941  2017 raft_consensus.cc:3055] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:11.830374  2017 raft_consensus.cc:740] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:11.830456  2052 leader_election.cc:290] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:11.789271  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.830641  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0110653f11d472797f30dc22496cb12" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.831561  1130 raft_consensus.cc:2393] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 in current term 2: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:11.831275  2017 consensus_queue.cc:260] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
W20260709 13:12:11.818415  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 48859.3us
I20260709 13:12:11.831760  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e038f78f3e9540399aeda1f38cbea3fd" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.832404  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b917eb6128a4341b3e3288130a65db1" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.788954  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8c8635d8e54625af9c95ecbcd5653b" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.833446  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:11.833629  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:11.788563  1127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "160ad44adbce4b7f97094b248fd05415" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.831710  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.841811  1133 raft_consensus.cc:2393] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 in current term 2: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:11.834237  2042 raft_consensus.cc:2749] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:11.835216  2035 raft_consensus.cc:2749] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.843523  1129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf962b21fa14634af6e810ca29f1bc7" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.844686  2040 consensus_queue.cc:1059] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:11.848021  2040 raft_consensus.cc:3055] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:11.848135  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:11.848443  2040 raft_consensus.cc:740] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:11.849548  2040 consensus_queue.cc:260] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:11.850848  2040 raft_consensus.cc:3060] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.845940  2017 raft_consensus.cc:3060] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.827729  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.845516  1770 leader_election.cc:304] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
W20260709 13:12:11.824389  1027 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 46970.7us
I20260709 13:12:11.854164  2045 raft_consensus.cc:2749] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.854383  1027 leader_election.cc:304] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.855818  1027 leader_election.cc:304] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.788350  1175 leader_election.cc:304] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6
W20260709 13:12:11.857090  1175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 68817us
I20260709 13:12:11.857878  1175 leader_election.cc:304] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.859027  2040 raft_consensus.cc:2804] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:11.859671  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eec3e72036941a5b16c3a0157986a93" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.859977  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.860937  2040 raft_consensus.cc:493] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:11.861333  2052 raft_consensus.cc:2749] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.861462  2040 raft_consensus.cc:3060] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.862186  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35cd7c6206834f5dabfca17b22a0605a" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.863875  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11adc51543f7421cb3140972d9f0a424" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.864677   879 catalog_manager.cc:5697] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 1 to 2, leader changed from 862b41697c74421fbac82c10c245434d (127.23.140.68) to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 2 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:11.866451  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.866744  1204 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219d22690fa145adb5b04192c1fc16bd" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.867121  2048 raft_consensus.cc:2749] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.865423  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f50253bf21654ca6937a357eb070beda" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.866891  2035 raft_consensus.cc:2749] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:11.868417  1204 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.869499  1175 leader_election.cc:304] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.870415  2057 raft_consensus.cc:2804] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:11.870877  2057 raft_consensus.cc:493] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:11.871294  2057 raft_consensus.cc:3060] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:11.871325  1175 leader_election.cc:304] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.872735  2052 raft_consensus.cc:2749] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.830754  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867d95d9806478ba50d324c565446f6" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.874161  1175 leader_election.cc:304] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.875743  2017 raft_consensus.cc:2749] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.876441  1204 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b89f56a91e004f408b36c36d7aee0076" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.878481  1099 leader_election.cc:304] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.879024  1027 leader_election.cc:304] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:11.881644  2015 raft_consensus.cc:2804] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:11.881752  2035 raft_consensus.cc:2749] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:11.882053  2015 raft_consensus.cc:493] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:11.882319  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.882493  1027 leader_election.cc:304] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.882751  2015 raft_consensus.cc:3060] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.882961  1204 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff504aa514be43da8b1aac12ab550daf" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.883833  2048 raft_consensus.cc:2749] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.883934  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "545027a75bf749969ab288105031caff" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.752208  2026 raft_consensus.cc:3060] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.885298  1027 leader_election.cc:304] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.885459  1204 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.886132  2048 raft_consensus.cc:2749] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.880985  1770 leader_election.cc:304] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.884064  2035 consensus_queue.cc:1059] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:11.887200  2045 raft_consensus.cc:2749] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.887418  2035 raft_consensus.cc:3055] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:11.887884  2035 raft_consensus.cc:740] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:11.888738  2035 consensus_queue.cc:260] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:11.889398  1099 leader_election.cc:304] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.889861  2035 raft_consensus.cc:3060] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:11.890122  1991 raft_consensus.cc:2804] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:11.879587  2040 raft_consensus.cc:515] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:11.879698  1175 leader_election.cc:304] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.890595  1991 raft_consensus.cc:493] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:11.890516  1099 leader_election.cc:304] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.903115  1991 raft_consensus.cc:3060] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.904780  1994 raft_consensus.cc:2804] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:11.905323  1994 raft_consensus.cc:493] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:11.905689  1994 raft_consensus.cc:3060] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.893172  1204 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.909937  1204 raft_consensus.cc:2468] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 3.
I20260709 13:12:11.910908  1204 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.912070  1204 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b75f9192e7246d2a560fd091fb68606" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.912854  1994 raft_consensus.cc:515] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:11.894800  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dde48ef51c4666a28b97181ec2ee21" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.895125  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.895354  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2312d768cbf4b3495d5b1574b0b6eb5" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.896034  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.896996  1205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.894495  1206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12df324e933f422c9a8ff5d37eb7d4ef" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.919642  1099 leader_election.cc:304] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:11.920423  2014 raft_consensus.cc:2749] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:11.920841  1099 leader_election.cc:304] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.897840  2038 raft_consensus.cc:2749] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.922725  1853 raft_consensus.cc:2804] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:11.898108  1208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8c8635d8e54625af9c95ecbcd5653b" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.923121  1853 raft_consensus.cc:493] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:11.923512  1853 raft_consensus.cc:3060] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.887398  1027 leader_election.cc:304] T 161b9f09b3fb47febfabacf867781986 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.898957  2057 raft_consensus.cc:515] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
W20260709 13:12:11.931110  1027 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 43781.1us
I20260709 13:12:11.931653  1853 raft_consensus.cc:515] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:11.924698  1204 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0110653f11d472797f30dc22496cb12" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.895015  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11cd8a46e76f4bc685aa01885b27cc57" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.932173  2057 leader_election.cc:290] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:11.892752  2040 leader_election.cc:290] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:11.925288  1025 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:12:11.900779  1817 raft_consensus.cc:1240] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Rejecting Update request from peer f9d473477839468bbdf000a21d5f9485 for earlier term 2. Current term is 3. Ops: []
I20260709 13:12:11.900147  1818 raft_consensus.cc:1240] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Rejecting Update request from peer f9d473477839468bbdf000a21d5f9485 for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:11.933465  1204 raft_consensus.cc:2468] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:11.934031  2048 raft_consensus.cc:2749] T 161b9f09b3fb47febfabacf867781986 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.902346  2015 raft_consensus.cc:515] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:11.927906  1206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6f69ac309b6488b81b409d785a2d465" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.894155  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.929018  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e038f78f3e9540399aeda1f38cbea3fd" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.930032  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21783150bb0b4141be797e841c3f46fa" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.931861  1027 leader_election.cc:304] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.926918  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.936803  2048 raft_consensus.cc:2749] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.935094  2014 consensus_queue.cc:1059] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:11.936928  1099 leader_election.cc:304] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.898164  1175 leader_election.cc:304] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.937868  2024 raft_consensus.cc:3055] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:11.938515  2024 raft_consensus.cc:740] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:11.938596  2046 consensus_queue.cc:1059] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:11.939044  1175 leader_election.cc:304] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.939589  2046 raft_consensus.cc:3055] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:11.939781  2057 raft_consensus.cc:2749] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.940022  2046 raft_consensus.cc:740] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:11.939419  2024 consensus_queue.cc:260] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:11.938046  2040 raft_consensus.cc:2804] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:11.940934  2024 raft_consensus.cc:3060] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:11.941375  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:11.940932  2046 consensus_queue.cc:260] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:11.937824  2014 raft_consensus.cc:2804] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:11.942600  2046 raft_consensus.cc:3060] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.938134  1099 leader_election.cc:304] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:11.946537  2027 raft_consensus.cc:2749] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:11.941372  2040 raft_consensus.cc:493] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:11.943475  1994 leader_election.cc:290] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:11.945000  1853 leader_election.cc:290] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:11.945349  2014 raft_consensus.cc:493] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:11.947687  2040 raft_consensus.cc:3060] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.948567  2014 raft_consensus.cc:3060] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.950103  1099 leader_election.cc:304] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.951684  1175 leader_election.cc:304] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.951972  1099 leader_election.cc:304] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:11.952551  2057 raft_consensus.cc:2749] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.952775  1994 raft_consensus.cc:3060] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:11.953636  1853 raft_consensus.cc:2749] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:11.954463  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.942714  2015 leader_election.cc:290] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:11.955242  1205 raft_consensus.cc:2468] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:11.955193  1099 leader_election.cc:304] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:11.955695  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b75f9192e7246d2a560fd091fb68606" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.956604  1099 leader_election.cc:304] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:11.956928  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.957289  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6f69ac309b6488b81b409d785a2d465" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.957851  1099 leader_election.cc:304] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.956885  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:11.957330  1853 raft_consensus.cc:2749] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.958745  2027 raft_consensus.cc:2749] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.958902  1128 raft_consensus.cc:2393] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 in current term 3: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:11.953828  1175 leader_election.cc:304] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.960103  2015 raft_consensus.cc:2749] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.959977  2040 raft_consensus.cc:515] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:11.960009  1175 leader_election.cc:304] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.960896  2026 raft_consensus.cc:2749] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.961373  2057 raft_consensus.cc:2749] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:11.962266  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867d95d9806478ba50d324c565446f6" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.962543  1175 leader_election.cc:304] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.963596  2057 raft_consensus.cc:2749] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.963984  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:11.965632  1175 leader_election.cc:304] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.966746  2014 raft_consensus.cc:515] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:11.965742  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:11.967374  2057 raft_consensus.cc:2749] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:12:11.966174  1173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 65971.8us
I20260709 13:12:11.934227  1991 raft_consensus.cc:515] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:11.968827  2014 leader_election.cc:290] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:11.959981  1099 leader_election.cc:304] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:11.970189  1991 leader_election.cc:290] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:11.970484  1099 leader_election.cc:304] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:11.970846  1991 raft_consensus.cc:2749] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.971763  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6de2d5263c9a4dfebfe53a9eb7bc0bba" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:12:11.971992  1083 debug-util.cc:398] Leaking SignalData structure 0x7b0800272ca0 after lost signal to thread 24116
I20260709 13:12:11.971916  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f50253bf21654ca6937a357eb070beda" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:11.972465  1054 raft_consensus.cc:2468] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 3.
W20260709 13:12:11.973316  1083 debug-util.cc:398] Leaking SignalData structure 0x7b08002d3360 after lost signal to thread 919
I20260709 13:12:11.974234  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
W20260709 13:12:11.974675  1083 debug-util.cc:398] Leaking SignalData structure 0x7b08001e5200 after lost signal to thread 1086
I20260709 13:12:11.975857  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:11.976428  2015 raft_consensus.cc:2749] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.977351  2040 leader_election.cc:290] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
W20260709 13:12:11.977914  1083 debug-util.cc:398] Leaking SignalData structure 0x7b080027b180 after lost signal to thread 1844
I20260709 13:12:11.975255  1027 leader_election.cc:304] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
W20260709 13:12:11.978261  1083 debug-util.cc:398] Leaking SignalData structure 0x7b080031aca0 after lost signal to thread 1845
I20260709 13:12:11.978766  2048 raft_consensus.cc:2749] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.979660  1205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11cd8a46e76f4bc685aa01885b27cc57" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.979739  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff504aa514be43da8b1aac12ab550daf" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.980931  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.981606  1203 raft_consensus.cc:2468] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 3.
I20260709 13:12:11.982864  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12df324e933f422c9a8ff5d37eb7d4ef" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.984436  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eec3e72036941a5b16c3a0157986a93" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.978189  1175 leader_election.cc:304] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.985194  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2312d768cbf4b3495d5b1574b0b6eb5" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.985740  1994 raft_consensus.cc:2749] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.986052  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.988085  1175 leader_election.cc:304] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.986783  1205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:11.988843  2057 raft_consensus.cc:2749] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.989195  1175 leader_election.cc:304] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:11.971661  1099 leader_election.cc:304] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:11.989557  2057 raft_consensus.cc:2749] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.990052  1994 raft_consensus.cc:2749] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:12:11.990254  2057 raft_consensus.cc:2749] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.991216  1025 leader_election.cc:304] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.991961  1128 raft_consensus.cc:1240] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Rejecting Update request from peer 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:11.992241  1131 raft_consensus.cc:1240] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Rejecting Update request from peer 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:11.992343  1025 leader_election.cc:304] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:11.993470  1025 leader_election.cc:304] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:11.994457  2042 raft_consensus.cc:2749] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.994783  2048 raft_consensus.cc:2749] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.995076  2042 raft_consensus.cc:2749] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.995977  1025 leader_election.cc:304] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:11.996587  2042 raft_consensus.cc:2749] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:11.997475  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:11.999290  1025 leader_election.cc:304] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.001235  1131 raft_consensus.cc:1240] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Rejecting Update request from peer 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:12.004411  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f50253bf21654ca6937a357eb070beda" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:12.005834  2036 raft_consensus.cc:2749] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.006325  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7780e3bac1204186aefaa6f574717fdb" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:12.006651  1025 leader_election.cc:304] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.007248  1770 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.007965  2037 raft_consensus.cc:2749] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.008256  1131 raft_consensus.cc:1240] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Rejecting Update request from peer 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:12.008857  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:12.010149  1131 raft_consensus.cc:1240] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Rejecting Update request from peer 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:12.010929  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:12.011384  1205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:12.011771  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:12.012018  1205 raft_consensus.cc:2468] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 3.
I20260709 13:12:12.012849  2045 raft_consensus.cc:2804] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:12.013291  2045 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:12.013778  2045 raft_consensus.cc:3060] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:12.016773  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b917eb6128a4341b3e3288130a65db1" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:12.017580  1173 leader_election.cc:304] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.018496  2040 raft_consensus.cc:2749] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.016987  1131 raft_consensus.cc:1240] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Rejecting Update request from peer 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:12.020289  1173 leader_election.cc:304] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.021078  2040 raft_consensus.cc:2749] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.021580  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:12.022177  1202 raft_consensus.cc:2468] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:12.023725  1770 leader_election.cc:304] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:12.024825  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:12.028504  1201 raft_consensus.cc:1240] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Rejecting Update request from peer 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:12.025835  1205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7780e3bac1204186aefaa6f574717fdb" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:12.026212  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:12.026893  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11cd8a46e76f4bc685aa01885b27cc57" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:12.033555  1209 raft_consensus.cc:2468] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
I20260709 13:12:12.034896  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3fbed7be9014cf5984a19768b6269d4" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:12.035718  2048 consensus_queue.cc:1059] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:12.037294  1986 raft_consensus.cc:3055] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:12.037674  1986 raft_consensus.cc:740] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:12.037473  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:12.038748  2055 raft_consensus.cc:2804] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:12.038514  1986 consensus_queue.cc:260] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:12.039292  2055 raft_consensus.cc:493] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:12.021014  2045 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:12.039726  2055 raft_consensus.cc:3060] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:12.039904  1986 raft_consensus.cc:3060] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:12.041005  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:12.041206  2045 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:12.038662  1770 leader_election.cc:304] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.041635  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:12.042202  2033 consensus_queue.cc:1059] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:12.043390  2033 raft_consensus.cc:3055] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:12.045030  2033 raft_consensus.cc:740] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:12.043202  2060 consensus_queue.cc:1059] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 13:12:12.046128  2060 raft_consensus.cc:3055] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:12.046552  2060 raft_consensus.cc:740] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:12.046378  2048 consensus_queue.cc:1059] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:12.047437  2048 raft_consensus.cc:3055] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:12.046026  2033 consensus_queue.cc:260] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:12.047415  2060 consensus_queue.cc:260] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:12.042721  2045 raft_consensus.cc:2749] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:12.049223  2060 raft_consensus.cc:3060] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:12.049505  2033 raft_consensus.cc:3060] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:12.049352  2055 raft_consensus.cc:515] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:12.037930  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:12.037999  1205 raft_consensus.cc:2468] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
I20260709 13:12:12.052211  2048 raft_consensus.cc:740] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:12.053442  1131 raft_consensus.cc:1240] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Rejecting Update request from peer 862b41697c74421fbac82c10c245434d for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:12.053548  2048 consensus_queue.cc:260] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:12.055184  2048 raft_consensus.cc:3060] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:12.055595  2055 leader_election.cc:290] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:12.042570  1770 leader_election.cc:304] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.058497  1131 raft_consensus.cc:1240] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Rejecting Update request from peer 862b41697c74421fbac82c10c245434d for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:12.034891  2037 consensus_queue.cc:1059] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:12.059150  1770 leader_election.cc:304] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.059453  2035 raft_consensus.cc:3055] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:12.028532  1175 leader_election.cc:304] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.059899  2035 raft_consensus.cc:740] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:12.060685  2040 raft_consensus.cc:2749] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:12:12.062598  1175 leader_election.cc:304] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.062521  2035 consensus_queue.cc:260] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:12.063721  2045 raft_consensus.cc:2749] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.064008  2035 raft_consensus.cc:3060] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:12.064476  2045 raft_consensus.cc:2749] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.065408  2058 raft_consensus.cc:2749] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.067339  1770 leader_election.cc:304] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.068584  1770 leader_election.cc:304] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.066537  1175 leader_election.cc:304] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.069968  2058 raft_consensus.cc:2749] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:12:12.071795  1205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:12.072427  1205 raft_consensus.cc:2468] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:12.073115  2045 raft_consensus.cc:2749] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:12:12.073856  2045 raft_consensus.cc:2749] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.075137  1770 leader_election.cc:304] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.076073  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:12.077174  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:12.077553  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3fbed7be9014cf5984a19768b6269d4" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:12.079003  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:12.079036  2059 consensus_queue.cc:1059] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:12.080430  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:12.081656  2055 raft_consensus.cc:2764] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 13:12:12.083437  2033 raft_consensus.cc:3055] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:12.083863  2033 raft_consensus.cc:740] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:12.085799  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "160ad44adbce4b7f97094b248fd05415" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:12.086119  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b917eb6128a4341b3e3288130a65db1" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:12.085919  2033 consensus_queue.cc:260] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:12.086490  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:12.086767  1054 raft_consensus.cc:2468] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:12.086936  2033 raft_consensus.cc:3060] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:12.084823  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:12.087844  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:12.089166  1209 raft_consensus.cc:1240] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Rejecting Update request from peer f9d473477839468bbdf000a21d5f9485 for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:12.094962  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:12.097915  1025 leader_election.cc:304] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:12.098392  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:12.098860  1986 raft_consensus.cc:2749] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:12.099776  2048 consensus_queue.cc:1059] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:12.100920  2037 raft_consensus.cc:3055] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:12.101056  1173 leader_election.cc:304] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.102483  2058 raft_consensus.cc:2749] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.101140  1025 leader_election.cc:304] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:12.103560  2048 raft_consensus.cc:2749] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.103994  1025 leader_election.cc:304] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.104847  2048 raft_consensus.cc:2804] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:12.105271  1025 leader_election.cc:304] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.105229  2048 raft_consensus.cc:493] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:12.101449  2037 raft_consensus.cc:740] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:12.106070  2060 raft_consensus.cc:2804] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:12.106166  1209 raft_consensus.cc:1240] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Rejecting Update request from peer 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 2. Current term is 3. Ops: []
I20260709 13:12:12.106669  2060 raft_consensus.cc:697] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:12.106161  2048 raft_consensus.cc:3060] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:12.107026  2037 consensus_queue.cc:260] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:12.107524  2060 consensus_queue.cc:237] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:12.108511  2037 raft_consensus.cc:3060] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:12.108637  2036 consensus_queue.cc:1059] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:12:12.110071  2042 raft_consensus.cc:3055] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:12.110416  2042 raft_consensus.cc:740] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:12.111215  2042 consensus_queue.cc:260] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:12.112551  2042 raft_consensus.cc:3060] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:12.111443  1770 leader_election.cc:304] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.122550  2055 raft_consensus.cc:2749] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.123346  2046 consensus_queue.cc:1059] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:12.124433  2046 raft_consensus.cc:3055] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:12.124150  2048 raft_consensus.cc:515] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:12.124858  2046 raft_consensus.cc:740] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:12.126950  2046 consensus_queue.cc:260] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:12.131347  1099 leader_election.cc:304] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.131937  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:12.132743  1209 raft_consensus.cc:3060] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:12.134388  1994 raft_consensus.cc:2749] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.135095  2046 raft_consensus.cc:3060] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:12.143567  2048 leader_election.cc:290] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:12.144702  1099 leader_election.cc:304] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.145560  1994 raft_consensus.cc:2749] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.146620   879 catalog_manager.cc:5697] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 1 to 2, leader changed from 862b41697c74421fbac82c10c245434d (127.23.140.68) to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:12.147507  1209 raft_consensus.cc:2468] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 3.
I20260709 13:12:12.147564  1099 leader_election.cc:304] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.148633  1994 raft_consensus.cc:2749] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.149083  1025 leader_election.cc:304] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:12.149729  1099 leader_election.cc:304] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.149988  2048 raft_consensus.cc:2804] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:12.150442  1994 raft_consensus.cc:2749] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.150532  2048 raft_consensus.cc:697] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:12.151818  2048 consensus_queue.cc:237] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:12.152076  1099 leader_election.cc:304] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.153456  1099 leader_election.cc:304] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.153555  1994 raft_consensus.cc:2749] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.155186  1994 raft_consensus.cc:2749] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.156991  1209 raft_consensus.cc:1240] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Rejecting Update request from peer f9d473477839468bbdf000a21d5f9485 for earlier term 1. Current term is 2. Ops: []
W20260709 13:12:12.158382  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 47039.4us
I20260709 13:12:12.159181  1770 leader_election.cc:304] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.162544  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:12.165759  1058 raft_consensus.cc:1275] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:12.167842  2014 consensus_queue.cc:1059] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:12.168882  1994 raft_consensus.cc:3055] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:12.169261  1994 raft_consensus.cc:740] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:12.170120  1994 consensus_queue.cc:260] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:12.171500  1994 raft_consensus.cc:3060] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:12.177309  1991 consensus_queue.cc:1048] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:12.183853  1818 raft_consensus.cc:1275] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:12.185611  2014 consensus_queue.cc:1048] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:12.186954  2055 raft_consensus.cc:2749] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.188278  1770 leader_election.cc:304] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.193845  2055 raft_consensus.cc:2749] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:12.198482  1770 leader_election.cc:304] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.201558  1770 leader_election.cc:304] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.201723  2055 raft_consensus.cc:2749] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.202504  2055 raft_consensus.cc:2749] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.202907  1770 leader_election.cc:304] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.203863  2055 raft_consensus.cc:2749] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.204154  1770 leader_election.cc:304] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.204969  2055 raft_consensus.cc:2749] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.205442  1770 leader_election.cc:304] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.206223  2055 raft_consensus.cc:2749] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.207314  1770 leader_election.cc:304] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.208025  2055 raft_consensus.cc:2804] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:12.208448  2055 raft_consensus.cc:493] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:12.208863  2055 raft_consensus.cc:3060] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:12.219152  2055 raft_consensus.cc:515] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:12.219152   879 catalog_manager.cc:5697] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 2 to 3, leader changed from f9d473477839468bbdf000a21d5f9485 (127.23.140.67) to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 3 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
W20260709 13:12:12.223810  1841 debug-util.cc:398] Leaking SignalData structure 0x7b08000c3ce0 after lost signal to thread 24116
W20260709 13:12:12.225000  1841 debug-util.cc:398] Leaking SignalData structure 0x7b08002ce0c0 after lost signal to thread 919
I20260709 13:12:12.220898  2055 leader_election.cc:290] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
W20260709 13:12:12.228706  1841 debug-util.cc:398] Leaking SignalData structure 0x7b08003fce60 after lost signal to thread 1844
I20260709 13:12:12.208452  1770 leader_election.cc:304] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.232216  1770 leader_election.cc:304] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.235363  2045 raft_consensus.cc:2749] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.236073  2045 raft_consensus.cc:2749] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.238660  1770 leader_election.cc:304] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.239559  2045 raft_consensus.cc:2749] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.240844  1770 leader_election.cc:304] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.241694  2045 raft_consensus.cc:2749] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.243038  1770 leader_election.cc:304] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.243978  2045 raft_consensus.cc:2749] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:12.245190  1770 leader_election.cc:304] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.255949  1209 raft_consensus.cc:1240] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Rejecting Update request from peer f9d473477839468bbdf000a21d5f9485 for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:12.269502  2014 consensus_queue.cc:1059] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:12.270617  2014 raft_consensus.cc:3055] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:12.271013  2014 raft_consensus.cc:740] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:12.271780  2014 consensus_queue.cc:260] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:12.274219  1058 raft_consensus.cc:1240] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Rejecting Update request from peer f9d473477839468bbdf000a21d5f9485 for earlier term 2. Current term is 3. Ops: []
I20260709 13:12:12.276948  1815 raft_consensus.cc:1240] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Rejecting Update request from peer 862b41697c74421fbac82c10c245434d for earlier term 2. Current term is 3. Ops: []
I20260709 13:12:12.278403  2014 raft_consensus.cc:3060] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:12.285856  2040 consensus_queue.cc:1059] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:12.286825  2058 raft_consensus.cc:3055] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:12.287164  2058 raft_consensus.cc:740] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:12.287914  2058 consensus_queue.cc:260] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:12.290467  1994 consensus_queue.cc:1059] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:12.291247  1994 consensus_queue.cc:1243] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Peer 9c45b6d917734835a8a37b0f4c789ae6 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 13:12:12.291883  2055 raft_consensus.cc:2804] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:12:12.291996  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 46928us
I20260709 13:12:12.292469  2055 raft_consensus.cc:493] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:12.292929  2055 raft_consensus.cc:3060] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:12.292964  1770 leader_election.cc:304] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.293694  2058 raft_consensus.cc:3060] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:12.294051  2045 raft_consensus.cc:2749] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.299141  2014 raft_consensus.cc:3055] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:12.299643  2014 raft_consensus.cc:740] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:12.300653  2014 consensus_queue.cc:260] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:12.307750  2014 raft_consensus.cc:3060] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:12.309451  1770 leader_election.cc:304] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.313961  1209 raft_consensus.cc:1240] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Rejecting Update request from peer 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 2. Current term is 3. Ops: []
I20260709 13:12:12.313977  2045 raft_consensus.cc:2749] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.314872  2055 raft_consensus.cc:515] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:12.315666  1770 leader_election.cc:304] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 
I20260709 13:12:12.316464  2045 raft_consensus.cc:2804] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:12.316802  2045 raft_consensus.cc:493] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:12.317173  2045 raft_consensus.cc:3060] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:12.317606  1770 leader_election.cc:304] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.320007  1128 raft_consensus.cc:1240] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Rejecting Update request from peer 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:12.331982  2048 consensus_queue.cc:1059] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:12.332563  2059 consensus_queue.cc:1059] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:12.334868  2059 raft_consensus.cc:3055] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:12.335232  2059 raft_consensus.cc:740] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:12.335297  1128 raft_consensus.cc:1240] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Rejecting Update request from peer 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:12.333019  2060 raft_consensus.cc:3055] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:12.337953  2048 consensus_queue.cc:1059] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:12.338469  2060 raft_consensus.cc:740] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:12.338868  2037 raft_consensus.cc:3055] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:12.339251  2037 raft_consensus.cc:740] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:12.339290  2060 consensus_queue.cc:260] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:12.342339  2060 raft_consensus.cc:3060] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:12.335448  2045 raft_consensus.cc:515] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:12.347031  2045 leader_election.cc:290] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:12.352340  2045 raft_consensus.cc:2749] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:12:12.362843  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 45330.1us
I20260709 13:12:12.363550  1770 leader_election.cc:304] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.364203  2045 raft_consensus.cc:2749] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.364660  1770 leader_election.cc:304] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.365451  2045 raft_consensus.cc:2749] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.365890  1770 leader_election.cc:304] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.367103  1770 leader_election.cc:304] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.368777  2068 raft_consensus.cc:2749] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:12.373325  2045 raft_consensus.cc:2749] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:12.317111  2055 leader_election.cc:290] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:12.340090  2037 consensus_queue.cc:260] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:12.381182  2037 raft_consensus.cc:3060] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:12.340853  2059 consensus_queue.cc:260] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:12.385888  2059 raft_consensus.cc:3060] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:12.406677  1818 raft_consensus.cc:1240] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Rejecting Update request from peer f9d473477839468bbdf000a21d5f9485 for earlier term 2. Current term is 3. Ops: []
I20260709 13:12:12.408788  1991 consensus_queue.cc:1059] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:12.409950  2014 raft_consensus.cc:3055] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:12.410374  2014 raft_consensus.cc:740] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:12.411345  2014 consensus_queue.cc:260] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:12.412653  2014 raft_consensus.cc:3060] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:12:12.430928  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 63890.7us
I20260709 13:12:12.435014  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae53ca608e74146a2f89675ddca72a6" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:12.437016  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8c8635d8e54625af9c95ecbcd5653b" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:12.438993  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b89f56a91e004f408b36c36d7aee0076" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:12.440927  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:12.443512  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35cd7c6206834f5dabfca17b22a0605a" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:12.445423  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11adc51543f7421cb3140972d9f0a424" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:12.447897  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0110653f11d472797f30dc22496cb12" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:12.448761  1128 raft_consensus.cc:2393] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 in current term 2: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:12.450881  1205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0110653f11d472797f30dc22496cb12" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:12.452179  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:12.452977  1128 raft_consensus.cc:2393] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 in current term 2: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:12.454820  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:12.461297  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:12.462246  1128 raft_consensus.cc:2393] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 in current term 3: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:12.464195  1770 leader_election.cc:304] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.464457  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:12.466956  1209 raft_consensus.cc:2468] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:12.468158  2069 raft_consensus.cc:2749] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.468575  1770 leader_election.cc:304] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.469481  2069 raft_consensus.cc:2804] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:12.469878  2069 raft_consensus.cc:493] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:12.469933  1770 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.470337  2069 raft_consensus.cc:3060] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:12.470846  2055 raft_consensus.cc:2749] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:12:12.471284  1770 leader_election.cc:304] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.472070  2055 raft_consensus.cc:2749] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:12:12.473454  1770 leader_election.cc:304] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.474678  2055 raft_consensus.cc:2749] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:12.474738  1770 leader_election.cc:304] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.475636  2055 raft_consensus.cc:2749] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.475986  1770 leader_election.cc:304] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.477319  1770 leader_election.cc:304] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.478102  2045 raft_consensus.cc:2749] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:12.478547  1770 leader_election.cc:304] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.479274  2045 raft_consensus.cc:2749] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.479737  1770 leader_election.cc:304] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:12.479862  2069 raft_consensus.cc:515] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:12.480911  1770 leader_election.cc:304] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.481685  2068 raft_consensus.cc:2749] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:12.482162  2045 raft_consensus.cc:2804] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:12.482649  2045 raft_consensus.cc:697] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 2 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:12:12.483520  2045 consensus_queue.cc:237] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:12.477340  2055 raft_consensus.cc:2749] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:12.490195  2069 leader_election.cc:290] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:12.502755  1205 raft_consensus.cc:2468] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:12.505311  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b917eb6128a4341b3e3288130a65db1" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:12.526151  1128 raft_consensus.cc:2393] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 in current term 2: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:12.528285  1770 leader_election.cc:304] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.507926  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b917eb6128a4341b3e3288130a65db1" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:12.524183   879 catalog_manager.cc:5697] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 1 to 2, leader changed from 862b41697c74421fbac82c10c245434d (127.23.140.68) to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:12.532635  2045 raft_consensus.cc:2804] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:12.533298  2045 raft_consensus.cc:697] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 2 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:12:12.534514  2045 consensus_queue.cc:237] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:12.500207  1203 raft_consensus.cc:2468] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 3.
I20260709 13:12:12.537770  1205 raft_consensus.cc:1275] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:12.543608  1770 leader_election.cc:304] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.547637  2069 raft_consensus.cc:2804] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:12.548180  2069 raft_consensus.cc:697] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 3 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:12:12.548311  2037 consensus_queue.cc:1048] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:12.549108  2069 consensus_queue.cc:237] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:12.563732  1128 raft_consensus.cc:1275] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 13:12:12.564168  1128 pending_rounds.cc:85] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485: Aborting all ops after (but not including) 1
I20260709 13:12:12.564384  1128 pending_rounds.cc:107] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 13:12:12.564682  1128 raft_consensus.cc:2889] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:12:12.567251  2037 consensus_queue.cc:1048] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:12.580754  1128 raft_consensus.cc:1275] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:12.582515  1203 raft_consensus.cc:1275] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:12.598444   885 catalog_manager.cc:5697] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 1 to 2, leader changed from 862b41697c74421fbac82c10c245434d (127.23.140.68) to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:12.600195   885 catalog_manager.cc:5697] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 2 to 3, leader changed from 862b41697c74421fbac82c10c245434d (127.23.140.68) to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 3 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:12.606603  2059 consensus_queue.cc:1048] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:12.608150  2060 consensus_queue.cc:1048] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:12.630928  1058 raft_consensus.cc:2468] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:12.632511  1770 leader_election.cc:304] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:12.633822  2045 raft_consensus.cc:2804] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:12.634334  2045 raft_consensus.cc:697] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 2 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:12:12.635169  2045 consensus_queue.cc:237] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:12.709894   879 catalog_manager.cc:5697] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 1 to 2, leader changed from 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66) to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:12.877513  1203 raft_consensus.cc:1275] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:12.879289  2045 consensus_queue.cc:1048] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:12.938831  1131 raft_consensus.cc:1275] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:12.941362  2055 consensus_queue.cc:1048] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:12.952980  1205 raft_consensus.cc:1275] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:12.961766  2069 consensus_queue.cc:1048] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:12.983390  1128 raft_consensus.cc:1275] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:12.985146  2055 consensus_queue.cc:1048] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:13.008219  1127 raft_consensus.cc:1275] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:13.019878  2055 consensus_queue.cc:1048] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:13.129541  2040 raft_consensus.cc:493] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:13.131278  2040 raft_consensus.cc:515] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.155279  2040 leader_election.cc:290] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.166617  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:13.167855  1058 raft_consensus.cc:2468] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:13.173187  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:13.174652  1128 raft_consensus.cc:2393] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 2: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:13.173411  1058 raft_consensus.cc:1275] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:13.186388  2055 consensus_queue.cc:1048] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:13.200090  1173 leader_election.cc:304] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:13.205560  2086 raft_consensus.cc:2804] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:13.206013  2086 raft_consensus.cc:493] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:13.206476  2086 raft_consensus.cc:3060] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:13.218094  1209 raft_consensus.cc:1275] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:13.221812  2055 consensus_queue.cc:1048] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 13:12:13.248487  1127 raft_consensus.cc:1275] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:13.258716  2045 consensus_queue.cc:1048] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:13.296192  2033 raft_consensus.cc:493] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:13.297142  2033 raft_consensus.cc:515] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:13.304847  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:13.314709  2060 raft_consensus.cc:493] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:13.315500  2060 raft_consensus.cc:515] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:13.318327  2033 leader_election.cc:290] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.321812  1127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:13.322806  1127 raft_consensus.cc:2468] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
I20260709 13:12:13.329371  1025 leader_election.cc:304] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:13.330374  2033 raft_consensus.cc:2804] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:13.330762  2033 raft_consensus.cc:493] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:13.331127  2033 raft_consensus.cc:3060] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:13.355381  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:13.356997  2060 leader_election.cc:290] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:13.358856  1205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:13.359817  1205 raft_consensus.cc:2468] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 3.
I20260709 13:12:13.375989  1025 leader_election.cc:304] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:13.377555  1986 raft_consensus.cc:2804] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:12:13.377969  1986 raft_consensus.cc:493] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:13.378295  1986 raft_consensus.cc:3060] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:13.384470  2086 raft_consensus.cc:515] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.400624  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:13.399744  2086 leader_election.cc:290] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.402496  2033 raft_consensus.cc:515] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:13.405678  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:13.408439  1127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:13.409632  1127 raft_consensus.cc:3060] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:13.409921  2046 raft_consensus.cc:493] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:13.410555  2046 raft_consensus.cc:515] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:13.413043  2046 leader_election.cc:290] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:13.420652  1127 raft_consensus.cc:2468] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 3.
I20260709 13:12:13.407696  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:13.440166  1128 raft_consensus.cc:2393] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 2: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:13.406708  2033 leader_election.cc:290] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.445317  1025 leader_election.cc:304] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:13.446363  2033 raft_consensus.cc:2804] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:13.387337  1986 raft_consensus.cc:515] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:13.393307 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:12:13.413676  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:13.447916  2033 raft_consensus.cc:697] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:13.414402  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:13.449328  1056 raft_consensus.cc:2468] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:13.449349  2033 consensus_queue.cc:237] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:13.450724  2045 raft_consensus.cc:493] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:13.451387  2045 raft_consensus.cc:515] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:13.451795  1099 leader_election.cc:304] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:13.453904  2045 leader_election.cc:290] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.454614  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:13.455454  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:13.456633  1128 raft_consensus.cc:2368] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 for earlier term 2. Current term is 3.
I20260709 13:12:13.457314  2046 raft_consensus.cc:2804] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:13.457708  2046 raft_consensus.cc:493] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:13.457926  1770 leader_election.cc:400] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Vote denied by peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869) with higher term. Message: Invalid argument: T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 for earlier term 2. Current term is 3.
I20260709 13:12:13.458042  2046 raft_consensus.cc:3060] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:13.458282  1770 leader_election.cc:403] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:12:13.459333  2045 raft_consensus.cc:3060] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:12:13.475153  1986 leader_election.cc:290] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 4 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:13.476046  1058 raft_consensus.cc:2468] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:13.476703  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:13.477166  1173 leader_election.cc:304] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:13.477340  1209 raft_consensus.cc:3060] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:13.478096  2045 raft_consensus.cc:2749] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869) with higher term. Message: Invalid argument: T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 for earlier term 2. Current term is 3.
I20260709 13:12:13.478413  2017 raft_consensus.cc:2804] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:13.478986  2017 raft_consensus.cc:697] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:13.479818  2017 consensus_queue.cc:237] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.483323  2042 raft_consensus.cc:493] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:13.483971  2042 raft_consensus.cc:515] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:13.489040  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:13.492522  1209 raft_consensus.cc:2468] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 4.
I20260709 13:12:13.492802  2042 leader_election.cc:290] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:13.581661  2048 raft_consensus.cc:493] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:13.567773  2059 raft_consensus.cc:493] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:13.587051  2048 raft_consensus.cc:515] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.587421  2059 raft_consensus.cc:515] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:13.551034  2085 raft_consensus.cc:493] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:13.497370  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f50253bf21654ca6937a357eb070beda" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:13.592353  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:13.578052   879 catalog_manager.cc:5697] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:13.594075  2048 leader_election.cc:290] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:13.594282  1128 raft_consensus.cc:2468] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
I20260709 13:12:13.595817  1025 leader_election.cc:304] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:13.596652  2048 raft_consensus.cc:2804] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:13.597105  2048 raft_consensus.cc:493] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:13.598002  2048 raft_consensus.cc:3060] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:13.547230  2086 raft_consensus.cc:493] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:13.537660  2091 raft_consensus.cc:493] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:13.589665  2085 raft_consensus.cc:515] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:13.546434  2040 raft_consensus.cc:493] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:13.538818  2037 raft_consensus.cc:493] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:13.603904  1818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f50253bf21654ca6937a357eb070beda" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:13.630071  2045 raft_consensus.cc:493] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:13.630715  2045 raft_consensus.cc:515] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:13.631155  2098 raft_consensus.cc:493] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:13.631855  2098 raft_consensus.cc:515] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.632355  1991 raft_consensus.cc:493] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:13.634956  2091 raft_consensus.cc:515] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.635448  1991 raft_consensus.cc:515] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.637542  1991 leader_election.cc:290] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:13.632966  2045 leader_election.cc:290] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.634575  1127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:13.640054  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8aead26aca7435abf7d4af1ab2321af" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:13.640611  2098 leader_election.cc:290] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.605288  2046 raft_consensus.cc:515] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:13.645004  2046 leader_election.cc:290] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:13.645838  1127 raft_consensus.cc:2368] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 for earlier term 2. Current term is 3.
I20260709 13:12:13.608229  2048 raft_consensus.cc:515] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.647822  2048 leader_election.cc:290] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:13.629086  1818 raft_consensus.cc:2468] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:12:13.608415  1986 raft_consensus.cc:493] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:13.649868  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:13.525772  2060 raft_consensus.cc:493] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:13.650091  1986 raft_consensus.cc:515] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.650750  2060 raft_consensus.cc:515] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.652225  1986 leader_election.cc:290] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:13.653772  2060 leader_election.cc:290] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:13.654076  1817 raft_consensus.cc:2393] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 3: Already voted for candidate 294145900dd3496cb4a1d722613e1558 in this term.
I20260709 13:12:13.609128  2059 leader_election.cc:290] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.657670  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:13.641243  2033 raft_consensus.cc:493] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:13.658605  1133 raft_consensus.cc:2468] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
I20260709 13:12:13.658656  2033 raft_consensus.cc:515] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:13.659967  1770 leader_election.cc:400] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Vote denied by peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869) with higher term. Message: Invalid argument: T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 for earlier term 2. Current term is 3.
I20260709 13:12:13.660277  1770 leader_election.cc:403] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:12:13.594282  1203 raft_consensus.cc:2468] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
I20260709 13:12:13.661223  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:13.660275  1025 leader_election.cc:304] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:13.661944  1058 raft_consensus.cc:3060] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:13.610481  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:13.663367  1128 raft_consensus.cc:2468] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 3.
I20260709 13:12:13.609686  2085 leader_election.cc:290] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:13.665612  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:13.666424  1025 leader_election.cc:304] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:13.613737  1812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:13.667387  2048 raft_consensus.cc:2804] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:13.667682  1986 raft_consensus.cc:2804] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:13.667809  2048 raft_consensus.cc:493] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:13.668237  1986 raft_consensus.cc:493] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:13.668454  2048 raft_consensus.cc:3060] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:13.668763  1986 raft_consensus.cc:3060] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:13.604739  2086 raft_consensus.cc:515] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.672029  2085 raft_consensus.cc:493] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:13.615581  2040 raft_consensus.cc:515] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.672647  2085 raft_consensus.cc:515] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.667966  2045 raft_consensus.cc:3060] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:12:13.667819  1812 raft_consensus.cc:2468] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:12:13.656817  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:13.677871  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:13.679342  1101 leader_election.cc:304] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:13.680212  2094 raft_consensus.cc:493] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:13.680841  2094 raft_consensus.cc:515] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.681340  1127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:13.681896  1991 raft_consensus.cc:2804] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:13.682436  1991 raft_consensus.cc:493] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:13.626679  2037 raft_consensus.cc:515] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.683239  2094 leader_election.cc:290] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.682948  1991 raft_consensus.cc:3060] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:13.683746  1056 raft_consensus.cc:2468] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 1.
I20260709 13:12:13.686748  2040 leader_election.cc:290] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.677081  1131 raft_consensus.cc:2468] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
W20260709 13:12:13.696520  1173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 111788us
I20260709 13:12:13.698591  1099 leader_election.cc:304] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:13.700243  2091 leader_election.cc:290] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.700773  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:13.701538  1059 raft_consensus.cc:2468] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:13.701588  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:13.702333  1060 raft_consensus.cc:2368] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d for earlier term 2. Current term is 3.
I20260709 13:12:13.703007  2017 raft_consensus.cc:493] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:13.703459  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8aead26aca7435abf7d4af1ab2321af" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:13.704100  2037 leader_election.cc:290] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:13.705462  2086 leader_election.cc:290] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.705670  2085 leader_election.cc:290] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:13.707787  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:13.707527  2048 raft_consensus.cc:515] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.708658  1133 raft_consensus.cc:2393] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 2: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:13.710721   883 catalog_manager.cc:5697] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 1 to 2, leader changed from 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 2 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:13.683837  2094 raft_consensus.cc:493] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:13.704144  1131 raft_consensus.cc:2468] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
I20260709 13:12:13.718204  2094 raft_consensus.cc:515] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:13.720077  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:13.729895  1203 raft_consensus.cc:2468] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:13.730130  1129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:13.720767  2048 leader_election.cc:290] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:13.703752  2017 raft_consensus.cc:515] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.731650  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8aead26aca7435abf7d4af1ab2321af" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:13.732347  1209 raft_consensus.cc:2468] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:13.732759  2017 leader_election.cc:290] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.721979  1027 leader_election.cc:304] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:13.733737  2048 raft_consensus.cc:2804] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:13.734112  2048 raft_consensus.cc:493] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:13.734462  2048 raft_consensus.cc:3060] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:13.737623  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3fbed7be9014cf5984a19768b6269d4" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:13.738495  1054 raft_consensus.cc:2468] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:13.741641  2094 leader_election.cc:290] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:13.757364  1055 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:13.681975  1127 raft_consensus.cc:3060] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:13.726502  1025 leader_election.cc:304] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:13.741957  2048 raft_consensus.cc:515] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:13.746986  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:13.750898  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:13.723208  2033 leader_election.cc:290] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.753796  1205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:13.772987  1205 raft_consensus.cc:2468] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:12:13.753862  1129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:13.754423  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:13.756860  2085 raft_consensus.cc:2804] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:13.724905  2014 raft_consensus.cc:493] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:13.781279  2014 raft_consensus.cc:515] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:13.782104  2085 raft_consensus.cc:493] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:13.782315  1129 raft_consensus.cc:2468] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
I20260709 13:12:13.782573  2085 raft_consensus.cc:3060] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:13.764251  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:13.764833  2069 raft_consensus.cc:493] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:13.766765  1209 raft_consensus.cc:2468] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 1.
I20260709 13:12:13.772197  1055 raft_consensus.cc:2468] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:13.737002  2045 raft_consensus.cc:2749] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869) with higher term. Message: Invalid argument: T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 for earlier term 2. Current term is 3.
I20260709 13:12:13.747061  1058 raft_consensus.cc:2468] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 3.
I20260709 13:12:13.780515  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8aead26aca7435abf7d4af1ab2321af" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:13.775374  2048 leader_election.cc:290] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:13.791070  1201 raft_consensus.cc:2393] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 2: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
I20260709 13:12:13.793619  2069 raft_consensus.cc:515] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:13.797034  1205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:13.797871  1205 raft_consensus.cc:2393] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 2: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
I20260709 13:12:13.798118  2069 leader_election.cc:290] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.799465  1055 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:13.775264  1986 raft_consensus.cc:515] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:13.795874  1173 leader_election.cc:304] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:13.800175  1055 raft_consensus.cc:2468] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 3.
I20260709 13:12:13.800531  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f50253bf21654ca6937a357eb070beda" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:13.801616  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:13.802100  1817 raft_consensus.cc:3060] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:12:13.802874  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:13.803498  1058 raft_consensus.cc:2468] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:13.805233  1202 raft_consensus.cc:2393] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 2: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
I20260709 13:12:13.784078  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:13.791126  1812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:13.807152  1814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:13.807749  1812 raft_consensus.cc:2393] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 2: Already voted for candidate 294145900dd3496cb4a1d722613e1558 in this term.
I20260709 13:12:13.800110  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:13.783607  2033 raft_consensus.cc:2804] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:12:13.808769  1056 raft_consensus.cc:2468] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:13.809293  2033 raft_consensus.cc:697] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:13.809633  1054 raft_consensus.cc:2468] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 3.
I20260709 13:12:13.810202  1986 leader_election.cc:290] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.810228  2033 consensus_queue.cc:237] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:13.812426  2014 leader_election.cc:290] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:13.813028  2014 raft_consensus.cc:493] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:13.816435  1173 leader_election.cc:400] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097) with higher term. Message: Invalid argument: T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d for earlier term 2. Current term is 3.
I20260709 13:12:13.816848  1173 leader_election.cc:403] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:12:13.817729  1173 leader_election.cc:304] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:13.821750  1173 leader_election.cc:304] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:13.824146  2098 raft_consensus.cc:2804] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:13.824563  2098 raft_consensus.cc:493] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:13.824944  2098 raft_consensus.cc:3060] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:13.813508  2014 raft_consensus.cc:515] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.832937  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:13.833802  1816 raft_consensus.cc:2468] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:13.833031  1127 raft_consensus.cc:2468] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 3.
I20260709 13:12:13.822340  2017 raft_consensus.cc:3060] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:12:13.821381  2086 raft_consensus.cc:2804] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:13.837553  2086 raft_consensus.cc:493] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:13.837920  2086 raft_consensus.cc:3060] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:13.843294  1817 raft_consensus.cc:2468] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 3.
I20260709 13:12:13.849750  1173 leader_election.cc:304] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:13.845346  1027 leader_election.cc:304] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:13.857875  1173 leader_election.cc:304] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:13.858083  1815 raft_consensus.cc:2468] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 3.
I20260709 13:12:13.859040  2094 raft_consensus.cc:2804] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:13.859457  2094 raft_consensus.cc:493] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:13.859802  2094 raft_consensus.cc:3060] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:13.859824  2040 raft_consensus.cc:2804] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:13.860357  2040 raft_consensus.cc:493] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:13.860729  2040 raft_consensus.cc:3060] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:13.869400  2091 raft_consensus.cc:2804] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:13.858719  1173 leader_election.cc:304] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:13.877614  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:13.813078  1991 raft_consensus.cc:515] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:13.870429  2048 raft_consensus.cc:2804] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:13.864094  2098 raft_consensus.cc:515] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.871606  2091 raft_consensus.cc:493] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:13.897498  2097 raft_consensus.cc:493] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:13.902037  2097 raft_consensus.cc:515] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:13.903661  1127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:13.904261  2097 leader_election.cc:290] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:13.904361  1127 raft_consensus.cc:2468] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 3.
I20260709 13:12:13.892546  2014 leader_election.cc:290] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:13.907003  2098 leader_election.cc:290] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.911229  1986 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:13.911609  1131 raft_consensus.cc:2468] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
I20260709 13:12:13.911878  1986 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:13.902070  2095 raft_consensus.cc:2804] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:12:13.913726  2095 raft_consensus.cc:493] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:13.914047  2095 raft_consensus.cc:3060] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:13.914810  1770 leader_election.cc:304] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:13.914074  1986 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.890938  2085 raft_consensus.cc:515] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.900841  2091 raft_consensus.cc:3060] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:13.918991  2048 raft_consensus.cc:697] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:13.938592  2048 consensus_queue.cc:237] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:13.941395  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:13.918846  2055 raft_consensus.cc:2804] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:12:13.942291  2055 raft_consensus.cc:493] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:13.942624  2055 raft_consensus.cc:3060] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:13.943385  2094 raft_consensus.cc:515] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.944377  1812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:13.945533  1812 raft_consensus.cc:2393] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 3: Already voted for candidate 294145900dd3496cb4a1d722613e1558 in this term.
I20260709 13:12:13.950244  2017 raft_consensus.cc:2749] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097) with higher term. Message: Invalid argument: T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d for earlier term 2. Current term is 3.
I20260709 13:12:13.953836  1202 raft_consensus.cc:2393] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 3: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
I20260709 13:12:13.957559  2094 leader_election.cc:290] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:13.959343  1991 leader_election.cc:290] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:13.951303  2086 raft_consensus.cc:515] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.983062  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
W20260709 13:12:13.983173  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 213986us
I20260709 13:12:13.983820  1815 raft_consensus.cc:3060] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:12:13.932080  2040 raft_consensus.cc:515] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:13.988418  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:13.991737  2085 leader_election.cc:290] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:13.994234  1055 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:12:13.996196  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 269725us
I20260709 13:12:14.001312  2045 raft_consensus.cc:493] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:14.002494  1101 leader_election.cc:304] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:14.004828  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3fbed7be9014cf5984a19768b6269d4" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:14.015388  2045 raft_consensus.cc:515] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:14.017354  1055 raft_consensus.cc:2468] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:14.017655  2045 leader_election.cc:290] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:14.018821  1815 raft_consensus.cc:2468] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 3.
I20260709 13:12:14.012185  1056 raft_consensus.cc:3060] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:14.020040  1173 leader_election.cc:304] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:14.022868  1101 leader_election.cc:304] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:14.022854  1025 leader_election.cc:304] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:14.025414  2110 raft_consensus.cc:2804] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:14.025944  2110 raft_consensus.cc:697] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:14.027004  2110 consensus_queue.cc:237] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:14.031672  2112 raft_consensus.cc:2804] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:12:14.032086  2112 raft_consensus.cc:493] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:14.032409  2112 raft_consensus.cc:3060] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:14.032963  1818 raft_consensus.cc:1275] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:14.037863  1025 leader_election.cc:304] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d
I20260709 13:12:14.040782  2085 raft_consensus.cc:2804] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:14.040628  2095 raft_consensus.cc:515] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:14.037335  2105 raft_consensus.cc:2804] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:14.041343  2085 raft_consensus.cc:697] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:14.041558  2105 raft_consensus.cc:493] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:14.041981  2105 raft_consensus.cc:3060] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:14.042323  2085 consensus_queue.cc:237] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:14.042867  2095 leader_election.cc:290] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:14.037487  2033 consensus_queue.cc:1048] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260709 13:12:14.045424  1055 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:14.046005  1055 raft_consensus.cc:3060] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:14.046088  2014 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:14.046759  2014 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:14.051031  1819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:14.051337  2040 leader_election.cc:290] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:14.051736  1819 raft_consensus.cc:2468] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 3.
I20260709 13:12:14.051918  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:14.055359  1101 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:14.056695  2102 raft_consensus.cc:2804] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:12:14.057060  2102 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:14.057402  2102 raft_consensus.cc:3060] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:12:14.068436  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 63949.4us
W20260709 13:12:14.062219  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 53287.3us
I20260709 13:12:14.048887  2014 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:14.071383  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:14.058777  2112 raft_consensus.cc:515] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:14.072737  1060 raft_consensus.cc:2468] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 1.
I20260709 13:12:14.071481  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3fbed7be9014cf5984a19768b6269d4" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:14.071847  1986 raft_consensus.cc:2804] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:14.073889  1986 raft_consensus.cc:493] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:14.071780  1127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:14.074211  1131 raft_consensus.cc:2393] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 2: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:14.074558  1127 raft_consensus.cc:2468] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:14.078919  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:14.074273  1986 raft_consensus.cc:3060] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:14.079532  1060 raft_consensus.cc:3060] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:12:14.084820  2045 raft_consensus.cc:493] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:14.085420  2045 raft_consensus.cc:515] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:14.085978  2086 leader_election.cc:290] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:14.086922  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:14.088927  2045 leader_election.cc:290] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:14.090018  1127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:14.090677  1127 raft_consensus.cc:2468] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:14.090890  2105 raft_consensus.cc:515] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:14.088714  1060 raft_consensus.cc:2468] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 4.
I20260709 13:12:14.094746  1173 leader_election.cc:304] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:14.100230  1059 raft_consensus.cc:2468] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:14.109308  1173 leader_election.cc:304] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:14.092998  2105 leader_election.cc:290] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:14.088855  1056 raft_consensus.cc:2468] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 3.
I20260709 13:12:14.116461  1127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:14.117394  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
W20260709 13:12:14.118042  1175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 56809.2us
I20260709 13:12:14.118381  1054 raft_consensus.cc:2468] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:14.118889  1127 raft_consensus.cc:2393] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 2: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:14.104382  2091 raft_consensus.cc:515] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:14.119625  1055 raft_consensus.cc:2468] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 4.
I20260709 13:12:14.119807  1173 leader_election.cc:304] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:14.120893  2111 raft_consensus.cc:2804] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:14.121814  1986 raft_consensus.cc:515] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:14.110692  2086 raft_consensus.cc:2804] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:12:14.121032  2091 leader_election.cc:290] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:14.123821  1986 leader_election.cc:290] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:14.125062  2091 raft_consensus.cc:2804] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:14.125501  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f50253bf21654ca6937a357eb070beda" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:14.125695  2086 raft_consensus.cc:697] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 4 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:14.126122  1202 raft_consensus.cc:3060] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:14.126655  2086 consensus_queue.cc:237] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:14.128528  2111 raft_consensus.cc:697] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:14.129158  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:14.129762  1815 raft_consensus.cc:2468] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 3.
I20260709 13:12:14.129501  2111 consensus_queue.cc:237] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:14.115461  1058 raft_consensus.cc:2368] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d for earlier term 2. Current term is 3.
I20260709 13:12:14.132042  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:14.132405  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:14.132598  1816 raft_consensus.cc:3060] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:14.132925  1815 raft_consensus.cc:3060] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:12:14.118089  1131 raft_consensus.cc:2468] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:14.117429  2055 raft_consensus.cc:515] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:14.140152  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:14.140741  1209 raft_consensus.cc:3060] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:14.098544  2112 leader_election.cc:290] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:14.148494  1173 leader_election.cc:304] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:14.149899  1173 leader_election.cc:304] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:14.151381  1815 raft_consensus.cc:2468] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 4.
I20260709 13:12:14.152135  1173 leader_election.cc:400] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Vote denied by peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097) with higher term. Message: Invalid argument: T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d for earlier term 2. Current term is 3.
I20260709 13:12:14.152411  1173 leader_election.cc:403] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 13:12:14.156631  1202 raft_consensus.cc:2468] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 3.
I20260709 13:12:14.163928  1816 raft_consensus.cc:2468] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 4.
I20260709 13:12:14.143714  2091 raft_consensus.cc:697] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 3 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:14.157167  2094 raft_consensus.cc:2804] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:12:14.166688  1817 raft_consensus.cc:2368] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 3. Current term is 4.
I20260709 13:12:14.156371  2095 raft_consensus.cc:2804] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:14.184464  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8aead26aca7435abf7d4af1ab2321af" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:14.185602  1201 raft_consensus.cc:2393] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 2: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
I20260709 13:12:14.194499  1027 leader_election.cc:400] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Vote denied by peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721) with higher term. Message: Invalid argument: T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 3. Current term is 4.
I20260709 13:12:14.194854  1027 leader_election.cc:403] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 13:12:14.192544  1025 leader_election.cc:304] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:14.174019  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:14.212997  1058 raft_consensus.cc:3060] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:14.213287  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3fbed7be9014cf5984a19768b6269d4" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:14.213928  1054 raft_consensus.cc:2468] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:14.214924  2085 raft_consensus.cc:493] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:14.215474  2085 raft_consensus.cc:515] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:14.217046  1055 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:14.217922  1129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3fbed7be9014cf5984a19768b6269d4" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:14.218288  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:14.218776  1129 raft_consensus.cc:2393] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 2: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:14.199667  1025 leader_election.cc:304] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d
I20260709 13:12:14.220383  1025 leader_election.cc:304] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d
I20260709 13:12:14.221699  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0afcfb9359a421b85fabc92b7733d9b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:14.222332  1131 raft_consensus.cc:3060] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:14.225155  2100 raft_consensus.cc:2804] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:14.226754  1060 raft_consensus.cc:2468] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:14.228050  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:14.228601  1202 raft_consensus.cc:3060] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:14.229974  1099 leader_election.cc:304] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:14.200863  1209 raft_consensus.cc:2468] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 3.
I20260709 13:12:14.199333  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8aead26aca7435abf7d4af1ab2321af" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:14.233546  1059 raft_consensus.cc:3060] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:14.204828  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:14.235976  1133 raft_consensus.cc:3060] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:14.206496  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3fbed7be9014cf5984a19768b6269d4" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:14.206928  2055 leader_election.cc:290] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 4 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:14.189254  2094 raft_consensus.cc:697] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 4 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:14.153498  2102 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:14.193723  2095 raft_consensus.cc:697] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:14.199380  1986 raft_consensus.cc:2804] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:14.221467  2085 leader_election.cc:290] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
W20260709 13:12:14.258214  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 159841us
I20260709 13:12:14.259045  1770 leader_election.cc:304] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:14.184609  2091 consensus_queue.cc:237] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:14.234740  2040 raft_consensus.cc:2749] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097) with higher term. Message: Invalid argument: T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d for earlier term 2. Current term is 3.
I20260709 13:12:14.260908  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:14.255479  2033 raft_consensus.cc:3060] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:14.266356  1209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:14.256363  1986 raft_consensus.cc:493] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:14.267177  1134 raft_consensus.cc:2393] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 2: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:14.260880  2045 raft_consensus.cc:2804] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:14.270993  1202 raft_consensus.cc:2468] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 3.
W20260709 13:12:14.279621  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 50696.8us
I20260709 13:12:14.280432  1025 leader_election.cc:304] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:14.271193  1986 raft_consensus.cc:3060] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:14.268149  2100 raft_consensus.cc:697] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:14.272496  2045 raft_consensus.cc:493] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:14.323107  2125 raft_consensus.cc:2804] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:14.273010  1129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:14.329887  1059 raft_consensus.cc:2468] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 3.
I20260709 13:12:14.320737  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:14.346333  1135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:14.339040  2095 consensus_queue.cc:237] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:14.349319  1129 raft_consensus.cc:2368] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 3. Current term is 4.
I20260709 13:12:14.350011  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8aead26aca7435abf7d4af1ab2321af" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:14.350116  2125 raft_consensus.cc:697] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:14.350986  1128 raft_consensus.cc:2368] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 2. Current term is 3.
I20260709 13:12:14.351554  2125 consensus_queue.cc:237] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:14.356314  1058 raft_consensus.cc:2468] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 3.
I20260709 13:12:14.358127  2033 raft_consensus.cc:2749] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721) with higher term. Message: Invalid argument: T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 3. Current term is 4.
I20260709 13:12:14.333884  2045 raft_consensus.cc:3060] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:14.366560  1131 raft_consensus.cc:2468] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 3.
I20260709 13:12:14.359859  1133 raft_consensus.cc:2468] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 3.
I20260709 13:12:14.368016  1135 raft_consensus.cc:2468] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 3.
I20260709 13:12:14.367534  1099 leader_election.cc:304] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
W20260709 13:12:14.390602  1027 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 161113us
I20260709 13:12:14.396289  1173 leader_election.cc:304] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:14.321210  2094 consensus_queue.cc:237] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:14.346374  2121 raft_consensus.cc:493] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:14.376567  2105 raft_consensus.cc:2764] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260709 13:12:14.376827  2110 raft_consensus.cc:2749] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:14.393682  1128 raft_consensus.cc:1275] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:14.417534  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:14.417783  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:14.418341  1058 raft_consensus.cc:2393] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 in current term 4: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
I20260709 13:12:14.423283  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:14.424290  1132 raft_consensus.cc:2393] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 2: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:14.426183  1057 raft_consensus.cc:1275] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:14.430044  2100 consensus_queue.cc:237] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:14.419409  1770 leader_election.cc:304] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:14.405213  2097 raft_consensus.cc:2804] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:14.432132  2097 raft_consensus.cc:697] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 3 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:14.432556  2102 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:14.433095  2097 consensus_queue.cc:237] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:14.405640  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:14.434674  2121 raft_consensus.cc:515] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:14.436146  2103 raft_consensus.cc:493] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:14.436708  2103 raft_consensus.cc:515] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:14.437638  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8aead26aca7435abf7d4af1ab2321af" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:14.438297  1203 raft_consensus.cc:3060] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:14.439044  2103 leader_election.cc:290] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:14.415400  1127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:14.445111  1818 raft_consensus.cc:3060] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:12:14.408209  2085 raft_consensus.cc:2804] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:14.446091  2085 raft_consensus.cc:493] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:14.446514  2085 raft_consensus.cc:3060] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:14.436952  2121 leader_election.cc:290] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:14.450552  2055 raft_consensus.cc:2749] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:14.459810  1986 raft_consensus.cc:515] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:14.462409  1055 raft_consensus.cc:2468] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:14.445289  1127 raft_consensus.cc:3060] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:12:14.478567  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 111089us
I20260709 13:12:14.492311  1130 raft_consensus.cc:1275] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:14.496073  1127 raft_consensus.cc:2468] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 3.
I20260709 13:12:14.484120  1099 leader_election.cc:304] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:14.501681  2103 raft_consensus.cc:2804] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:14.502230  2103 raft_consensus.cc:697] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:14.503261  2103 consensus_queue.cc:237] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:14.518352  2085 raft_consensus.cc:515] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:14.522086  1818 raft_consensus.cc:1275] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:12:14.551126  1059 raft_consensus.cc:2468] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 4.
I20260709 13:12:14.486171  1986 leader_election.cc:290] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:14.551807  1816 raft_consensus.cc:1275] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:14.552891  1770 leader_election.cc:304] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:14.534447  2085 leader_election.cc:290] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:14.545743  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:14.551913  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:14.560506  2121 raft_consensus.cc:2804] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:12:14.559054  1812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:14.553578  2125 consensus_queue.cc:1048] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260709 13:12:14.573304  2121 raft_consensus.cc:697] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 4 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:12:14.574157  1053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:14.574306  2121 consensus_queue.cc:237] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:14.574972  1053 raft_consensus.cc:2468] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:14.591277  1025 leader_election.cc:400] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Vote denied by peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869) with higher term. Message: Invalid argument: T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 3. Current term is 4.
I20260709 13:12:14.591848  1025 leader_election.cc:403] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:12:14.592751   879 catalog_manager.cc:5697] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 1 to 3, leader changed from 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66) to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:14.605641  1025 leader_election.cc:400] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Vote denied by peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869) with higher term. Message: Invalid argument: T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 2. Current term is 3.
I20260709 13:12:14.606184  1025 leader_election.cc:403] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 13:12:14.609432  2125 raft_consensus.cc:2749] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869) with higher term. Message: Invalid argument: T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 2. Current term is 3.
I20260709 13:12:14.610918  2048 raft_consensus.cc:3060] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:12:14.614303  1210 raft_consensus.cc:1275] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:14.616009  1814 raft_consensus.cc:1275] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:12:14.624513  1201 raft_consensus.cc:1275] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:12:14.625274  1986 consensus_queue.cc:1048] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:14.630213  1053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:14.630911  1053 raft_consensus.cc:2468] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:14.634850  1203 raft_consensus.cc:2468] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 3.
I20260709 13:12:14.638224  1099 leader_election.cc:304] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:14.642329  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3fbed7be9014cf5984a19768b6269d4" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:14.643471  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3fbed7be9014cf5984a19768b6269d4" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:14.644110  1059 raft_consensus.cc:3060] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:14.645769  1210 raft_consensus.cc:1275] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:14.648880  2124 consensus_queue.cc:1048] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:14.630227  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:14.654496   882 catalog_manager.cc:5697] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:14.656167   882 catalog_manager.cc:5697] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:14.657305  2112 consensus_queue.cc:1048] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:14.668952  1099 leader_election.cc:304] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d
I20260709 13:12:14.671249  1099 leader_election.cc:304] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:14.672844  1059 raft_consensus.cc:2468] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 3.
I20260709 13:12:14.654961  2045 raft_consensus.cc:515] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:14.671288  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:14.682314  1210 raft_consensus.cc:2368] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 for earlier term 3. Current term is 4.
I20260709 13:12:14.669869  2046 raft_consensus.cc:2804] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:14.675917  2098 consensus_queue.cc:1048] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260709 13:12:14.693602  1770 leader_election.cc:400] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 862b41697c74421fbac82c10c245434d (127.23.140.68:34477) with higher term. Message: Invalid argument: T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 for earlier term 3. Current term is 4.
I20260709 13:12:14.693950  1770 leader_election.cc:403] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:12:14.640743  2097 consensus_queue.cc:1048] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260709 13:12:14.676924  2085 raft_consensus.cc:2804] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:14.670148  2042 consensus_queue.cc:1048] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
W20260709 13:12:14.846328  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 152863us
I20260709 13:12:14.698799  2132 raft_consensus.cc:2804] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:14.693686  2046 raft_consensus.cc:493] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:14.698522  1812 raft_consensus.cc:2368] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 2. Current term is 3.
I20260709 13:12:14.820257  2085 raft_consensus.cc:493] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:14.859472  2126 consensus_queue.cc:1048] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.170s
I20260709 13:12:14.869244  2085 raft_consensus.cc:3060] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:14.875799  2048 raft_consensus.cc:2749] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869) with higher term. Message: Invalid argument: T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 3. Current term is 4.
I20260709 13:12:14.852207  2046 raft_consensus.cc:3060] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:14.850154  2132 raft_consensus.cc:493] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:14.916167  1055 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:14.918468  1055 raft_consensus.cc:3060] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:14.916067  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:14.921178  1817 raft_consensus.cc:3060] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:14.925073  1054 raft_consensus.cc:2468] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 4.
I20260709 13:12:14.926467  2114 raft_consensus.cc:2749] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 862b41697c74421fbac82c10c245434d (127.23.140.68:34477) with higher term. Message: Invalid argument: T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 294145900dd3496cb4a1d722613e1558 for earlier term 3. Current term is 4.
I20260709 13:12:14.902350  2132 raft_consensus.cc:3060] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:12:14.929978  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 248317us
I20260709 13:12:14.931702  1099 leader_election.cc:304] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:14.930342  1027 leader_election.cc:400] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Vote denied by peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721) with higher term. Message: Invalid argument: T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 2. Current term is 3.
I20260709 13:12:14.937541  1027 leader_election.cc:403] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 13:12:14.944056  1127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:14.945762  2014 raft_consensus.cc:2804] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:14.912515  2045 leader_election.cc:290] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:14.962378  2014 raft_consensus.cc:697] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:14.963443  2014 consensus_queue.cc:237] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:14.965226  1055 raft_consensus.cc:2468] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 3.
I20260709 13:12:14.965842  2037 raft_consensus.cc:2749] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721) with higher term. Message: Invalid argument: T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 2. Current term is 3.
W20260709 13:12:14.979298  2138 log.cc:927] Time spent T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 13:12:14.991734  1991 consensus_queue.cc:1048] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:15.007182  1817 raft_consensus.cc:2468] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 4.
I20260709 13:12:15.033116  1099 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:15.021016  2085 raft_consensus.cc:515] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:15.027388  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:15.051447  1060 raft_consensus.cc:1275] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:15.051762  1133 raft_consensus.cc:1275] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:15.059898  1053 raft_consensus.cc:1275] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:15.068491  1770 leader_election.cc:304] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:15.046051  2102 raft_consensus.cc:2804] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:12:15.073167  1210 raft_consensus.cc:1275] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:15.074736  1810 raft_consensus.cc:1275] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:12:15.082499  1136 raft_consensus.cc:1240] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Rejecting Update request from peer 862b41697c74421fbac82c10c245434d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260709 13:12:15.109290  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 54692.4us
I20260709 13:12:15.080945  2102 raft_consensus.cc:697] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 4 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
W20260709 13:12:15.161002  1173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 81005.2us
W20260709 13:12:15.171939  1175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 85859us
I20260709 13:12:15.181067  2102 consensus_queue.cc:237] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:15.136109  2046 raft_consensus.cc:515] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:15.169425  2128 raft_consensus.cc:2804] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:15.181767  2094 consensus_queue.cc:1048] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:15.209291  2128 raft_consensus.cc:697] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 3 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:12:15.187757  2112 consensus_queue.cc:1048] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260709 13:12:15.229182  1127 raft_consensus.cc:2368] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 2. Current term is 3.
I20260709 13:12:15.238171  1057 raft_consensus.cc:1275] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:12:15.239156  2159 log.cc:927] Time spent T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 13:12:15.239188  1205 raft_consensus.cc:1275] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 13:12:15.241282  2157 log.cc:927] Time spent T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260709 13:12:15.230441  2085 leader_election.cc:290] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:15.245716  2046 leader_election.cc:290] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:15.246914  2128 consensus_queue.cc:237] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:15.234915  1130 raft_consensus.cc:3060] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:15.248558  1810 raft_consensus.cc:1275] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:15.264896  1054 raft_consensus.cc:1240] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Rejecting Update request from peer 862b41697c74421fbac82c10c245434d for earlier term 2. Current term is 3. Ops: []
I20260709 13:12:15.266434  1059 raft_consensus.cc:1275] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:12:15.272666  1810 raft_consensus.cc:1275] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:15.268266  2094 consensus_queue.cc:1048] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260709 13:12:15.277570  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558"
W20260709 13:12:15.281683  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 64048us
I20260709 13:12:15.286579  2156 raft_consensus.cc:493] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:15.287400  2156 raft_consensus.cc:515] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:15.236734  2095 consensus_queue.cc:1048] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260709 13:12:15.299467  2149 consensus_queue.cc:1048] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:15.283416  2014 consensus_queue.cc:1048] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260709 13:12:15.303906  2040 consensus_queue.cc:1059] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260709 13:12:15.235705  2132 raft_consensus.cc:515] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
W20260709 13:12:15.324752  1175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 132034us
I20260709 13:12:15.327455  1059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:15.330127  1136 raft_consensus.cc:1240] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Rejecting Update request from peer 862b41697c74421fbac82c10c245434d for earlier term 2. Current term is 3. Ops: []
I20260709 13:12:15.340760  1057 raft_consensus.cc:1275] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:15.325783  2114 consensus_queue.cc:1048] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260709 13:12:15.341198  1131 raft_consensus.cc:1240] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Rejecting Update request from peer 862b41697c74421fbac82c10c245434d for earlier term 2. Current term is 3. Ops: []
I20260709 13:12:15.354727  1132 raft_consensus.cc:1275] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:15.356329  2132 leader_election.cc:290] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:15.385972  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:15.442121  2169 consensus_queue.cc:1048] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.118s
I20260709 13:12:15.448792  2166 consensus_queue.cc:1048] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.163s
I20260709 13:12:15.352005  2091 consensus_queue.cc:1059] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260709 13:12:15.521591  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21783150bb0b4141be797e841c3f46fa" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:15.520355  2128 raft_consensus.cc:493] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:15.526824  2169 raft_consensus.cc:493] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:15.519496  2156 leader_election.cc:290] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:15.544549  1130 raft_consensus.cc:1275] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:12:15.547780  2055 consensus_queue.cc:1048] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:12:15.548707  1136 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84"
dest_uuid: "f9d473477839468bbdf000a21d5f9485"
mode: GRACEFUL
new_leader_uuid: "294145900dd3496cb4a1d722613e1558"
 from {username='slave'} at 127.0.0.1:46914
I20260709 13:12:15.549516  1136 raft_consensus.cc:606] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Received request to transfer leadership to TS 294145900dd3496cb4a1d722613e1558
I20260709 13:12:15.563970  1201 raft_consensus.cc:1275] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:15.566344  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:15.567199  1054 raft_consensus.cc:2393] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 3: Already voted for candidate 294145900dd3496cb4a1d722613e1558 in this term.
I20260709 13:12:15.527268  2128 raft_consensus.cc:515] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:15.527503  2169 raft_consensus.cc:515] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:15.667119  1055 raft_consensus.cc:1275] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 13:12:15.684922  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 167465us
W20260709 13:12:15.691506  2181 log.cc:927] Time spent T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 13:12:15.709187  2169 leader_election.cc:290] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:15.720607  2130 raft_consensus.cc:493] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:15.721314  1055 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:15.721268  2130 raft_consensus.cc:515] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:15.722007  1055 raft_consensus.cc:3060] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:15.726884  1099 leader_election.cc:304] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:15.727726  2014 consensus_queue.cc:1048] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260709 13:12:15.742005  2147 raft_consensus.cc:2749] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:12:15.742255  2140 raft_consensus.cc:493] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:15.743084  2140 raft_consensus.cc:515] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:15.745404  2148 raft_consensus.cc:493] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:15.746014  2140 leader_election.cc:290] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:15.745987  2148 raft_consensus.cc:515] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:15.747478  1059 raft_consensus.cc:1275] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:12:15.748479  2148 leader_election.cc:290] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:15.771727  1816 raft_consensus.cc:1275] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:12:15.734565  2102 consensus_queue.cc:1048] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260709 13:12:15.783354  2128 leader_election.cc:290] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:15.785197  2121 raft_consensus.cc:493] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:15.785805  2121 raft_consensus.cc:515] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:15.787837  2121 leader_election.cc:290] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:15.793277  2040 raft_consensus.cc:3055] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:15.793787  2040 raft_consensus.cc:740] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:15.794592  2040 consensus_queue.cc:260] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:15.795509  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
W20260709 13:12:15.793180  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 53432.3us
I20260709 13:12:15.799942  1131 raft_consensus.cc:1275] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:15.804356  1057 raft_consensus.cc:1275] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:15.815716  2170 consensus_queue.cc:1048] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:15.817098  2040 raft_consensus.cc:3060] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:12:15.823506  1175 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:12:15.824246  2130 leader_election.cc:290] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:15.825029  1205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:15.828825  1099 leader_election.cc:304] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:15.834021  1202 raft_consensus.cc:3055] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:15.834435  1202 raft_consensus.cc:740] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:15.835197  1202 consensus_queue.cc:260] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:15.835760  2033 raft_consensus.cc:493] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:15.836091  1202 raft_consensus.cc:3060] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:15.836272  2033 raft_consensus.cc:515] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:15.838442  2033 leader_election.cc:290] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:15.838984  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "161b9f09b3fb47febfabacf867781986" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:15.832886  2114 consensus_queue.cc:1048] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 13:12:15.844084  2191 raft_consensus.cc:2749] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:12:15.863319  2191 raft_consensus.cc:493] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:15.863858  2191 raft_consensus.cc:515] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:15.866437  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867d95d9806478ba50d324c565446f6" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
W20260709 13:12:15.867041  1027 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 135099us
I20260709 13:12:15.886444  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:15.894474  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:15.878379  2033 consensus_queue.cc:1048] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260709 13:12:15.903451  1202 raft_consensus.cc:1275] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 13:12:15.891549  2100 raft_consensus.cc:493] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:15.918702  2191 leader_election.cc:290] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:15.887578  2124 consensus_queue.cc:1048] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260709 13:12:15.920449  1202 pending_rounds.cc:85] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d: Aborting all ops after (but not including) 1
I20260709 13:12:15.923065  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf962b21fa14634af6e810ca29f1bc7" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:15.926136  1057 raft_consensus.cc:1275] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:12:15.922356  1202 pending_rounds.cc:107] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 13:12:15.949482  1202 raft_consensus.cc:2889] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:12:15.952579  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 88090.3us
I20260709 13:12:15.954097  1025 leader_election.cc:400] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 election: Vote denied by peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869) with higher term. Message: Invalid argument: T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 2. Current term is 3.
W20260709 13:12:15.949296  1027 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 76018.4us
I20260709 13:12:15.958242  1055 raft_consensus.cc:2468] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 3.
I20260709 13:12:15.965706  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf962b21fa14634af6e810ca29f1bc7" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:15.965921  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1971394c7d40bfa460d909b6f47438" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:15.963406  2166 consensus_queue.cc:1048] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260709 13:12:15.969807  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867d95d9806478ba50d324c565446f6" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:15.974772  2106 consensus_queue.cc:1048] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:15.977483  1127 raft_consensus.cc:1275] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:15.921900  2100 raft_consensus.cc:515] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:15.981254  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:15.991691  1129 raft_consensus.cc:1275] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:15.992496  2100 leader_election.cc:290] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:16.011793  1055 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21783150bb0b4141be797e841c3f46fa" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:12:16.013978  2173 log.cc:927] Time spent T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d: Append to log took a long time: real 0.512s	user 0.004s	sys 0.000s
I20260709 13:12:16.015884  1203 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43486 (request call id 485) took 1008 ms. Trace:
W20260709 13:12:16.017756  1173 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:12:16.022588  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 62949.4us
I20260709 13:12:16.027222   879 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37874 (request call id 131) took 1819 ms. Trace:
I20260709 13:12:16.031980  1099 leader_election.cc:304] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d
I20260709 13:12:16.029595  2180 consensus_queue.cc:1059] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260709 13:12:16.055337  2177 consensus_queue.cc:1048] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:16.069655  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:12:16.077294  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 52297us
I20260709 13:12:16.077468  2133 consensus_queue.cc:1048] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:12:16.078964  2205 consensus_queue.cc:1048] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:16.081386  2145 raft_consensus.cc:3055] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:16.081807  2145 raft_consensus.cc:740] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:16.082206  2103 raft_consensus.cc:493] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:16.082748  2103 raft_consensus.cc:515] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:16.016294  1203 rpcz_store.cc:276] 0709 13:12:15.007676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:12:15.008260 (+   584us) service_pool.cc:224] Handling call
0709 13:12:15.029773 (+ 21513us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:12:15.085284 (+ 55511us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:12:15.103429 (+ 18145us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:12:15.167277 (+ 63848us) log.cc:844] Serialized 26 byte log entry
0709 13:12:15.321981 (+154704us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:12:15.324006 (+  2025us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:12:15.328194 (+  4188us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:12:16.013396 (+685202us) spinlock_profiling.cc:243] Waited 4.38 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f297684d86e 00007f297684cbb6 00007f297684c7d8 00007f2976856221 00007f29768561d9 00007f2976856169
0709 13:12:16.015541 (+  2145us) raft_consensus.cc:1685] finished
0709 13:12:16.015586 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:12:16.015821 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18378,"raft.queue_time_us":78513,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":23656,"spinlock_wait_cycles":10083072,"thread_start_us":117825,"threads_started":1,"wal-append.queue_time_us":178440}
I20260709 13:12:16.085641  1053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "161b9f09b3fb47febfabacf867781986" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:16.086118  1056 raft_consensus.cc:1275] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:16.087198  1053 raft_consensus.cc:1240] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Rejecting Update request from peer 862b41697c74421fbac82c10c245434d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:12:16.082683  2145 consensus_queue.cc:260] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:16.095082  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:16.096230  1173 leader_election.cc:304] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:16.096675  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:16.097311  1136 raft_consensus.cc:2468] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 3.
I20260709 13:12:16.105106  1025 leader_election.cc:304] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
W20260709 13:12:16.108891  1175 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:12:16.109328  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:16.110285  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ca8b4123ce4ab3a2e36c3b422e8af5" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:16.111802  1127 raft_consensus.cc:1240] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Rejecting Update request from peer 862b41697c74421fbac82c10c245434d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:12:16.072988  2132 raft_consensus.cc:2804] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:16.065730  2040 consensus_queue.cc:1048] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260709 13:12:16.085031  2103 leader_election.cc:290] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:16.114012  1173 leader_election.cc:304] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
W20260709 13:12:16.101681  2111 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260709 13:12:16.104624  2094 raft_consensus.cc:2749] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:16.117434  2145 raft_consensus.cc:3060] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:16.113729  1058 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4c162b2e111b472b88cd037925284de7"
dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
mode: GRACEFUL
new_leader_uuid: "294145900dd3496cb4a1d722613e1558"
 from {username='slave'} at 127.0.0.1:44572
W20260709 13:12:16.187335  2193 log.cc:927] Time spent T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 13:12:16.199848  1058 raft_consensus.cc:606] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Received request to transfer leadership to TS 294145900dd3496cb4a1d722613e1558
I20260709 13:12:16.200347  1058 raft_consensus.cc:614] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:12:16.033214   879 rpcz_store.cc:276] 0709 13:12:14.207500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:12:14.207644 (+   144us) service_pool.cc:224] Handling call
0709 13:12:16.027128 (+1819484us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:12:15.045775 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:12:15.063923 (+ 18148us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:12:15.063958 (+    35us) write_op.cc:435] Acquired schema lock
0709 13:12:15.063976 (+    18us) tablet.cc:662] Decoding operations
0709 13:12:15.064332 (+   356us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:12:15.064382 (+    50us) write_op.cc:641] Partition lock acquired for write op
0709 13:12:15.064416 (+    34us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:12:15.064612 (+   196us) tablet.cc:701] Row locks acquired
0709 13:12:15.064628 (+    16us) write_op.cc:260] PREPARE: finished
0709 13:12:15.064783 (+   155us) write_op.cc:270] Start()
0709 13:12:15.064888 (+   105us) write_op.cc:276] Timestamp: P: 1783602735064744 usec, L: 0
0709 13:12:15.064908 (+    20us) op_driver.cc:348] REPLICATION: starting
0709 13:12:15.065604 (+   696us) log.cc:844] Serialized 540 byte log entry
0709 13:12:15.218577 (+152973us) op_driver.cc:464] REPLICATION: finished
0709 13:12:15.322816 (+104239us) write_op.cc:301] APPLY: starting
0709 13:12:15.326175 (+  3359us) tablet.cc:1367] starting BulkCheckPresence
0709 13:12:15.335856 (+  9681us) tablet.cc:1370] finished BulkCheckPresence
0709 13:12:15.354430 (+ 18574us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:12:15.502083 (+147653us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:12:15.518060 (+ 15977us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:12:15.520662 (+  2602us) write_op.cc:312] APPLY: finished
0709 13:12:15.524557 (+  3895us) log.cc:844] Serialized 22 byte log entry
0709 13:12:15.524732 (+   175us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:12:15.524820 (+    88us) write_op.cc:454] Released schema lock
0709 13:12:15.524860 (+    40us) write_op.cc:341] FINISH: Updating metrics
0709 13:12:15.966835 (+441975us) spinlock_profiling.cc:243] Waited 4.26 ms on lock 0x7b540093c980. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f297a61c161 00007f297a6f133d 00007f297a6ea3c2 00007f297a6e9b53 00007f297a6edd61 00007f297a6edd19 00007f297a6edca9 00007f297a6edc71 00007f297a6ece9d 000000000038f354 000000000038f199
0709 13:12:15.967069 (+   234us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:12:15.967097 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":61240,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":61180,"apply.run_cpu_time_us":5619,"apply.run_wall_time_us":652671,"num_ops":1,"prepare.queue_time_us":441563,"prepare.run_cpu_time_us":2745,"prepare.run_wall_time_us":26766,"replication_time_us":141757,"spinlock_wait_cycles":9790848,"thread_start_us":374855,"threads_started":2}]]}
I20260709 13:12:16.185750  2177 raft_consensus.cc:2804] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:12:16.203802  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 104210us
I20260709 13:12:16.204722  2132 raft_consensus.cc:697] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:16.206152  2177 raft_consensus.cc:493] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:12:16.223126  1101 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 51602us
I20260709 13:12:16.225766  1101 leader_election.cc:304] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:16.258889  1054 raft_consensus.cc:1240] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Rejecting Update request from peer 862b41697c74421fbac82c10c245434d for earlier term 2. Current term is 3. Ops: []
I20260709 13:12:16.223075  2177 raft_consensus.cc:3060] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:12:16.265416  2209 log.cc:927] Time spent T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 13:12:16.268712  2103 raft_consensus.cc:2749] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:12:16.269886  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 62149.9us
W20260709 13:12:16.340389  1027 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 237929us
I20260709 13:12:16.269785  2106 raft_consensus.cc:993] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485: : Instructing follower 294145900dd3496cb4a1d722613e1558 to start an election
I20260709 13:12:16.269663  2148 raft_consensus.cc:2749] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:16.328864  2040 raft_consensus.cc:493] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:12:16.395715  2213 log.cc:927] Time spent T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
I20260709 13:12:16.395999  2106 raft_consensus.cc:1081] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Signalling peer 294145900dd3496cb4a1d722613e1558 to start an election
I20260709 13:12:16.373734  2208 consensus_queue.cc:1048] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260709 13:12:16.414536  2132 consensus_queue.cc:237] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:16.427990  2177 raft_consensus.cc:515] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:16.431164  2177 leader_election.cc:290] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:16.387086  2040 raft_consensus.cc:515] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
W20260709 13:12:16.449961  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 343020us
I20260709 13:12:16.456290  1210 raft_consensus.cc:1240] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Rejecting Update request from peer 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:12:16.458874  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91c2241524a24f7e911295c67f13bd26" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
W20260709 13:12:16.466696  1027 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 80275.7us
I20260709 13:12:16.476953  2170 raft_consensus.cc:493] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:16.511552  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7526fd343f7d40528597242eb03386f8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:16.568143  1053 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6aa365e967a941129ed031939342b1d8"
dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
mode: GRACEFUL
new_leader_uuid: "294145900dd3496cb4a1d722613e1558"
 from {username='slave'} at 127.0.0.1:44572
I20260709 13:12:16.568886  1053 raft_consensus.cc:606] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 LEADER]: Received request to transfer leadership to TS 294145900dd3496cb4a1d722613e1558
W20260709 13:12:16.574453  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 79297.1us
I20260709 13:12:16.547668  2170 raft_consensus.cc:515] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:16.613304  1770 leader_election.cc:304] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:16.616882  2040 leader_election.cc:290] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:16.623245  2121 raft_consensus.cc:2749] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:16.637857  1025 leader_election.cc:304] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:16.624814  2186 raft_consensus.cc:493] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:16.641258  2186 raft_consensus.cc:515] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:16.643533  2186 leader_election.cc:290] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:16.644765  2170 leader_election.cc:290] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:16.647125  1025 leader_election.cc:304] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:16.650511  1062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:16.652040  2227 raft_consensus.cc:2749] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:12:16.667311  1810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:16.649381  2033 consensus_queue.cc:1059] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260709 13:12:16.686344  2196 raft_consensus.cc:3055] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:12:16.686647  1175 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:12:16.686720  2196 raft_consensus.cc:740] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:16.688429  2196 consensus_queue.cc:260] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
W20260709 13:12:16.695698  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 82568.3us
I20260709 13:12:16.693073  1053 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7526fd343f7d40528597242eb03386f8"
dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
mode: GRACEFUL
new_leader_uuid: "294145900dd3496cb4a1d722613e1558"
 from {username='slave'} at 127.0.0.1:44572
I20260709 13:12:16.700696  1027 leader_election.cc:304] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:16.704334  1770 leader_election.cc:304] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:16.696475  2095 raft_consensus.cc:493] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:16.709487  1027 leader_election.cc:304] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:16.707428  2037 raft_consensus.cc:2749] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:16.714048  2150 raft_consensus.cc:2749] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:16.715497  2095 raft_consensus.cc:515] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:16.718071  2095 leader_election.cc:290] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:16.703140  2196 raft_consensus.cc:3060] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:16.708882  1053 raft_consensus.cc:606] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Received request to transfer leadership to TS 294145900dd3496cb4a1d722613e1558
I20260709 13:12:16.723901  2231 raft_consensus.cc:2749] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:16.706874  2128 consensus_queue.cc:1048] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260709 13:12:16.731982  2204 raft_consensus.cc:2749] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:12:16.728049  1053 raft_consensus.cc:614] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:12:16.749287   882 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37862 (request call id 126) took 2521 ms. Trace:
I20260709 13:12:16.742830   883 catalog_manager.cc:5697] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 2 to 4, leader changed from 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 4 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:16.751196   883 catalog_manager.cc:5697] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:16.749833  2235 raft_consensus.cc:493] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:16.749753   882 rpcz_store.cc:276] 0709 13:12:14.222050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:12:14.222204 (+   154us) service_pool.cc:224] Handling call
0709 13:12:16.742892 (+2520688us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:12:15.066022 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:12:15.066514 (+   492us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:12:15.066538 (+    24us) write_op.cc:435] Acquired schema lock
0709 13:12:15.066559 (+    21us) tablet.cc:662] Decoding operations
0709 13:12:15.133551 (+ 66992us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:12:15.139696 (+  6145us) write_op.cc:641] Partition lock acquired for write op
0709 13:12:15.141261 (+  1565us) tablet.cc:685] Acquiring locks for 2 operations
0709 13:12:15.201525 (+ 60264us) tablet.cc:701] Row locks acquired
0709 13:12:15.202457 (+   932us) write_op.cc:260] PREPARE: finished
0709 13:12:15.210813 (+  8356us) write_op.cc:270] Start()
0709 13:12:15.219102 (+  8289us) write_op.cc:276] Timestamp: P: 1783602735208114 usec, L: 0
0709 13:12:15.224512 (+  5410us) op_driver.cc:348] REPLICATION: starting
0709 13:12:15.304059 (+ 79547us) log.cc:844] Serialized 928 byte log entry
0709 13:12:15.308159 (+  4100us) op_driver.cc:464] REPLICATION: finished
0709 13:12:15.437683 (+129524us) spinlock_profiling.cc:243] Waited 44.7 ms on lock 0x7b540093cbd4. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2976c1b571 00007f2976cade28 00007f297a6e8581 00007f297a6e7d58 00007f297a6ec9f1 00007f297a6ec9a9 00007f297a6ec939 00007f297a6ec901 00007f297a6ebb2d 000000000038f354 000000000038f199
0709 13:12:15.542750 (+105067us) write_op.cc:301] APPLY: starting
0709 13:12:15.542846 (+    96us) tablet.cc:1367] starting BulkCheckPresence
0709 13:12:15.542973 (+   127us) tablet.cc:1370] finished BulkCheckPresence
0709 13:12:15.542992 (+    19us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:12:15.547536 (+  4544us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:12:15.547717 (+   181us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0709 13:12:15.547773 (+    56us) write_op.cc:312] APPLY: finished
0709 13:12:15.547969 (+   196us) log.cc:844] Serialized 28 byte log entry
0709 13:12:15.548141 (+   172us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:12:15.548235 (+    94us) write_op.cc:454] Released schema lock
0709 13:12:15.548280 (+    45us) write_op.cc:341] FINISH: Updating metrics
0709 13:12:15.548637 (+   357us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:12:15.548663 (+    26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":476946,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":225534,"apply.run_cpu_time_us":2606,"apply.run_wall_time_us":9188,"num_ops":2,"prepare.queue_time_us":375861,"prepare.run_cpu_time_us":9238,"prepare.run_wall_time_us":372406,"replication_time_us":83505,"spinlock_wait_cycles":104791168,"thread_start_us":193205,"threads_started":1}]]}
I20260709 13:12:16.764971  1053 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d"
dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
mode: GRACEFUL
new_leader_uuid: "294145900dd3496cb4a1d722613e1558"
 from {username='slave'} at 127.0.0.1:44572
I20260709 13:12:16.766530  2234 raft_consensus.cc:493] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:16.767014  2234 raft_consensus.cc:515] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:16.765694  1053 raft_consensus.cc:606] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Received request to transfer leadership to TS 294145900dd3496cb4a1d722613e1558
I20260709 13:12:16.769979  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219d22690fa145adb5b04192c1fc16bd" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:16.772439  2140 raft_consensus.cc:993] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6: : Instructing follower 294145900dd3496cb4a1d722613e1558 to start an election
I20260709 13:12:16.772917  2140 raft_consensus.cc:1081] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Signalling peer 294145900dd3496cb4a1d722613e1558 to start an election
I20260709 13:12:16.765666  2235 raft_consensus.cc:515] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:16.775447  1057 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f267e3e7ffec446c9dfb1cfd18ace091"
dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
mode: GRACEFUL
new_leader_uuid: "862b41697c74421fbac82c10c245434d"
 from {username='slave'} at 127.0.0.1:44572
I20260709 13:12:16.775832  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:16.776008  1057 raft_consensus.cc:606] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Received request to transfer leadership to TS 862b41697c74421fbac82c10c245434d
I20260709 13:12:16.776466  1057 raft_consensus.cc:614] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:12:16.777709 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:12:16.777916  1812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d"
dest_uuid: "294145900dd3496cb4a1d722613e1558"
 from {username='slave'} at 127.0.0.1:55850
I20260709 13:12:16.778587  1812 raft_consensus.cc:493] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:16.778955  1812 raft_consensus.cc:3060] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:16.779454  2204 raft_consensus.cc:993] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6: : Instructing follower 294145900dd3496cb4a1d722613e1558 to start an election
I20260709 13:12:16.779572 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:12:16.779973  2204 raft_consensus.cc:1081] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 LEADER]: Signalling peer 294145900dd3496cb4a1d722613e1558 to start an election
I20260709 13:12:16.780617  2128 raft_consensus.cc:493] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:16.781230  2128 raft_consensus.cc:515] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:16.752719   883 catalog_manager.cc:5697] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:16.785662   883 catalog_manager.cc:5697] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 1 to 2, leader changed from 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 2 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:16.788260  1812 raft_consensus.cc:515] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:16.787154   883 catalog_manager.cc:5697] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 1 to 2, leader changed from 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 2 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:16.807693  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dde48ef51c4666a28b97181ec2ee21" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:16.818390  1099 leader_election.cc:304] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:16.789000  2235 leader_election.cc:290] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:16.823489  1810 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84"
dest_uuid: "294145900dd3496cb4a1d722613e1558"
 from {username='slave'} at 127.0.0.1:55840
I20260709 13:12:16.824111  1810 raft_consensus.cc:493] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:16.824502  1810 raft_consensus.cc:3060] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:16.789114  2128 leader_election.cc:290] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:16.829468  1816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:16.830547  1812 leader_election.cc:290] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:16.837272  2144 raft_consensus.cc:2749] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:16.839337  1057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:16.839964  1057 raft_consensus.cc:3055] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:16.840301  1057 raft_consensus.cc:740] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:16.841078  1057 consensus_queue.cc:260] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:16.842412  1057 raft_consensus.cc:3060] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:16.820004  1819 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6aa365e967a941129ed031939342b1d8"
dest_uuid: "294145900dd3496cb4a1d722613e1558"
 from {username='slave'} at 127.0.0.1:55850
I20260709 13:12:16.843955  1819 raft_consensus.cc:493] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:16.844370  1819 raft_consensus.cc:3060] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:12:16.845368  2171 raft_consensus.cc:493] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:16.845885  2171 raft_consensus.cc:515] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:16.848039  2171 leader_election.cc:290] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:16.848524  2171 raft_consensus.cc:493] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:16.804104  2234 leader_election.cc:290] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
W20260709 13:12:16.852252  1025 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:12:16.860140  2190 raft_consensus.cc:493] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:16.861552  2095 raft_consensus.cc:493] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:16.862097  2095 raft_consensus.cc:515] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:16.865919  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa73a73b72b2499fa5219cbf455099b4" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:16.873160  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:16.878938  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f70fc428ff6447f8dca2e27ed05683a" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:16.879415  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:16.904718  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dde48ef51c4666a28b97181ec2ee21" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:16.885502  1810 raft_consensus.cc:515] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:16.910040  1815 raft_consensus.cc:1240] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Rejecting Update request from peer f9d473477839468bbdf000a21d5f9485 for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:16.867851  2095 leader_election.cc:290] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:16.910858  1061 raft_consensus.cc:2410] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:12:16.860694  2190 raft_consensus.cc:515] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:16.913219  1810 leader_election.cc:290] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:16.848961  2171 raft_consensus.cc:515] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:16.883912  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:16.914991  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219d22690fa145adb5b04192c1fc16bd" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:16.920037  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:16.920662  1207 raft_consensus.cc:3060] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:16.883724  2069 raft_consensus.cc:493] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:16.899909  1819 raft_consensus.cc:515] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:16.925544  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:16.883668  1057 raft_consensus.cc:2468] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:16.911572  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6f69ac309b6488b81b409d785a2d465" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:16.933209  1132 raft_consensus.cc:3055] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:16.946710  1770 leader_election.cc:304] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:16.937644  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:16.953157  1770 leader_election.cc:304] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:16.953897  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12df324e933f422c9a8ff5d37eb7d4ef" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:16.954187  1770 leader_election.cc:304] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:16.934311  2069 raft_consensus.cc:515] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:16.958710  2231 raft_consensus.cc:2749] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:12:16.960511  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 43987.8us
I20260709 13:12:16.957044  2114 raft_consensus.cc:2749] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:16.963071  1811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6f69ac309b6488b81b409d785a2d465" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:16.947302  1132 raft_consensus.cc:740] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:16.957293  2235 raft_consensus.cc:2749] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:16.972839  1819 leader_election.cc:290] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 5 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:16.971545  2171 leader_election.cc:290] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:16.981460  1210 raft_consensus.cc:1275] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 13:12:16.982261  1210 pending_rounds.cc:85] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d: Aborting all ops after (but not including) 1
I20260709 13:12:16.982503  1210 pending_rounds.cc:107] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 13:12:16.982726  1210 raft_consensus.cc:2889] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:12:16.965654  1203 raft_consensus.cc:3060] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:16.992750  1810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb722a3043814db7a188e49c9861ceca" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:16.989777  1815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12df324e933f422c9a8ff5d37eb7d4ef" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:17.000792  1770 leader_election.cc:304] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:17.001303  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:17.001938  1210 raft_consensus.cc:3060] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:12:17.003536  1101 leader_election.cc:304] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:16.988456  2147 raft_consensus.cc:493] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:17.030800  2128 raft_consensus.cc:2804] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:17.037036  1813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "160ad44adbce4b7f97094b248fd05415" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:17.038398  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:17.039124  1058 raft_consensus.cc:3055] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:12:17.039481  1058 raft_consensus.cc:740] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:17.041378  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:17.042089  1132 consensus_queue.cc:260] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:17.043152  1132 raft_consensus.cc:3060] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:12:17.045264  2109 consensus_queue.cc:1175] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "862b41697c74421fbac82c10c245434d" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
W20260709 13:12:17.047295  2242 log.cc:927] Time spent T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 13:12:17.048858  1060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb722a3043814db7a188e49c9861ceca" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:17.049335  2069 leader_election.cc:290] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:17.058672  2144 consensus_queue.cc:1048] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:17.062124  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:17.062251  1136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:17.062587  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:17.062798  1128 raft_consensus.cc:2468] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 3.
I20260709 13:12:17.063071  1136 raft_consensus.cc:2468] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 3.
I20260709 13:12:17.040416  1058 consensus_queue.cc:260] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:17.054162  1207 raft_consensus.cc:2468] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:17.070078  1055 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "160ad44adbce4b7f97094b248fd05415" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:17.040439  2163 raft_consensus.cc:2749] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:17.087213  1210 raft_consensus.cc:2468] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 5.
I20260709 13:12:17.095082  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:17.087263  2128 raft_consensus.cc:697] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 2 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:12:17.098351  2128 consensus_queue.cc:237] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:17.054790  2147 raft_consensus.cc:515] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:17.105367  1131 raft_consensus.cc:2468] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:17.115092  1770 leader_election.cc:304] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 
W20260709 13:12:17.116194  1175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 64479.1us
I20260709 13:12:17.116987  2040 raft_consensus.cc:493] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:17.121227  2040 raft_consensus.cc:515] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:17.105044  2147 leader_election.cc:290] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:17.123271  2040 leader_election.cc:290] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
W20260709 13:12:17.123999  2247 log.cc:927] Time spent T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485: Append to log took a long time: real 0.093s	user 0.002s	sys 0.000s
I20260709 13:12:17.126623  2231 raft_consensus.cc:2804] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:12:17.127022  2234 raft_consensus.cc:493] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:17.127173  2231 raft_consensus.cc:697] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 5 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:12:17.127229  1173 leader_election.cc:304] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:17.127573  2234 raft_consensus.cc:515] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:17.128163  2231 consensus_queue.cc:237] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:17.128886  1173 leader_election.cc:304] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:17.129606  2177 raft_consensus.cc:2749] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:12:17.130553  1173 leader_election.cc:304] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:17.133183  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:17.140528  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e038f78f3e9540399aeda1f38cbea3fd" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
W20260709 13:12:17.141508  1101 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 101652us
I20260709 13:12:17.129789  2112 raft_consensus.cc:2804] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:12:17.129817  2234 leader_election.cc:290] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:17.141455  2111 raft_consensus.cc:2804] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:12:17.147217  2190 leader_election.cc:290] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:17.164558  1132 raft_consensus.cc:2468] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:17.166496  1203 raft_consensus.cc:2468] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:17.172513  1770 leader_election.cc:304] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:17.172566  1175 leader_election.cc:304] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:17.154357  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e038f78f3e9540399aeda1f38cbea3fd" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:17.171155  2111 raft_consensus.cc:493] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:17.178748  2111 raft_consensus.cc:3060] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:17.182440  1056 raft_consensus.cc:1275] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 13:12:17.183535  2244 log.cc:927] Time spent T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d: Append to log took a long time: real 0.181s	user 0.003s	sys 0.000s
I20260709 13:12:17.183933  1055 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:17.188742  2095 raft_consensus.cc:2749] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:17.152602  2112 raft_consensus.cc:493] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:17.191978  2112 raft_consensus.cc:3060] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:17.194469  2069 raft_consensus.cc:2804] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:17.195060  2069 raft_consensus.cc:697] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 2 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:12:17.195963  2069 consensus_queue.cc:237] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:17.204160  1175 leader_election.cc:304] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:17.205087  1770 leader_election.cc:304] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294145900dd3496cb4a1d722613e1558; no voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:17.208822  1770 leader_election.cc:304] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d
I20260709 13:12:17.219309  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:17.222759  1101 leader_election.cc:304] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
W20260709 13:12:17.226987  1028 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:12:17.230423  1101 leader_election.cc:304] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:17.210004  2180 raft_consensus.cc:2749] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:17.211881  2248 raft_consensus.cc:2749] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:17.217319  2234 raft_consensus.cc:2804] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:17.242588  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:17.232770  2144 raft_consensus.cc:2749] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:17.237375  2190 raft_consensus.cc:2749] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:17.251713  1207 raft_consensus.cc:1240] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Rejecting Update request from peer 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 4. Current term is 5. Ops: []
I20260709 13:12:17.240399  2208 raft_consensus.cc:493] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:17.244184  2234 raft_consensus.cc:493] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 862b41697c74421fbac82c10c245434d)
I20260709 13:12:17.272167  1099 leader_election.cc:304] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:17.263542  1058 raft_consensus.cc:3060] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:12:17.273593  2163 raft_consensus.cc:2749] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:17.281118  2111 raft_consensus.cc:515] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:17.266052  2234 raft_consensus.cc:3060] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:17.261220  2249 consensus_queue.cc:1048] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260709 13:12:17.330552  2111 leader_election.cc:290] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:17.336480  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:17.266211  2208 raft_consensus.cc:515] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:17.347785  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:17.348860  2208 leader_election.cc:290] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:17.353760  1819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:17.348788  1132 raft_consensus.cc:2468] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 3.
I20260709 13:12:17.357524  1173 leader_election.cc:304] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:17.358263  1101 leader_election.cc:304] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:17.359722  2109 raft_consensus.cc:2749] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:17.360437  2094 raft_consensus.cc:2804] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:12:17.360877  2094 raft_consensus.cc:493] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:17.361217  2094 raft_consensus.cc:3060] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:17.368468  1058 raft_consensus.cc:2468] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 5.
I20260709 13:12:17.388146  2180 raft_consensus.cc:493] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:17.388907  2180 raft_consensus.cc:515] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:17.391304  2180 leader_election.cc:290] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:17.393383  2112 raft_consensus.cc:515] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:17.396484  2112 leader_election.cc:290] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:17.402029  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:17.408630  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2312d768cbf4b3495d5b1574b0b6eb5" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "294145900dd3496cb4a1d722613e1558" is_pre_election: true
I20260709 13:12:17.419859  2234 raft_consensus.cc:515] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
W20260709 13:12:17.423434  2252 log.cc:927] Time spent T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558: Append to log took a long time: real 0.209s	user 0.008s	sys 0.000s
I20260709 13:12:17.429332  1814 raft_consensus.cc:1240] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Rejecting Update request from peer 862b41697c74421fbac82c10c245434d for earlier term 2. Current term is 3. Ops: []
W20260709 13:12:17.433763  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 164162us
I20260709 13:12:17.438174  1210 raft_consensus.cc:1240] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Rejecting Update request from peer 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 3. Current term is 4. Ops: []
I20260709 13:12:17.442346  1203 raft_consensus.cc:1240] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Rejecting Update request from peer 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 3. Current term is 4. Ops: []
I20260709 13:12:17.442605  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:17.443478  1132 raft_consensus.cc:2393] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 3: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:17.434928  2112 consensus_queue.cc:1059] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260709 13:12:17.444497  2112 raft_consensus.cc:3055] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:17.444952  2112 raft_consensus.cc:740] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:17.445874  2112 consensus_queue.cc:260] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:17.447307  2112 raft_consensus.cc:3060] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:17.449393  1057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:17.451627  1053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2312d768cbf4b3495d5b1574b0b6eb5" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:17.444711  1056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:17.463508  2258 consensus_queue.cc:1059] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:17.470412  2196 consensus_queue.cc:1059] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:12:17.471503  2196 raft_consensus.cc:3055] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:12:17.471859  2196 raft_consensus.cc:740] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:17.473462  1173 leader_election.cc:304] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:17.474506  2111 raft_consensus.cc:2749] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:17.474723  1173 leader_election.cc:304] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:17.477275  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:17.477865  2112 raft_consensus.cc:2749] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:12:17.478825  1173 leader_election.cc:304] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:17.485399  2111 raft_consensus.cc:2749] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:12:17.490031  2234 leader_election.cc:290] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:17.491278  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:17.491743  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:17.472690  2196 consensus_queue.cc:260] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:17.494304  2196 raft_consensus.cc:3060] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:17.495025  2042 raft_consensus.cc:3055] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:12:17.495673  2042 raft_consensus.cc:740] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:17.496524  2042 consensus_queue.cc:260] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:17.497784  2042 raft_consensus.cc:3060] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:17.489720  2094 raft_consensus.cc:515] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:17.515403  1203 raft_consensus.cc:2368] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 3. Current term is 4.
I20260709 13:12:17.516280  1201 raft_consensus.cc:2468] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 3.
I20260709 13:12:17.518160  1770 leader_election.cc:304] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:17.518947  1025 leader_election.cc:400] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 862b41697c74421fbac82c10c245434d (127.23.140.68:34477) with higher term. Message: Invalid argument: T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 3. Current term is 4.
I20260709 13:12:17.518316  2094 leader_election.cc:290] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:17.519306  2234 raft_consensus.cc:2804] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:17.519335  1025 leader_election.cc:403] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:12:17.519865  2234 raft_consensus.cc:697] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 3 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:12:17.521368  2234 consensus_queue.cc:237] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:17.521821  1128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:17.522545  1128 raft_consensus.cc:3060] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:17.522683  1053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81738ce3431443d9e332eb931ec3c0b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:17.523432  1053 raft_consensus.cc:3060] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:12:17.542464  1128 raft_consensus.cc:2468] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 4.
I20260709 13:12:17.544157  1173 leader_election.cc:304] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:17.544929  2094 raft_consensus.cc:2804] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:12:17.545524  2094 raft_consensus.cc:697] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 4 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:17.546383  2094 consensus_queue.cc:237] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:17.573989  1201 raft_consensus.cc:1275] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:17.576653  2231 consensus_queue.cc:1048] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:17.580451   883 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37882 (request call id 146) took 2398 ms. Trace:
I20260709 13:12:17.583863   882 catalog_manager.cc:5697] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 1 to 3, leader changed from f9d473477839468bbdf000a21d5f9485 (127.23.140.67) to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 3 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:17.586112   882 catalog_manager.cc:5697] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 2 to 4, leader changed from f9d473477839468bbdf000a21d5f9485 (127.23.140.67) to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 4 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:17.587882   882 catalog_manager.cc:5697] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } }
I20260709 13:12:17.593711  1058 raft_consensus.cc:1275] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:17.595213  2248 consensus_queue.cc:1048] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:17.580873   883 rpcz_store.cc:276] 0709 13:12:15.181855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:12:15.182005 (+   150us) service_pool.cc:224] Handling call
0709 13:12:17.580386 (+2398381us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:12:16.936216 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:12:16.999514 (+ 63298us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:12:17.001822 (+  2308us) write_op.cc:435] Acquired schema lock
0709 13:12:17.004348 (+  2526us) tablet.cc:662] Decoding operations
0709 13:12:17.031056 (+ 26708us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:12:17.031114 (+    58us) write_op.cc:641] Partition lock acquired for write op
0709 13:12:17.031146 (+    32us) tablet.cc:685] Acquiring locks for 5 operations
0709 13:12:17.031669 (+   523us) tablet.cc:701] Row locks acquired
0709 13:12:17.031689 (+    20us) write_op.cc:260] PREPARE: finished
0709 13:12:17.031831 (+   142us) write_op.cc:270] Start()
0709 13:12:17.031943 (+   112us) write_op.cc:276] Timestamp: P: 1783602737031787 usec, L: 0
0709 13:12:17.031965 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 13:12:17.032625 (+   660us) log.cc:844] Serialized 2093 byte log entry
0709 13:12:17.104881 (+ 72256us) spinlock_profiling.cc:243] Waited 34.2 ms on lock 0x7b4400022b4c. stack: 00007f2973b93343 00007f2973b93240 00007f29735221d4 00007f297d260304 00007f297d260299 00007f2973a4ddb1 00007f2973aebc85 00007f2973aec6a2 00007f297685e1db 00007f2976859aaf 00007f2973ba80ac 00007f297ca02ab9 00007f2973bc469b 00007f2973bc293f 00007f2973bc4c02 00007f297684c20b
0709 13:12:17.441170 (+336289us) op_driver.cc:464] REPLICATION: finished
0709 13:12:17.558910 (+117740us) write_op.cc:301] APPLY: starting
0709 13:12:17.559018 (+   108us) tablet.cc:1367] starting BulkCheckPresence
0709 13:12:17.559222 (+   204us) tablet.cc:1370] finished BulkCheckPresence
0709 13:12:17.559254 (+    32us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:12:17.560115 (+   861us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:12:17.560359 (+   244us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0709 13:12:17.560428 (+    69us) write_op.cc:312] APPLY: finished
0709 13:12:17.560713 (+   285us) log.cc:844] Serialized 46 byte log entry
0709 13:12:17.560880 (+   167us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:12:17.560977 (+    97us) write_op.cc:454] Released schema lock
0709 13:12:17.561016 (+    39us) write_op.cc:341] FINISH: Updating metrics
0709 13:12:17.561392 (+   376us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:12:17.561422 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":938933,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":117476,"apply.run_cpu_time_us":2479,"apply.run_wall_time_us":2488,"num_ops":5,"prepare.queue_time_us":74320,"prepare.run_cpu_time_us":38428,"prepare.run_wall_time_us":280589,"raft.queue_time_us":159144,"raft.run_cpu_time_us":6340,"raft.run_wall_time_us":72318,"replication_time_us":408410,"spinlock_wait_cycles":78673152,"thread_start_us":397819,"threads_started":4,"wal-append.queue_time_us":210487}]]}
I20260709 13:12:17.601624  1053 raft_consensus.cc:2468] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 4.
I20260709 13:12:17.602207  2208 raft_consensus.cc:2749] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 862b41697c74421fbac82c10c245434d (127.23.140.68:34477) with higher term. Message: Invalid argument: T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 for earlier term 3. Current term is 4.
I20260709 13:12:17.621835  1128 raft_consensus.cc:1275] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:17.623708  2248 consensus_queue.cc:1048] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:17.629119   880 catalog_manager.cc:5697] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:17.632450   880 catalog_manager.cc:5697] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } }
I20260709 13:12:17.634641   880 catalog_manager.cc:5697] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 2 to 3, leader changed from 862b41697c74421fbac82c10c245434d (127.23.140.68) to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } }
I20260709 13:12:17.635949   880 catalog_manager.cc:5697] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } }
I20260709 13:12:17.637537   880 catalog_manager.cc:5697] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 1 to 3, leader changed from f9d473477839468bbdf000a21d5f9485 (127.23.140.67) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 3 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } }
I20260709 13:12:17.664981  1058 raft_consensus.cc:1275] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:12:17.667079  2248 consensus_queue.cc:1048] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:12:17.665932   879 catalog_manager.cc:5697] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 2 to 3, leader changed from 862b41697c74421fbac82c10c245434d (127.23.140.68) to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:17.671147   879 catalog_manager.cc:5697] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } }
I20260709 13:12:17.697798  1201 raft_consensus.cc:1275] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:17.699522  2248 consensus_queue.cc:1048] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:17.721949  1203 raft_consensus.cc:1275] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:12:17.723822  2248 consensus_queue.cc:1048] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:12:17.728397   882 catalog_manager.cc:5697] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 1 to 2, leader changed from 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66) to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } }
I20260709 13:12:17.730744   882 catalog_manager.cc:5697] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:17.768242   880 catalog_manager.cc:5697] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 4 to 5, leader changed from 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66) to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 5 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:17.769798   880 catalog_manager.cc:5697] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 2 to 3, leader changed from 862b41697c74421fbac82c10c245434d (127.23.140.68) to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 3 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:17.772454   880 catalog_manager.cc:5697] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 1 to 2, leader changed from f9d473477839468bbdf000a21d5f9485 (127.23.140.67) to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:17.781284 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:12:17.871656 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:12:17.903709  1201 raft_consensus.cc:1275] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:17.907022  2128 consensus_queue.cc:1048] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:17.923867  1131 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0867d95d9806478ba50d324c565446f6"
dest_uuid: "f9d473477839468bbdf000a21d5f9485"
mode: GRACEFUL
new_leader_uuid: "294145900dd3496cb4a1d722613e1558"
 from {username='slave'} at 127.0.0.1:46914
I20260709 13:12:17.924516  1131 raft_consensus.cc:606] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Received request to transfer leadership to TS 294145900dd3496cb4a1d722613e1558
I20260709 13:12:17.927392  1130 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "09039e63d8db4e06b365f43533071272"
dest_uuid: "f9d473477839468bbdf000a21d5f9485"
mode: GRACEFUL
new_leader_uuid: "294145900dd3496cb4a1d722613e1558"
 from {username='slave'} at 127.0.0.1:46914
I20260709 13:12:17.927906  1130 raft_consensus.cc:606] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 4 LEADER]: Received request to transfer leadership to TS 294145900dd3496cb4a1d722613e1558
I20260709 13:12:17.931471  1130 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032"
dest_uuid: "f9d473477839468bbdf000a21d5f9485"
mode: GRACEFUL
new_leader_uuid: "294145900dd3496cb4a1d722613e1558"
 from {username='slave'} at 127.0.0.1:46914
I20260709 13:12:17.931965  1130 raft_consensus.cc:606] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Received request to transfer leadership to TS 294145900dd3496cb4a1d722613e1558
I20260709 13:12:17.935113  1130 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4c162b2e111b472b88cd037925284de7"
dest_uuid: "f9d473477839468bbdf000a21d5f9485"
mode: GRACEFUL
new_leader_uuid: "294145900dd3496cb4a1d722613e1558"
 from {username='slave'} at 127.0.0.1:46914
I20260709 13:12:17.935690  1130 raft_consensus.cc:606] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Received request to transfer leadership to TS 294145900dd3496cb4a1d722613e1558
I20260709 13:12:17.940037  1130 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fc8c8635d8e54625af9c95ecbcd5653b"
dest_uuid: "f9d473477839468bbdf000a21d5f9485"
mode: GRACEFUL
new_leader_uuid: "294145900dd3496cb4a1d722613e1558"
 from {username='slave'} at 127.0.0.1:46914
I20260709 13:12:17.954048  1130 raft_consensus.cc:606] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Received request to transfer leadership to TS 294145900dd3496cb4a1d722613e1558
I20260709 13:12:17.955654 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:12:17.958560 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:12:17.961017  1130 raft_consensus.cc:1275] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:12:17.975468  2111 consensus_queue.cc:1048] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:17.995404  1058 raft_consensus.cc:1275] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:12:17.999913  2111 consensus_queue.cc:1048] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260709 13:12:18.026176  2085 raft_consensus.cc:993] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485: : Instructing follower 294145900dd3496cb4a1d722613e1558 to start an election
I20260709 13:12:18.026913  2109 raft_consensus.cc:1081] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Signalling peer 294145900dd3496cb4a1d722613e1558 to start an election
I20260709 13:12:18.051635  1814 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fc8c8635d8e54625af9c95ecbcd5653b"
dest_uuid: "294145900dd3496cb4a1d722613e1558"
 from {username='slave'} at 127.0.0.1:55840
I20260709 13:12:18.052289  1814 raft_consensus.cc:493] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:18.052682  1814 raft_consensus.cc:3060] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:18.099483  1130 raft_consensus.cc:3060] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:18.095058  1814 raft_consensus.cc:515] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:18.103449  1814 leader_election.cc:290] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:18.104996  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8c8635d8e54625af9c95ecbcd5653b" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:18.106209  1131 raft_consensus.cc:3055] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:18.106575  1131 raft_consensus.cc:740] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:18.107306  1131 consensus_queue.cc:260] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:18.108533  1131 raft_consensus.cc:3060] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:18.110949  1130 raft_consensus.cc:1275] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:18.112483  2128 consensus_queue.cc:1048] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:18.122987  1131 raft_consensus.cc:2468] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:18.126987  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8c8635d8e54625af9c95ecbcd5653b" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:18.127777  1207 raft_consensus.cc:3060] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:18.127120  1770 leader_election.cc:304] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:18.128669  2128 raft_consensus.cc:2804] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:18.130273  2128 raft_consensus.cc:697] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 2 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:12:18.131171  2128 consensus_queue.cc:237] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:18.161336  2226 raft_consensus.cc:993] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485: : Instructing follower 294145900dd3496cb4a1d722613e1558 to start an election
I20260709 13:12:18.162034  2085 raft_consensus.cc:1081] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Signalling peer 294145900dd3496cb4a1d722613e1558 to start an election
I20260709 13:12:18.154630  1207 raft_consensus.cc:2468] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:18.175462  1814 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0867d95d9806478ba50d324c565446f6"
dest_uuid: "294145900dd3496cb4a1d722613e1558"
 from {username='slave'} at 127.0.0.1:55840
I20260709 13:12:18.176263  1814 raft_consensus.cc:493] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:18.176729  1814 raft_consensus.cc:3060] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:18.195070  2085 raft_consensus.cc:993] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485: : Instructing follower 294145900dd3496cb4a1d722613e1558 to start an election
I20260709 13:12:18.195670  2109 raft_consensus.cc:1081] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Signalling peer 294145900dd3496cb4a1d722613e1558 to start an election
I20260709 13:12:18.198096  1812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032"
dest_uuid: "294145900dd3496cb4a1d722613e1558"
 from {username='slave'} at 127.0.0.1:55840
I20260709 13:12:18.198876  1812 raft_consensus.cc:493] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:18.199329  1812 raft_consensus.cc:3060] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:18.236378  1812 raft_consensus.cc:515] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:18.261808  1814 raft_consensus.cc:515] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:18.264600  1057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867d95d9806478ba50d324c565446f6" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:18.265224  1057 raft_consensus.cc:3060] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:18.273093  1057 raft_consensus.cc:2468] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:18.287060  1053 raft_consensus.cc:1240] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Rejecting Update request from peer f9d473477839468bbdf000a21d5f9485 for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:18.254652   880 catalog_manager.cc:5697] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 1 to 2, leader changed from f9d473477839468bbdf000a21d5f9485 (127.23.140.67) to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:18.289835  2149 consensus_queue.cc:1059] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:18.289253  1770 leader_election.cc:304] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:18.290869  2085 raft_consensus.cc:3055] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:18.290973  2231 raft_consensus.cc:2804] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:18.291283  2085 raft_consensus.cc:740] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:18.291515  2231 raft_consensus.cc:697] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 2 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:12:18.291690  2149 raft_consensus.cc:993] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485: : Instructing follower 294145900dd3496cb4a1d722613e1558 to start an election
I20260709 13:12:18.292230  2149 raft_consensus.cc:1081] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 4 LEADER]: Signalling peer 294145900dd3496cb4a1d722613e1558 to start an election
I20260709 13:12:18.293924  1817 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "09039e63d8db4e06b365f43533071272"
dest_uuid: "294145900dd3496cb4a1d722613e1558"
 from {username='slave'} at 127.0.0.1:55840
I20260709 13:12:18.294539  1817 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:18.294899  1817 raft_consensus.cc:3060] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:12:18.292166  2085 consensus_queue.cc:260] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:18.301206  2085 raft_consensus.cc:3060] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:18.293488  2231 consensus_queue.cc:237] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:18.306540  1053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:18.307178  1053 raft_consensus.cc:3060] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:18.310202  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867d95d9806478ba50d324c565446f6" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:18.311185  1814 leader_election.cc:290] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:18.307185  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:18.313905  1132 raft_consensus.cc:3055] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:18.314308  1132 raft_consensus.cc:740] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:18.314225  1817 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:18.315346  1132 consensus_queue.cc:260] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:18.289913  1812 leader_election.cc:290] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:18.324852  1132 raft_consensus.cc:3060] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:18.318182  1053 raft_consensus.cc:2468] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:18.323788  1130 raft_consensus.cc:2468] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:18.326640  1770 leader_election.cc:304] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:18.327425  2231 raft_consensus.cc:2804] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:18.329352  2231 raft_consensus.cc:697] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 2 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:12:18.330247  2231 consensus_queue.cc:237] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:18.334688  1817 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 5 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:18.332870  1132 raft_consensus.cc:2468] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 2.
I20260709 13:12:18.340561  1053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:18.341187  1053 raft_consensus.cc:3060] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:12:18.341957  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:18.342792  1132 raft_consensus.cc:3055] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:12:18.343113  1132 raft_consensus.cc:740] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:18.346252  1132 consensus_queue.cc:260] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:18.347533  1132 raft_consensus.cc:3060] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:12:18.348933  1053 raft_consensus.cc:2468] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 5.
I20260709 13:12:18.350234  1770 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:18.350991  2128 raft_consensus.cc:2804] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:12:18.351459  2128 raft_consensus.cc:697] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 5 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:12:18.352298  2128 consensus_queue.cc:237] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:18.356420  1132 raft_consensus.cc:2468] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 5.
I20260709 13:12:18.363652  2149 raft_consensus.cc:493] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:18.364275  2149 raft_consensus.cc:515] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:18.366446  2149 leader_election.cc:290] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:18.367663  1057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:18.368392  1057 raft_consensus.cc:2468] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 4.
I20260709 13:12:18.370018  1099 leader_election.cc:304] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:18.370963  2226 raft_consensus.cc:2804] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:12:18.371357  2226 raft_consensus.cc:493] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:18.371707  2226 raft_consensus.cc:3060] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:18.373167  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:18.373986  1201 raft_consensus.cc:2393] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 4: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
I20260709 13:12:18.398926  2226 raft_consensus.cc:515] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:18.412878  1053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:18.424546  2226 leader_election.cc:290] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:18.428599  1053 raft_consensus.cc:2468] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 4.
I20260709 13:12:18.429230  2283 raft_consensus.cc:993] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485: : Instructing follower 294145900dd3496cb4a1d722613e1558 to start an election
I20260709 13:12:18.429824  1099 leader_election.cc:304] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:18.430480  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:18.431882  2226 raft_consensus.cc:2804] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:12:18.432220  1201 raft_consensus.cc:2393] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 4: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
I20260709 13:12:18.432436  2226 raft_consensus.cc:697] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 4 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:18.431404  2283 raft_consensus.cc:1081] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Signalling peer 294145900dd3496cb4a1d722613e1558 to start an election
I20260709 13:12:18.435199  2226 consensus_queue.cc:237] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:18.445776  1814 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4c162b2e111b472b88cd037925284de7"
dest_uuid: "294145900dd3496cb4a1d722613e1558"
 from {username='slave'} at 127.0.0.1:55840
I20260709 13:12:18.446501  1814 raft_consensus.cc:493] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:18.446960  1814 raft_consensus.cc:3060] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:18.471362  1814 raft_consensus.cc:515] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:18.475155  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:18.475741  1130 raft_consensus.cc:3055] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:12:18.476043  1130 raft_consensus.cc:740] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:18.476840  1130 consensus_queue.cc:260] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:18.478632  1130 raft_consensus.cc:3060] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:18.494083  1053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "294145900dd3496cb4a1d722613e1558" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:18.494863  1053 raft_consensus.cc:3060] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:18.477545  1814 leader_election.cc:290] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 4 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:18.498837   883 catalog_manager.cc:5697] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 3 to 4, leader changed from 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66) to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 4 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:18.514377  1130 raft_consensus.cc:2468] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 4.
I20260709 13:12:18.515637  1770 leader_election.cc:304] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:18.516393  2069 raft_consensus.cc:2804] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:12:18.516880  2069 raft_consensus.cc:697] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 4 LEADER]: Becoming Leader. State: Replica: 294145900dd3496cb4a1d722613e1558, State: Running, Role: LEADER
I20260709 13:12:18.517781  2069 consensus_queue.cc:237] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:18.545804  1053 raft_consensus.cc:2468] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294145900dd3496cb4a1d722613e1558 in term 4.
I20260709 13:12:18.571116   880 catalog_manager.cc:5697] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 1 to 2, leader changed from f9d473477839468bbdf000a21d5f9485 (127.23.140.67) to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:18.573062   880 catalog_manager.cc:5697] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 1 to 2, leader changed from f9d473477839468bbdf000a21d5f9485 (127.23.140.67) to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:18.577693   880 catalog_manager.cc:5697] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 4 to 5, leader changed from f9d473477839468bbdf000a21d5f9485 (127.23.140.67) to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 5 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:18.579825   880 catalog_manager.cc:5697] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 reported cstate change: term changed from 3 to 4, leader changed from f9d473477839468bbdf000a21d5f9485 (127.23.140.67) to 294145900dd3496cb4a1d722613e1558 (127.23.140.65). New cstate: current_term: 4 leader_uuid: "294145900dd3496cb4a1d722613e1558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:18.637962  2208 raft_consensus.cc:493] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:18.638597  2208 raft_consensus.cc:515] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:18.641002  2208 leader_election.cc:290] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:18.644146  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:18.645431  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:18.646291  1201 raft_consensus.cc:2468] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 4.
I20260709 13:12:18.648196  1025 leader_election.cc:304] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:18.657924  2286 raft_consensus.cc:2804] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:12:18.658394  2286 raft_consensus.cc:493] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:18.658737  2286 raft_consensus.cc:3060] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:12:18.667019  2286 raft_consensus.cc:515] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:18.670229  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:18.676949  2286 leader_election.cc:290] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:18.678632  1132 raft_consensus.cc:1275] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:18.682484  2231 consensus_queue.cc:1048] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:18.709690  1053 raft_consensus.cc:1275] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:18.711776  2069 consensus_queue.cc:1048] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:18.718590  1203 raft_consensus.cc:1275] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:18.704582  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:18.745671  1201 raft_consensus.cc:3060] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:12:18.760360  1057 raft_consensus.cc:1275] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:18.743643  2231 consensus_queue.cc:1048] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 13:12:18.767485  2231 consensus_queue.cc:1048] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:18.805460  1130 raft_consensus.cc:1275] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:18.811028  2234 consensus_queue.cc:1048] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:18.817284  1131 raft_consensus.cc:1275] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:12:18.818830  2128 consensus_queue.cc:1048] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:12:18.869081  1058 raft_consensus.cc:1275] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:12:18.873416  2286 raft_consensus.cc:493] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:18.856637  2283 raft_consensus.cc:493] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:18.874022  2286 raft_consensus.cc:515] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:18.874594  2283 raft_consensus.cc:515] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:18.883483  1201 raft_consensus.cc:2468] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 5.
I20260709 13:12:18.886003  2283 leader_election.cc:290] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:18.896855  2292 consensus_queue.cc:1048] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:12:18.905690  2286 leader_election.cc:290] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
W20260709 13:12:18.918690  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 64232.9us
I20260709 13:12:18.922418  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:18.923240  1058 raft_consensus.cc:1275] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:12:18.924193  1207 raft_consensus.cc:2393] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 4: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
W20260709 13:12:18.925853  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 50841.7us
I20260709 13:12:18.929998  1207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:18.930894  1207 raft_consensus.cc:2468] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 4.
I20260709 13:12:18.929800  1025 leader_election.cc:304] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: f9d473477839468bbdf000a21d5f9485
I20260709 13:12:18.940065  2303 raft_consensus.cc:2804] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:12:18.940621  2303 raft_consensus.cc:697] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
W20260709 13:12:18.941614  1173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 53580.8us
I20260709 13:12:18.941484  2303 consensus_queue.cc:237] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:18.943048  1025 leader_election.cc:304] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:18.945847  2208 raft_consensus.cc:2804] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:12:18.946414  2208 raft_consensus.cc:493] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:18.946856  2208 raft_consensus.cc:3060] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:12:18.960265 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:12:18.963058  2208 raft_consensus.cc:515] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:18.966747  2208 leader_election.cc:290] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:19.006321  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:19.007184  1132 raft_consensus.cc:2468] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 3.
I20260709 13:12:18.957459  1054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:19.010907  1054 raft_consensus.cc:2368] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 for earlier term 4. Current term is 5.
I20260709 13:12:19.015604  2283 consensus_queue.cc:1048] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:19.017953  1099 leader_election.cc:400] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097) with higher term. Message: Invalid argument: T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 for earlier term 4. Current term is 5.
I20260709 13:12:19.018365  1099 leader_election.cc:403] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:12:19.019383  2315 raft_consensus.cc:3060] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Advancing to term 5
I20260709 13:12:19.031271  1130 raft_consensus.cc:1275] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:19.081398  1201 raft_consensus.cc:1275] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:12:19.088023  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 142918us
W20260709 13:12:19.120051  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 129979us
I20260709 13:12:19.123703  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:19.190742  2284 consensus_queue.cc:1048] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:12:19.205147  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 48794us
I20260709 13:12:19.128988  2231 consensus_queue.cc:1048] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20260709 13:12:19.167567  1201 raft_consensus.cc:3060] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:12:19.222378  1061 raft_consensus.cc:1275] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:12:19.222922  1173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 111301us
W20260709 13:12:19.264204  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 108180us
I20260709 13:12:19.275672  1132 raft_consensus.cc:1275] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Refusing update from remote peer 294145900dd3496cb4a1d722613e1558: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:12:19.224649  2315 raft_consensus.cc:2749] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097) with higher term. Message: Invalid argument: T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 for earlier term 4. Current term is 5.
W20260709 13:12:19.282581  2299 log.cc:927] Time spent T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:12:19.329452 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:12:19.344094  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 77497.9us
I20260709 13:12:19.326588  2326 raft_consensus.cc:493] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:19.385143  2326 raft_consensus.cc:515] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:19.401604  2326 leader_election.cc:290] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:19.405408  2330 consensus_queue.cc:1048] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:12:19.416889  1101 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 114602us
I20260709 13:12:19.418497  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
W20260709 13:12:19.442330  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 54543.7us
I20260709 13:12:19.469874  1201 raft_consensus.cc:2468] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 5.
I20260709 13:12:19.477449  1131 raft_consensus.cc:2468] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 5.
I20260709 13:12:19.485526  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:12:19.487465  1101 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 50188us
W20260709 13:12:19.489606  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 45073.1us
I20260709 13:12:19.497380  1025 leader_election.cc:304] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:19.498266  2334 raft_consensus.cc:2804] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:12:19.498800  2334 raft_consensus.cc:697] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:19.501405  2334 consensus_queue.cc:237] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:19.503543   879 catalog_manager.cc:5697] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:19.506898  1131 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9eec3e72036941a5b16c3a0157986a93"
dest_uuid: "f9d473477839468bbdf000a21d5f9485"
mode: GRACEFUL
new_leader_uuid: "862b41697c74421fbac82c10c245434d"
 from {username='slave'} at 127.0.0.1:46914
I20260709 13:12:19.507582  1131 raft_consensus.cc:606] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Received request to transfer leadership to TS 862b41697c74421fbac82c10c245434d
I20260709 13:12:19.514159  1128 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94"
dest_uuid: "f9d473477839468bbdf000a21d5f9485"
mode: GRACEFUL
new_leader_uuid: "862b41697c74421fbac82c10c245434d"
 from {username='slave'} at 127.0.0.1:46914
I20260709 13:12:19.514848  1128 raft_consensus.cc:606] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Received request to transfer leadership to TS 862b41697c74421fbac82c10c245434d
I20260709 13:12:19.532284  1210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:19.534317  2234 consensus_queue.cc:1048] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:12:19.535848 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:12:19.537386  1210 raft_consensus.cc:2368] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 for earlier term 4. Current term is 5.
I20260709 13:12:19.538040 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:12:19.621729  1202 raft_consensus.cc:1275] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:12:19.627779  2335 consensus_queue.cc:1048] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:12:19.640599  1099 leader_election.cc:400] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 862b41697c74421fbac82c10c245434d (127.23.140.68:34477) with higher term. Message: Invalid argument: T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 for earlier term 4. Current term is 5.
I20260709 13:12:19.640992  1099 leader_election.cc:403] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:12:19.642563  2337 raft_consensus.cc:3060] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Advancing to term 5
I20260709 13:12:19.646281  2226 raft_consensus.cc:993] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485: : Instructing follower 862b41697c74421fbac82c10c245434d to start an election
I20260709 13:12:19.604840  2305 raft_consensus.cc:493] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:19.646512  2085 raft_consensus.cc:993] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485: : Instructing follower 862b41697c74421fbac82c10c245434d to start an election
I20260709 13:12:19.647142  2226 raft_consensus.cc:1081] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Signalling peer 862b41697c74421fbac82c10c245434d to start an election
I20260709 13:12:19.674896  2085 raft_consensus.cc:1081] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Signalling peer 862b41697c74421fbac82c10c245434d to start an election
I20260709 13:12:19.653324  2305 raft_consensus.cc:515] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
W20260709 13:12:19.766603  1173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 77586.3us
W20260709 13:12:19.767500  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 77142us
W20260709 13:12:19.769946  1027 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 79386.7us
W20260709 13:12:19.756991  2347 log.cc:927] Time spent T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485: Append to log took a long time: real 0.123s	user 0.000s	sys 0.002s
I20260709 13:12:19.783222  1206 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94"
dest_uuid: "862b41697c74421fbac82c10c245434d"
 from {username='slave'} at 127.0.0.1:43486
I20260709 13:12:19.783960  1206 raft_consensus.cc:493] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:19.784360  1206 raft_consensus.cc:3060] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:19.787889  2305 leader_election.cc:290] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:19.792946  1201 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9eec3e72036941a5b16c3a0157986a93"
dest_uuid: "862b41697c74421fbac82c10c245434d"
 from {username='slave'} at 127.0.0.1:43486
I20260709 13:12:19.795351  1201 raft_consensus.cc:493] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:19.795749  1201 raft_consensus.cc:3060] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:19.796892  2337 raft_consensus.cc:2749] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 862b41697c74421fbac82c10c245434d (127.23.140.68:34477) with higher term. Message: Invalid argument: T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 for earlier term 4. Current term is 5.
I20260709 13:12:19.797936  1130 raft_consensus.cc:1275] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:12:19.815254  1057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:19.821003  2338 consensus_queue.cc:1048] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:12:19.827359  1206 raft_consensus.cc:515] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:19.865741  1205 raft_consensus.cc:1240] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Rejecting Update request from peer f9d473477839468bbdf000a21d5f9485 for earlier term 3. Current term is 4. Ops: []
I20260709 13:12:19.845850  1201 raft_consensus.cc:515] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:19.869313  1206 leader_election.cc:290] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:19.871510  1201 leader_election.cc:290] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:19.879283  1817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:19.879304  1811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eec3e72036941a5b16c3a0157986a93" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "294145900dd3496cb4a1d722613e1558"
I20260709 13:12:19.880767  1811 raft_consensus.cc:3060] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:19.883522  2324 consensus_queue.cc:1059] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:19.884665  2337 raft_consensus.cc:3055] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:12:19.885182  2337 raft_consensus.cc:740] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:19.886015  1817 raft_consensus.cc:3060] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:19.890056  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c216d860ee834d4b8f77c8c39f0dba94" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:19.886108  2337 consensus_queue.cc:260] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:19.893953  2337 raft_consensus.cc:3060] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:19.887380  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbe4fae89eb429b989a7e1d87ab7dcf" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:19.888376  1129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eec3e72036941a5b16c3a0157986a93" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:19.896390  1129 raft_consensus.cc:3055] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:19.896777  1129 raft_consensus.cc:740] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:19.897699  1129 consensus_queue.cc:260] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:19.899187  1129 raft_consensus.cc:3060] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:19.907367  1173 leader_election.cc:304] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:19.908618  2319 raft_consensus.cc:2749] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:12:19.955336  1811 raft_consensus.cc:2468] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:19.950999  1817 raft_consensus.cc:2468] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 4.
I20260709 13:12:19.941322   879 catalog_manager.cc:5697] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:19.963474  1175 leader_election.cc:304] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 
I20260709 13:12:19.968777  2319 raft_consensus.cc:2804] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:19.969467  2319 raft_consensus.cc:697] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:19.970329  2319 consensus_queue.cc:237] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:19.968125  1175 leader_election.cc:304] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d; no voters: 
I20260709 13:12:19.972395  2321 raft_consensus.cc:2804] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:12:19.972882  2321 raft_consensus.cc:697] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 4 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:19.974110  2321 consensus_queue.cc:237] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:19.993422  1206 raft_consensus.cc:1275] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:12:19.961957  1129 raft_consensus.cc:2468] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:19.995522  1131 raft_consensus.cc:1275] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:12:19.996517  1132 raft_consensus.cc:2468] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 4.
I20260709 13:12:19.997731  2334 consensus_queue.cc:1048] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.008s
I20260709 13:12:20.009330  2338 consensus_queue.cc:1048] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:20.013458   879 catalog_manager.cc:5697] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 1 to 2, leader changed from f9d473477839468bbdf000a21d5f9485 (127.23.140.67) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 2 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:20.093297   879 catalog_manager.cc:5697] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 3 to 4, leader changed from f9d473477839468bbdf000a21d5f9485 (127.23.140.67) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 4 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:20.539124 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:12:20.545913  1131 raft_consensus.cc:1275] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:20.548062  1817 raft_consensus.cc:1275] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:20.547993  2305 consensus_queue.cc:1048] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:20.550843  2319 consensus_queue.cc:1048] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:20.601486  1811 raft_consensus.cc:1275] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:12:20.605475  2305 consensus_queue.cc:1048] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:12:20.613783  1130 raft_consensus.cc:1275] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:12:20.721879  2319 consensus_queue.cc:1048] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 13:12:20.755017  1173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 107842us
I20260709 13:12:20.842499 24113 tablet_server.cc:179] TabletServer@127.23.140.65:43721 shutting down...
W20260709 13:12:20.854559  1027 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 45 similar messages]
W20260709 13:12:20.956583  1173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 88599.6us
W20260709 13:12:20.966794  1027 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 112256us
W20260709 13:12:20.975159  1101 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 113328us
W20260709 13:12:20.976804  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 86109.6us
I20260709 13:12:20.981864 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:12:20.985437 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:20.986436 24113 raft_consensus.cc:2243] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:20.987068 24113 raft_consensus.cc:2272] T 5cf962b21fa14634af6e810ca29f1bc7 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:20.991072  1027 consensus_peers.cc:597] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:20.991084 24113 tablet_replica.cc:333] stopping tablet replica
W20260709 13:12:20.994171  1027 consensus_peers.cc:597] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:20.998556 24113 raft_consensus.cc:2243] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:20.999336 24113 raft_consensus.cc:2272] T 21783150bb0b4141be797e841c3f46fa P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.010479 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.011543 24113 raft_consensus.cc:2243] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:21.012524 24113 raft_consensus.cc:2272] T 69355f8224b14b23be2cf00be247206c P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.016134 24113 tablet_replica.cc:333] stopping tablet replica
W20260709 13:12:21.016206  1101 consensus_peers.cc:597] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:21.016674  1175 consensus_peers.cc:597] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:21.017205 24113 raft_consensus.cc:2243] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 5 LEADER]: Raft consensus shutting down.
W20260709 13:12:21.029611  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 55439.8us
W20260709 13:12:21.032085  1027 consensus_peers.cc:597] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:21.032742  1027 consensus_peers.cc:597] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:21.041378 24113 raft_consensus.cc:2272] T 6aa365e967a941129ed031939342b1d8 P 294145900dd3496cb4a1d722613e1558 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.044437 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.050794 24113 raft_consensus.cc:2243] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.051451 24113 raft_consensus.cc:2272] T f6f69ac309b6488b81b409d785a2d465 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:21.061776  1027 consensus_peers.cc:597] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:21.053215  1101 consensus_peers.cc:597] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:21.017782  1175 consensus_peers.cc:597] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:21.063627  1175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 45905.7us
W20260709 13:12:21.063824  1101 consensus_peers.cc:597] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:21.064481  1101 consensus_peers.cc:597] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:21.070269  1027 consensus_peers.cc:597] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:21.071137  1175 consensus_peers.cc:597] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:21.071758  1175 consensus_peers.cc:597] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:21.071888  1027 consensus_peers.cc:597] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:21.073114 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.074142 24113 raft_consensus.cc:2243] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.078079 24113 raft_consensus.cc:2272] T eb722a3043814db7a188e49c9861ceca P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.081401 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.082378 24113 raft_consensus.cc:2243] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.082959 24113 raft_consensus.cc:2272] T 7526fd343f7d40528597242eb03386f8 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:21.087132  1027 consensus_peers.cc:597] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:21.088930 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.089852 24113 raft_consensus.cc:2243] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.090386 24113 raft_consensus.cc:2272] T 35cd7c6206834f5dabfca17b22a0605a P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:21.091912  1101 consensus_peers.cc:597] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:21.094053 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.094940 24113 raft_consensus.cc:2243] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.095449 24113 raft_consensus.cc:2272] T 161b9f09b3fb47febfabacf867781986 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.098438 24113 tablet_replica.cc:333] stopping tablet replica
W20260709 13:12:21.099097  1175 consensus_peers.cc:597] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:21.099337 24113 raft_consensus.cc:2243] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.099949 24113 raft_consensus.cc:2272] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.102742 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.103636 24113 raft_consensus.cc:2243] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.104125 24113 raft_consensus.cc:2272] T 29dde48ef51c4666a28b97181ec2ee21 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.107272 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.107892 24113 raft_consensus.cc:2243] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.108448 24113 raft_consensus.cc:2272] T ff504aa514be43da8b1aac12ab550daf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.111371 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.112277 24113 raft_consensus.cc:2243] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.112828 24113 raft_consensus.cc:2272] T fa73a73b72b2499fa5219cbf455099b4 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.115463 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.116158 24113 raft_consensus.cc:2243] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:21.116863 24113 raft_consensus.cc:2272] T 8acf44a40c084675a4f076a7acbda99d P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.119251 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.119977 24113 raft_consensus.cc:2243] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:21.120909 24113 raft_consensus.cc:2272] T 75b503b2e43f4996b5f494ab0e65afb0 P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.123752 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.124547 24113 raft_consensus.cc:2243] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.125038 24113 raft_consensus.cc:2272] T 9b75f9192e7246d2a560fd091fb68606 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.127981 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.128784 24113 raft_consensus.cc:2243] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:21.130090 24113 raft_consensus.cc:2272] T a0110653f11d472797f30dc22496cb12 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.132719 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.133590 24113 raft_consensus.cc:2243] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.134052 24113 raft_consensus.cc:2272] T 11adc51543f7421cb3140972d9f0a424 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.136725 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.137463 24113 raft_consensus.cc:2243] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:21.138396 24113 raft_consensus.cc:2272] T f31edd3b8d2546a2a98a8a8d3d038f8b P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.141170 24113 tablet_replica.cc:333] stopping tablet replica
W20260709 13:12:21.141403  1175 consensus_peers.cc:597] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:21.141924 24113 raft_consensus.cc:2243] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 13:12:21.142408  1175 consensus_peers.cc:597] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:21.142506 24113 raft_consensus.cc:2272] T 9eec3e72036941a5b16c3a0157986a93 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.146214 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.147141 24113 raft_consensus.cc:2243] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:21.148204 24113 raft_consensus.cc:2272] T 0867d95d9806478ba50d324c565446f6 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.151335 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.152215 24113 raft_consensus.cc:2243] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:21.153271 24113 raft_consensus.cc:2272] T 3b917eb6128a4341b3e3288130a65db1 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.156028 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.156862 24113 raft_consensus.cc:2243] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.157416 24113 raft_consensus.cc:2272] T c216d860ee834d4b8f77c8c39f0dba94 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.160763 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.161608 24113 raft_consensus.cc:2243] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.162122 24113 raft_consensus.cc:2272] T 545027a75bf749969ab288105031caff P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.164983 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.165761 24113 raft_consensus.cc:2243] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.166343 24113 raft_consensus.cc:2272] T 7f70fc428ff6447f8dca2e27ed05683a P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.169449 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.170274 24113 raft_consensus.cc:2243] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:21.171458 24113 raft_consensus.cc:2272] T 3e9125adfbbd414ca6b40457fc69ad84 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:21.171780  1101 consensus_peers.cc:597] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:21.174670 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.175590 24113 raft_consensus.cc:2243] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.176102 24113 raft_consensus.cc:2272] T c2312d768cbf4b3495d5b1574b0b6eb5 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.179453 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.180298 24113 raft_consensus.cc:2243] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:21.181342 24113 raft_consensus.cc:2272] T fc8c8635d8e54625af9c95ecbcd5653b P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.184259 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.185063 24113 raft_consensus.cc:2243] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.185642 24113 raft_consensus.cc:2272] T 91c2241524a24f7e911295c67f13bd26 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.188544 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.189328 24113 raft_consensus.cc:2243] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:12:21.190294 24113 raft_consensus.cc:2272] T 47f79cfcb8a04931bc19562b5d2c7754 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.192965 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.193718 24113 raft_consensus.cc:2243] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.194262 24113 raft_consensus.cc:2272] T 8f1971394c7d40bfa460d909b6f47438 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.197042 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.197885 24113 raft_consensus.cc:2243] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.198447 24113 raft_consensus.cc:2272] T 160ad44adbce4b7f97094b248fd05415 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.201215 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.201972 24113 raft_consensus.cc:2243] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:21.204098 24113 raft_consensus.cc:2272] T 33ceafb69dc74411898ea99b377d1032 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.209409 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.210317 24113 raft_consensus.cc:2243] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.210887 24113 raft_consensus.cc:2272] T f50253bf21654ca6937a357eb070beda P 294145900dd3496cb4a1d722613e1558 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:21.212246  1027 consensus_peers.cc:597] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:21.213716 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.214615 24113 raft_consensus.cc:2243] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:12:21.215610 24113 raft_consensus.cc:2272] T 4c162b2e111b472b88cd037925284de7 P 294145900dd3496cb4a1d722613e1558 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.218611 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.219379 24113 raft_consensus.cc:2243] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.219877 24113 raft_consensus.cc:2272] T 12df324e933f422c9a8ff5d37eb7d4ef P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.222707 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.223481 24113 raft_consensus.cc:2243] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.224025 24113 raft_consensus.cc:2272] T cae53ca608e74146a2f89675ddca72a6 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:21.225838  1101 consensus_peers.cc:597] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:21.227041 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.227960 24113 raft_consensus.cc:2243] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 5 LEADER]: Raft consensus shutting down.
I20260709 13:12:21.228999 24113 raft_consensus.cc:2272] T 09039e63d8db4e06b365f43533071272 P 294145900dd3496cb4a1d722613e1558 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.232087 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.232892 24113 raft_consensus.cc:2243] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.233326 24113 raft_consensus.cc:2272] T 90e13f7e16774e68bca79e3f141ed6bc P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.236235 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.237011 24113 raft_consensus.cc:2243] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.237568 24113 raft_consensus.cc:2272] T 483b49446c4e4a23b780921d7fabfbdf P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.240422 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.241242 24113 raft_consensus.cc:2243] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.241786 24113 raft_consensus.cc:2272] T b89f56a91e004f408b36c36d7aee0076 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.244771 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.245601 24113 raft_consensus.cc:2243] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.246104 24113 raft_consensus.cc:2272] T e038f78f3e9540399aeda1f38cbea3fd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.249667 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.250432 24113 raft_consensus.cc:2243] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.250998 24113 raft_consensus.cc:2272] T 11cd8a46e76f4bc685aa01885b27cc57 P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.253882 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:21.254585 24113 raft_consensus.cc:2243] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:21.255630 24113 raft_consensus.cc:2272] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 294145900dd3496cb4a1d722613e1558 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:21.258567 24113 tablet_replica.cc:333] stopping tablet replica
W20260709 13:12:21.261425  1027 consensus_peers.cc:597] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:21.259256 24113 raft_consensus.cc:2243] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:21.262480 24113 raft_consensus.cc:2272] T 219d22690fa145adb5b04192c1fc16bd P 294145900dd3496cb4a1d722613e1558 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:21.268805  1027 consensus_peers.cc:597] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:21.273825  1175 consensus_peers.cc:597] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:21.322489  1175 consensus_peers.cc:597] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:21.324504  1175 consensus_peers.cc:597] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:21.342154  1175 consensus_peers.cc:597] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:21.342906  1175 consensus_peers.cc:597] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:22.187557  2324 raft_consensus.cc:493] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.188100  2324 raft_consensus.cc:515] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.190276  2379 raft_consensus.cc:493] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.190857  2379 raft_consensus.cc:515] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.192889  2379 leader_election.cc:290] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:22.195572  1101 leader_election.cc:336] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.196630  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:12:22.199087  1101 leader_election.cc:336] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.199488  1101 leader_election.cc:304] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:22.200194  2379 raft_consensus.cc:2749] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:22.204208  2324 leader_election.cc:290] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:22.205926  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b917eb6128a4341b3e3288130a65db1" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:22.207311  1099 leader_election.cc:304] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:22.208308  2324 raft_consensus.cc:2749] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:22.232153  2338 raft_consensus.cc:493] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.232750  2338 raft_consensus.cc:515] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:22.235176  2338 leader_election.cc:290] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:22.236202  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
W20260709 13:12:22.240854  1027 leader_election.cc:336] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.241289  1027 leader_election.cc:304] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:22.242367  2338 raft_consensus.cc:2749] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:12:22.245069  2284 raft_consensus.cc:493] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.245656  2284 raft_consensus.cc:515] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.248988  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0110653f11d472797f30dc22496cb12" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:22.250627  2284 leader_election.cc:290] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
W20260709 13:12:22.260393  1101 leader_election.cc:336] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.260936  1101 leader_election.cc:304] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:22.262369  2284 raft_consensus.cc:2749] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:22.263157  2338 raft_consensus.cc:493] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.263947  2338 raft_consensus.cc:515] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
W20260709 13:12:22.270687  1027 leader_election.cc:336] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.272603  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:22.272936  2338 leader_election.cc:290] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:22.274623  1025 leader_election.cc:304] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:22.275368  2385 raft_consensus.cc:2749] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:12:22.295166  2376 raft_consensus.cc:493] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.295729  2376 raft_consensus.cc:515] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.297899  2376 leader_election.cc:290] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:22.299816  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0110653f11d472797f30dc22496cb12" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:22.300496  1132 raft_consensus.cc:2468] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 2.
I20260709 13:12:22.301530  1173 leader_election.cc:304] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:22.302536  2376 raft_consensus.cc:2804] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:12:22.302884  1175 leader_election.cc:336] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.303293  2284 raft_consensus.cc:493] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.302915  2376 raft_consensus.cc:493] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.303830  2284 raft_consensus.cc:515] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.304502  2376 raft_consensus.cc:3060] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:22.306013  2284 leader_election.cc:290] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:22.307345  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
W20260709 13:12:22.312589  1101 leader_election.cc:336] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.313055  1101 leader_election.cc:304] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:22.314530  2284 raft_consensus.cc:2749] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:12:22.317696  2376 raft_consensus.cc:515] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.320868  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0110653f11d472797f30dc22496cb12" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:22.321430  1132 raft_consensus.cc:3060] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:22.325696  2385 raft_consensus.cc:493] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.326242  2385 raft_consensus.cc:515] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.328603  2385 leader_election.cc:290] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:22.329746  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b917eb6128a4341b3e3288130a65db1" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:22.330502  1130 raft_consensus.cc:2468] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
I20260709 13:12:22.331781  1025 leader_election.cc:304] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:22.332602  2385 raft_consensus.cc:2804] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:22.333022  2385 raft_consensus.cc:493] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.333473  2385 raft_consensus.cc:3060] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:12:22.333999  1027 leader_election.cc:336] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.335704  2376 leader_election.cc:290] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:22.337188  1132 raft_consensus.cc:2468] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 3.
I20260709 13:12:22.342275  1173 leader_election.cc:304] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:22.343114  2367 raft_consensus.cc:2804] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:22.343576  2367 raft_consensus.cc:697] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 3 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:22.344362  2367 consensus_queue.cc:237] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.345309  2385 raft_consensus.cc:515] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.350931  2376 raft_consensus.cc:493] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.351503  2376 raft_consensus.cc:515] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.355453  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:22.355638  2376 leader_election.cc:290] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
W20260709 13:12:22.356557  1027 leader_election.cc:336] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.357931  2385 leader_election.cc:290] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:22.358898  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b917eb6128a4341b3e3288130a65db1" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:22.360852  1132 raft_consensus.cc:3060] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:12:22.363967 24113 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 1841)
W20260709 13:12:22.366550  1175 leader_election.cc:336] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.366633  2284 raft_consensus.cc:493] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.367267  2284 raft_consensus.cc:515] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.375204   880 catalog_manager.cc:5697] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 2 to 3, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 3 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
W20260709 13:12:22.382988  1101 leader_election.cc:336] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:22.384450  1175 leader_election.cc:336] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.384855  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:22.385641  1061 raft_consensus.cc:2468] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 4.
I20260709 13:12:22.386576  1132 raft_consensus.cc:2468] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 3.
I20260709 13:12:22.386922  1099 leader_election.cc:304] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:22.384927  1175 leader_election.cc:304] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:22.387923  2367 raft_consensus.cc:2749] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:22.388094  1025 leader_election.cc:304] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:22.389112  2385 raft_consensus.cc:2804] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:22.389600  2385 raft_consensus.cc:697] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:22.390427  2385 consensus_queue.cc:237] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.397547  2379 raft_consensus.cc:2804] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:12:22.398090  2379 raft_consensus.cc:493] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.398473  2379 raft_consensus.cc:3060] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:12:22.405750  2324 raft_consensus.cc:493] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.406311  2324 raft_consensus.cc:515] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.414022  2324 leader_election.cc:290] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:22.416168  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:22.416808  1201 raft_consensus.cc:2468] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:22.433861  2284 leader_election.cc:290] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:22.450474  2324 raft_consensus.cc:493] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.451052  2324 raft_consensus.cc:515] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.453477  2324 leader_election.cc:290] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:22.454082  2324 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.466033  2367 raft_consensus.cc:493] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.466812  2367 raft_consensus.cc:515] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.468782  2376 raft_consensus.cc:493] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.469334  2376 raft_consensus.cc:515] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.471825  2376 leader_election.cc:290] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:22.474609  2367 leader_election.cc:290] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:22.472332  2376 raft_consensus.cc:493] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.477985  2376 raft_consensus.cc:515] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:22.481536  2376 leader_election.cc:290] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:22.482411  1101 leader_election.cc:336] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.501935  2400 raft_consensus.cc:493] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.513260  2400 raft_consensus.cc:515] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:22.517241  2400 leader_election.cc:290] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:22.454679  2324 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.523447  2324 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:22.537971  1175 leader_election.cc:336] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:22.538563  1175 leader_election.cc:336] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:22.539084  1175 leader_election.cc:336] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.540743  2324 raft_consensus.cc:493] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.541644  2376 raft_consensus.cc:493] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.542183  2376 raft_consensus.cc:515] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.544562  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:22.505273  2397 raft_consensus.cc:493] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
W20260709 13:12:22.545542  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 129769us
I20260709 13:12:22.544704  2376 leader_election.cc:290] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:22.507460  2401 raft_consensus.cc:493] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.545414  2397 raft_consensus.cc:515] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.546653  2392 raft_consensus.cc:493] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.546597  2401 raft_consensus.cc:515] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.547313  2392 raft_consensus.cc:515] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.548175  2397 leader_election.cc:290] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:22.549022  2401 leader_election.cc:290] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:22.549587  2392 leader_election.cc:290] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:22.545416  1061 raft_consensus.cc:2468] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 5.
I20260709 13:12:22.558465  2379 raft_consensus.cc:515] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.560562  2379 leader_election.cc:290] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 5 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:22.562638  1129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:22.563169  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:22.563939  1131 raft_consensus.cc:2468] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 3.
I20260709 13:12:22.564067  1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8c8635d8e54625af9c95ecbcd5653b" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:22.563267  1129 raft_consensus.cc:2468] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 3.
I20260709 13:12:22.556988  2385 raft_consensus.cc:493] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.565289  2385 raft_consensus.cc:515] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.541393  2324 raft_consensus.cc:515] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.568616  2324 leader_election.cc:290] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:22.569607  1173 leader_election.cc:304] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:22.572866  2376 raft_consensus.cc:2804] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:12:22.573319  2376 raft_consensus.cc:493] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.573732  2376 raft_consensus.cc:3060] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:12:22.574946  1057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:22.568631  2393 raft_consensus.cc:493] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.579207  2393 raft_consensus.cc:515] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.581456  2393 leader_election.cc:290] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
W20260709 13:12:22.515122  1101 leader_election.cc:336] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:22.582171  1101 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 67098.3us
I20260709 13:12:22.591359  2367 raft_consensus.cc:493] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.591964  2367 raft_consensus.cc:515] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.594067  2367 leader_election.cc:290] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:22.595424  2376 raft_consensus.cc:515] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:22.603029  2385 leader_election.cc:290] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:22.607347  2376 leader_election.cc:290] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 6 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:22.610136  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:22.610844  1201 raft_consensus.cc:2468] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 3.
W20260709 13:12:22.612383  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 219233us
I20260709 13:12:22.616456  2392 raft_consensus.cc:493] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.617050  2392 raft_consensus.cc:515] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.618139  1057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa365e967a941129ed031939342b1d8" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:22.618726  1057 raft_consensus.cc:3060] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:12:22.623369  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:22.623993  1201 raft_consensus.cc:2468] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:22.626096  1057 raft_consensus.cc:2468] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 6.
W20260709 13:12:22.626327  1175 leader_election.cc:336] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.627569  1173 leader_election.cc:304] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:22.628600  1099 leader_election.cc:304] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:22.632997  2379 raft_consensus.cc:2804] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:22.628751  2376 raft_consensus.cc:2804] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:12:22.633785  2379 raft_consensus.cc:493] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.634241  2376 raft_consensus.cc:697] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 6 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:22.634316  2379 raft_consensus.cc:3060] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:22.635154  2376 consensus_queue.cc:237] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:22.637403  1057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c162b2e111b472b88cd037925284de7" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:22.638813  2367 raft_consensus.cc:493] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.640256  2338 raft_consensus.cc:493] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.646528  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:22.659628  1061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:22.660092  1131 raft_consensus.cc:2468] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
I20260709 13:12:22.657815  2392 leader_election.cc:290] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:22.660878  2338 raft_consensus.cc:515] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.658282  1129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:22.662073  1061 raft_consensus.cc:2468] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 3.
I20260709 13:12:22.647096  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867d95d9806478ba50d324c565446f6" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:22.663921  1130 raft_consensus.cc:2468] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
I20260709 13:12:22.641198  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:22.641111   883 catalog_manager.cc:5697] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 2 to 3, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 3 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:22.643141  1173 leader_election.cc:304] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:22.665402  2398 raft_consensus.cc:2804] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:12:22.665763  2398 raft_consensus.cc:493] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.665843  1173 leader_election.cc:304] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:22.666169  2398 raft_consensus.cc:3060] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:12:22.640890  1027 leader_election.cc:336] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.667347  1057 raft_consensus.cc:3060] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:12:22.663223  1129 raft_consensus.cc:2468] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 4.
W20260709 13:12:22.667310  1027 leader_election.cc:336] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.669529  2367 raft_consensus.cc:515] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:22.671540  2338 leader_election.cc:290] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:22.683872  1057 raft_consensus.cc:2468] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 5.
W20260709 13:12:22.676066  1027 leader_election.cc:336] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:22.663185  1101 leader_election.cc:336] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.678824  1201 raft_consensus.cc:2468] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 2.
I20260709 13:12:22.680408  2367 leader_election.cc:290] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:22.681401  2391 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.682163  2379 raft_consensus.cc:515] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
W20260709 13:12:22.686076  1101 leader_election.cc:336] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.686691  1099 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
W20260709 13:12:22.688627  1101 leader_election.cc:336] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:22.689107  1101 leader_election.cc:336] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:22.689575  1101 leader_election.cc:336] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.691497  2391 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
W20260709 13:12:22.692430  1101 leader_election.cc:336] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.694022  2391 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:22.694604  2379 leader_election.cc:290] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:22.695955  2393 raft_consensus.cc:2749] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:12:22.698047  1027 leader_election.cc:336] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:22.699159  1027 leader_election.cc:336] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.699856  1173 leader_election.cc:304] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:22.701375  1099 leader_election.cc:304] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:22.702167  2393 raft_consensus.cc:2804] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:12:22.702507  2393 raft_consensus.cc:493] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
W20260709 13:12:22.702713  1027 leader_election.cc:336] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.702885  2393 raft_consensus.cc:3060] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:22.703862  1099 leader_election.cc:304] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
W20260709 13:12:22.705667  1101 leader_election.cc:336] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.710242  1099 leader_election.cc:304] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:22.712116  1099 leader_election.cc:304] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:22.706339  2379 raft_consensus.cc:2804] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:12:22.706238  2367 raft_consensus.cc:2749] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:22.715296  2324 raft_consensus.cc:2804] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:12:22.716051  2284 raft_consensus.cc:2804] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:22.719242  2324 raft_consensus.cc:493] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.719280  2284 raft_consensus.cc:493] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.719823  2324 raft_consensus.cc:3060] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:22.719864  2284 raft_consensus.cc:3060] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:22.720549  2406 raft_consensus.cc:2804] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:12:22.720956  2406 raft_consensus.cc:493] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.721326  2406 raft_consensus.cc:3060] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:22.719731  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:22.724812  2379 raft_consensus.cc:697] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 5 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:22.730171  2398 raft_consensus.cc:515] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.730782  2284 raft_consensus.cc:515] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.732038  1130 raft_consensus.cc:2393] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 3: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:22.732406  2398 leader_election.cc:290] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:22.728740  2324 raft_consensus.cc:515] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.736948  2324 leader_election.cc:290] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
W20260709 13:12:22.740293  1101 leader_election.cc:336] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.748378  1201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8c8635d8e54625af9c95ecbcd5653b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:22.749433  1201 raft_consensus.cc:2468] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:22.747437  2284 leader_election.cc:290] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:22.725068  2393 raft_consensus.cc:515] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.753415  2379 consensus_queue.cc:237] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.755774  2393 leader_election.cc:290] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:22.756825  1101 leader_election.cc:336] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:22.761726  1175 leader_election.cc:336] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:22.762293  1175 leader_election.cc:336] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.763834  1025 leader_election.cc:304] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:22.764797  2391 raft_consensus.cc:2804] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:22.765205  2391 raft_consensus.cc:493] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.765604  2391 raft_consensus.cc:3060] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:12:22.766952  1101 leader_election.cc:336] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:22.769116  1173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 69319us
I20260709 13:12:22.771015  1057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:22.781685  2406 raft_consensus.cc:515] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.784364  2406 leader_election.cc:290] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
W20260709 13:12:22.797415  1175 leader_election.cc:336] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:22.799015  1175 leader_election.cc:336] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.828543   883 catalog_manager.cc:5697] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 5 to 6, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 6 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } }
W20260709 13:12:22.829949  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 117945us
I20260709 13:12:22.827042  2391 raft_consensus.cc:515] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
W20260709 13:12:22.880780  1027 consensus_peers.cc:597] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:22.886668  1175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 87657us
I20260709 13:12:22.934381  1127 raft_consensus.cc:1275] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:22.915894  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75b503b2e43f4996b5f494ab0e65afb0" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:22.937517  1131 raft_consensus.cc:3060] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:22.888645  1025 leader_election.cc:304] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:22.939287  2397 raft_consensus.cc:2804] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:12:22.939465  1025 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.67:33869, user_credentials={real_user=slave}} blocked reactor thread for 50929.7us
I20260709 13:12:22.939728  2397 raft_consensus.cc:493] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.940138  2397 raft_consensus.cc:3060] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:22.940238  1025 leader_election.cc:304] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
W20260709 13:12:22.941250  1027 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 60557.4us
I20260709 13:12:22.941393  2399 raft_consensus.cc:2804] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:22.941816  2399 raft_consensus.cc:493] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.942199  2399 raft_consensus.cc:3060] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:22.894860  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:22.944298  1130 raft_consensus.cc:2393] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c45b6d917734835a8a37b0f4c789ae6 in current term 4: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:22.895324  1129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:22.945118  1129 raft_consensus.cc:2468] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 5.
I20260709 13:12:22.946266  1057 raft_consensus.cc:2393] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 3: Already voted for candidate 9c45b6d917734835a8a37b0f4c789ae6 in this term.
I20260709 13:12:22.946740  1025 leader_election.cc:304] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:22.947520  2385 raft_consensus.cc:2804] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:12:22.947893  2385 raft_consensus.cc:493] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.948280  2385 raft_consensus.cc:3060] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:12:22.950439  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9125adfbbd414ca6b40457fc69ad84" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:22.951061  1202 raft_consensus.cc:3060] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:22.955950  2385 raft_consensus.cc:515] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.957864  1202 raft_consensus.cc:2468] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 3.
I20260709 13:12:22.897110  1058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867d95d9806478ba50d324c565446f6" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6" is_pre_election: true
I20260709 13:12:22.960188  1202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39dcc0888dc64bff9ce20b35e4bc8f89" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:22.960284  1206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:22.960769  1202 raft_consensus.cc:3060] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:22.905475  1128 raft_consensus.cc:1275] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:22.961017  1206 raft_consensus.cc:2393] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 4: Already voted for candidate 862b41697c74421fbac82c10c245434d in this term.
I20260709 13:12:22.967363  1099 leader_election.cc:304] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:22.947113  2391 leader_election.cc:290] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:22.970275  2385 leader_election.cc:290] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
W20260709 13:12:22.970819  1175 leader_election.cc:336] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.971086  2393 raft_consensus.cc:2804] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:22.971598  2393 raft_consensus.cc:493] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:22.971997  2393 raft_consensus.cc:3060] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:22.977509  1205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8acf44a40c084675a4f076a7acbda99d" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:22.977479  2399 raft_consensus.cc:515] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.978371  1205 raft_consensus.cc:3060] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:22.978919  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:22.979801  1130 raft_consensus.cc:2393] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 862b41697c74421fbac82c10c245434d in current term 4: Already voted for candidate f9d473477839468bbdf000a21d5f9485 in this term.
I20260709 13:12:22.982614  1099 leader_election.cc:304] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:22.897342  1055 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31edd3b8d2546a2a98a8a8d3d038f8b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c45b6d917734835a8a37b0f4c789ae6"
I20260709 13:12:22.983952  2392 raft_consensus.cc:2804] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:22.984109  1099 leader_election.cc:304] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 862b41697c74421fbac82c10c245434d
I20260709 13:12:22.984505  2392 raft_consensus.cc:697] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:22.985422  2410 raft_consensus.cc:2749] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:12:22.985355  2392 consensus_queue.cc:237] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.958112  2397 raft_consensus.cc:515] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:22.988539  1202 raft_consensus.cc:2468] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 3.
I20260709 13:12:22.988698  1131 raft_consensus.cc:2468] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 4.
I20260709 13:12:22.980020  2393 raft_consensus.cc:515] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:22.990219  1055 raft_consensus.cc:3060] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:22.993950  2397 leader_election.cc:290] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:22.994714  1205 raft_consensus.cc:2468] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 3.
I20260709 13:12:23.002015  1173 leader_election.cc:304] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:23.002846  2409 raft_consensus.cc:2749] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:23.004830  1058 raft_consensus.cc:2393] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d473477839468bbdf000a21d5f9485 in current term 3: Already voted for candidate 9c45b6d917734835a8a37b0f4c789ae6 in this term.
W20260709 13:12:23.007586  1175 consensus_peers.cc:597] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:23.009069  1101 leader_election.cc:336] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:23.011044  2393 leader_election.cc:290] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Requested vote from peers 862b41697c74421fbac82c10c245434d (127.23.140.68:34477), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:23.012425  1173 leader_election.cc:304] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:23.015126  1206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8c8635d8e54625af9c95ecbcd5653b" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:23.015726  1206 raft_consensus.cc:3060] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:23.018543  2399 leader_election.cc:290] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:23.018654  1025 leader_election.cc:304] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:23.020463  2407 raft_consensus.cc:2749] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:23.022300  2409 consensus_queue.cc:1048] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:23.020522  1025 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:23.026870  1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47f79cfcb8a04931bc19562b5d2c7754" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:23.027199  2397 raft_consensus.cc:2804] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:12:23.027555  1131 raft_consensus.cc:3060] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:12:23.027596  2397 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 294145900dd3496cb4a1d722613e1558)
I20260709 13:12:23.028095  2397 raft_consensus.cc:3060] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:12:23.028734  1173 leader_election.cc:304] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:23.030807  2406 raft_consensus.cc:2749] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:12:23.031104  2400 consensus_queue.cc:1048] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:23.032071  1173 leader_election.cc:304] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:23.036617  1025 leader_election.cc:304] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6; no voters: 
I20260709 13:12:23.030844  1206 raft_consensus.cc:2468] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 3.
I20260709 13:12:23.037446  1099 leader_election.cc:304] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:23.038595  1131 raft_consensus.cc:2468] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 5.
I20260709 13:12:23.040517  2400 raft_consensus.cc:2804] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:23.040904  2324 raft_consensus.cc:2804] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:12:23.041195  1027 leader_election.cc:336] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:23.041680  2324 raft_consensus.cc:697] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:23.042894  2324 consensus_queue.cc:237] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:23.044571  2406 raft_consensus.cc:2804] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:12:23.057819  2406 raft_consensus.cc:697] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 4 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:23.058789  2406 consensus_queue.cc:237] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:23.127920  1055 raft_consensus.cc:2468] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 4.
W20260709 13:12:23.149734  1027 leader_election.cc:336] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:22.937209   883 catalog_manager.cc:5697] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 4 to 5, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 5 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:23.151307  2397 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:23.138401  2338 raft_consensus.cc:2749] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:12:23.153040  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.68:34477, user_credentials={real_user=slave}} blocked reactor thread for 113875us
I20260709 13:12:23.133327  2400 raft_consensus.cc:697] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:23.160987  2400 consensus_queue.cc:237] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:23.160977  1099 leader_election.cc:304] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:23.162950  2393 raft_consensus.cc:2804] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:23.163477  2393 raft_consensus.cc:697] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:23.164438  2393 consensus_queue.cc:237] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:23.174687  1099 leader_election.cc:304] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
W20260709 13:12:23.302523  1099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.66:44097, user_credentials={real_user=slave}} blocked reactor thread for 127960us
I20260709 13:12:23.308811  1130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33ceafb69dc74411898ea99b377d1032" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:23.312949  1130 raft_consensus.cc:3060] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:23.302775  2393 raft_consensus.cc:3060] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:23.331691  1129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867d95d9806478ba50d324c565446f6" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:23.339993  2393 raft_consensus.cc:2749] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:23.351788  2397 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 6 election: Requested vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
W20260709 13:12:23.381644 24113 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 1841)
I20260709 13:12:23.388036  1025 leader_election.cc:304] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:23.420948  2414 raft_consensus.cc:2804] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:12:23.422261  2414 raft_consensus.cc:697] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:23.424512  2414 consensus_queue.cc:237] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:23.463351  1129 raft_consensus.cc:2468] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 3.
I20260709 13:12:23.474378  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09039e63d8db4e06b365f43533071272" candidate_uuid: "9c45b6d917734835a8a37b0f4c789ae6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:23.475651  1134 raft_consensus.cc:3060] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:12:23.484414  1130 raft_consensus.cc:2468] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 3.
I20260709 13:12:23.501384  1134 raft_consensus.cc:2468] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45b6d917734835a8a37b0f4c789ae6 in term 6.
I20260709 13:12:23.506316  1025 leader_election.cc:304] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:23.509181  2426 raft_consensus.cc:2804] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:23.509949  2426 raft_consensus.cc:697] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:23.511700  2426 consensus_queue.cc:237] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:23.516746  1025 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:23.519079  2414 raft_consensus.cc:2804] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:12:23.520031  2414 raft_consensus.cc:697] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 6 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:23.529155  2414 consensus_queue.cc:237] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
W20260709 13:12:23.585309  1027 leader_election.cc:336] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:23.586803  1027 leader_election.cc:336] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:23.599411  1027 leader_election.cc:336] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:23.600984  1027 consensus_peers.cc:597] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:23.645852  1099 leader_election.cc:304] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:23.653374  2284 raft_consensus.cc:2804] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:12:23.654100  2284 raft_consensus.cc:697] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 4 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:23.655411  2284 consensus_queue.cc:237] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:23.658488  1207 raft_consensus.cc:1275] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:23.660884  1025 leader_election.cc:304] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:23.666430  2385 raft_consensus.cc:2804] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:23.667097  2385 raft_consensus.cc:697] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Becoming Leader. State: Replica: 9c45b6d917734835a8a37b0f4c789ae6, State: Running, Role: LEADER
I20260709 13:12:23.670910   880 catalog_manager.cc:5697] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 2 to 3, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 3 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:23.668737  2385 consensus_queue.cc:237] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:23.684676   879 catalog_manager.cc:5697] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 3 to 4, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 4 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
W20260709 13:12:23.695406  1027 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 76073.2us
I20260709 13:12:23.698210  1056 raft_consensus.cc:1275] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 13:12:23.695188  1101 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.65:43721, user_credentials={real_user=slave}} blocked reactor thread for 81084.6us
W20260709 13:12:23.706538  1101 consensus_peers.cc:597] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:23.711266  1101 consensus_peers.cc:597] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:23.719123  1101 consensus_peers.cc:597] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:23.719942  1101 consensus_peers.cc:597] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:23.718431  1027 consensus_peers.cc:597] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:23.730408  1027 consensus_peers.cc:597] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:23.731421  1101 consensus_peers.cc:597] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:23.740902  1132 raft_consensus.cc:1275] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:12:23.753347  2437 consensus_queue.cc:1048] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:12:23.758661  2367 consensus_queue.cc:1048] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:12:23.767397  1054 raft_consensus.cc:1275] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 6 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:12:23.784570  2410 consensus_queue.cc:1048] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:12:23.785606  2376 consensus_queue.cc:1048] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:12:23.790982  1175 consensus_peers.cc:597] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:23.793398  1175 consensus_peers.cc:597] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:23.794075  1175 consensus_peers.cc:597] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:23.796598  1175 consensus_peers.cc:597] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:23.827822  1027 consensus_peers.cc:597] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:23.801483  1175 consensus_peers.cc:597] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:23.831853  1175 consensus_peers.cc:597] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:23.835621  1207 raft_consensus.cc:1275] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:23.841095  2397 consensus_queue.cc:1048] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:12:23.850448  1175 consensus_peers.cc:597] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:23.881639  1205 raft_consensus.cc:1275] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:23.883265  1203 raft_consensus.cc:1275] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 13:12:23.884285  1175 consensus_peers.cc:597] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:23.884883  2437 consensus_queue.cc:1048] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 13:12:23.887620  1101 consensus_peers.cc:597] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:23.891259  1101 consensus_peers.cc:597] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:23.892617  2284 consensus_queue.cc:1048] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 13:12:23.903831  1027 consensus_peers.cc:597] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:23.901481   880 catalog_manager.cc:5697] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 5 to 6, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 6 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:23.906136   880 catalog_manager.cc:5697] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 4 to 5, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 5 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:23.941370   880 catalog_manager.cc:5697] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 2 to 3, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 3 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:23.942873   879 catalog_manager.cc:5697] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 2 to 3, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } }
I20260709 13:12:23.943416   880 catalog_manager.cc:5697] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 reported cstate change: term changed from 2 to 3, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66). New cstate: current_term: 3 leader_uuid: "9c45b6d917734835a8a37b0f4c789ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } }
W20260709 13:12:23.971926  1027 consensus_peers.cc:597] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:23.973604  1101 consensus_peers.cc:597] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:23.976914  1175 consensus_peers.cc:597] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:23.978518  1027 consensus_peers.cc:597] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:23.987563   883 catalog_manager.cc:5697] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 2 to 3, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:23.989758   883 catalog_manager.cc:5697] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 2 to 3, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:23.991500   883 catalog_manager.cc:5697] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 3 to 4, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 4 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } }
W20260709 13:12:23.998108  1101 consensus_peers.cc:597] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:24.013870  1027 consensus_peers.cc:597] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:24.023094  1134 raft_consensus.cc:1275] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:24.024803  1132 raft_consensus.cc:1275] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 6 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:12:24.026206  2400 consensus_queue.cc:1048] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:12:24.027872  1027 consensus_peers.cc:597] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:24.030210  1134 raft_consensus.cc:1275] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:12:24.032240  2400 consensus_queue.cc:1048] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:12:24.033447  2385 consensus_queue.cc:1048] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260709 13:12:24.046290  1175 consensus_peers.cc:597] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:24.054520  1027 consensus_peers.cc:597] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:24.056370  1175 consensus_peers.cc:597] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:24.098008  1101 consensus_peers.cc:597] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:24.100065  1027 consensus_peers.cc:597] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:24.100612  1027 consensus_peers.cc:597] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:24.115710  1027 consensus_peers.cc:597] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:24.130391  1101 consensus_peers.cc:597] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:24.132808  1175 consensus_peers.cc:597] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:24.132761  1027 consensus_peers.cc:597] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:24.139941  1129 raft_consensus.cc:1275] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Refusing update from remote peer 9c45b6d917734835a8a37b0f4c789ae6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:24.143827  2401 consensus_queue.cc:1048] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:12:24.175892  1027 consensus_peers.cc:597] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:24.193176  1101 consensus_peers.cc:597] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:24.199769  1027 consensus_peers.cc:597] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:24.258010  1057 raft_consensus.cc:1275] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:12:24.259763  1175 consensus_peers.cc:597] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:24.267531  2437 consensus_queue.cc:1048] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:12:24.329097  1175 consensus_peers.cc:597] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:24.382622 24113 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 1841)
W20260709 13:12:24.408072  1157 debug-util.cc:398] Leaking SignalData structure 0x7b08000ecf20 after lost signal to thread 24116
W20260709 13:12:24.409883  1157 debug-util.cc:398] Leaking SignalData structure 0x7b0800110a80 after lost signal to thread 919
W20260709 13:12:24.411725  1157 debug-util.cc:398] Leaking SignalData structure 0x7b080024d3c0 after lost signal to thread 1086
W20260709 13:12:24.413156  1157 debug-util.cc:398] Leaking SignalData structure 0x7b08002e6900 after lost signal to thread 1160
W20260709 13:12:24.414453  1157 debug-util.cc:398] Leaking SignalData structure 0x7b080032b3e0 after lost signal to thread 1234
I20260709 13:12:24.581640  2367 raft_consensus.cc:493] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:24.582345  2367 raft_consensus.cc:515] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:24.587350  1134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:24.588135  1134 raft_consensus.cc:2468] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 4.
I20260709 13:12:24.590670  1173 leader_election.cc:304] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:24.591812  2407 raft_consensus.cc:2804] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:12:24.592213  2407 raft_consensus.cc:493] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:24.592556  2407 raft_consensus.cc:3060] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:12:24.608791  1175 leader_election.cc:336] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:24.617511  2367 leader_election.cc:290] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:24.636461  2407 raft_consensus.cc:515] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:24.674431  1129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69355f8224b14b23be2cf00be247206c" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:24.675104  1129 raft_consensus.cc:3060] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:12:24.683781  1129 raft_consensus.cc:2468] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 5.
W20260709 13:12:24.689436  1175 leader_election.cc:336] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:24.694586  2407 leader_election.cc:290] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 5 election: Requested vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:24.702858  1173 leader_election.cc:304] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558
I20260709 13:12:24.707051  2367 raft_consensus.cc:2804] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:12:24.707607  2367 raft_consensus.cc:697] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 5 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:24.708739  2367 consensus_queue.cc:237] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:24.735491   883 catalog_manager.cc:5697] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 3 to 5, leader changed from 294145900dd3496cb4a1d722613e1558 (127.23.140.65) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 5 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:25.093899 24113 tablet_server.cc:196] TabletServer@127.23.140.65:43721 shutdown complete.
I20260709 13:12:25.181344  1133 raft_consensus.cc:1275] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:12:25.205054  2367 consensus_queue.cc:1048] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 13:12:25.249250  1175 consensus_peers.cc:597] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:25.327626 24113 tablet_server.cc:179] TabletServer@127.23.140.66:0 shutting down...
W20260709 13:12:25.351645  1173 consensus_peers.cc:597] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:25.393057  1099 consensus_peers.cc:597] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:25.411470  1173 consensus_peers.cc:597] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:25.415081 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260709 13:12:25.423933  1173 consensus_peers.cc:597] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:25.425698 24113 tablet_replica.cc:333] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.427919 24113 raft_consensus.cc:2243] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.428754 24113 raft_consensus.cc:2272] T f31edd3b8d2546a2a98a8a8d3d038f8b P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.431828 24113 tablet_replica.cc:333] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.432796 24113 raft_consensus.cc:2243] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:25.433878 24113 raft_consensus.cc:2272] T 8acf44a40c084675a4f076a7acbda99d P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.436890 24113 tablet_replica.cc:333] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.437518 24113 raft_consensus.cc:2243] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:25.438477 24113 raft_consensus.cc:2272] T 21783150bb0b4141be797e841c3f46fa P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.441099 24113 tablet_replica.cc:333] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.441876 24113 raft_consensus.cc:2243] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 6 LEADER]: Raft consensus shutting down.
I20260709 13:12:25.442759 24113 raft_consensus.cc:2272] T 09039e63d8db4e06b365f43533071272 P 9c45b6d917734835a8a37b0f4c789ae6 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.445493 24113 tablet_replica.cc:333] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.446332 24113 raft_consensus.cc:2243] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:25.447381 24113 raft_consensus.cc:2272] T 0867d95d9806478ba50d324c565446f6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.450188 24113 tablet_replica.cc:333] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.453047 24113 raft_consensus.cc:2243] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.456962 24113 raft_consensus.cc:2272] T 9b75f9192e7246d2a560fd091fb68606 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.461999 24113 tablet_replica.cc:333] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.470080 24113 raft_consensus.cc:2243] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Raft consensus shutting down.
W20260709 13:12:25.470842  1173 consensus_peers.cc:597] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:25.472376 24113 raft_consensus.cc:2272] T eb722a3043814db7a188e49c9861ceca P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:25.473523  1099 consensus_peers.cc:597] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:25.479542 24113 tablet_replica.cc:333] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
W20260709 13:12:25.480911  1099 consensus_peers.cc:597] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:25.480499 24113 raft_consensus.cc:2243] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.482322 24113 raft_consensus.cc:2272] T 5d3595711f1e49dfb415fb2f6720e66d P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.485615 24113 tablet_replica.cc:333] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.486541 24113 raft_consensus.cc:2243] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:25.487663 24113 raft_consensus.cc:2272] T f6f69ac309b6488b81b409d785a2d465 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.490986 24113 tablet_replica.cc:333] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.491837 24113 raft_consensus.cc:2243] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.492560 24113 raft_consensus.cc:2272] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:25.493003  1173 consensus_peers.cc:597] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:25.495878 24113 tablet_replica.cc:333] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.496878 24113 raft_consensus.cc:2243] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.497550 24113 raft_consensus.cc:2272] T f267e3e7ffec446c9dfb1cfd18ace091 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.500949 24113 tablet_replica.cc:333] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.501891 24113 raft_consensus.cc:2243] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.502437 24113 raft_consensus.cc:2272] T 11adc51543f7421cb3140972d9f0a424 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:25.505522  1099 consensus_peers.cc:597] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:25.508687 24113 tablet_replica.cc:333] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.509714 24113 raft_consensus.cc:2243] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.510386 24113 raft_consensus.cc:2272] T d3fbed7be9014cf5984a19768b6269d4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.514581 24113 tablet_replica.cc:333] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.515413 24113 raft_consensus.cc:2243] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.515977 24113 raft_consensus.cc:2272] T 483b49446c4e4a23b780921d7fabfbdf P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.519029 24113 tablet_replica.cc:333] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.519877 24113 raft_consensus.cc:2243] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.520491 24113 raft_consensus.cc:2272] T 90e13f7e16774e68bca79e3f141ed6bc P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.523819 24113 tablet_replica.cc:333] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.524709 24113 raft_consensus.cc:2243] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 LEADER]: Raft consensus shutting down.
I20260709 13:12:25.525830 24113 raft_consensus.cc:2272] T 47f79cfcb8a04931bc19562b5d2c7754 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:25.527760  1173 consensus_peers.cc:597] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:25.532459  1099 consensus_peers.cc:597] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:25.533915  1099 consensus_peers.cc:597] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:25.536738 24113 tablet_replica.cc:333] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.537855 24113 raft_consensus.cc:2243] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.538945 24113 raft_consensus.cc:2272] T 8f1971394c7d40bfa460d909b6f47438 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.543499 24113 tablet_replica.cc:333] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.544802 24113 raft_consensus.cc:2243] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.545588 24113 raft_consensus.cc:2272] T 91c2241524a24f7e911295c67f13bd26 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.548718 24113 tablet_replica.cc:333] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.549584 24113 raft_consensus.cc:2243] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:25.550874 24113 raft_consensus.cc:2272] T 7780e3bac1204186aefaa6f574717fdb P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.554095 24113 tablet_replica.cc:333] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.555023 24113 raft_consensus.cc:2243] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:25.556170 24113 raft_consensus.cc:2272] T 160ad44adbce4b7f97094b248fd05415 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.559219 24113 tablet_replica.cc:333] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.560081 24113 raft_consensus.cc:2243] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 LEADER]: Raft consensus shutting down.
I20260709 13:12:25.562109 24113 raft_consensus.cc:2272] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.565405 24113 tablet_replica.cc:333] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.566313 24113 raft_consensus.cc:2243] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:25.567461 24113 raft_consensus.cc:2272] T 33ceafb69dc74411898ea99b377d1032 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.570878 24113 tablet_replica.cc:333] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.571902 24113 raft_consensus.cc:2243] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.572866 24113 raft_consensus.cc:2272] T 7526fd343f7d40528597242eb03386f8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.576634 24113 tablet_replica.cc:333] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.578078 24113 raft_consensus.cc:2243] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:25.579759 24113 raft_consensus.cc:2272] T f50253bf21654ca6937a357eb070beda P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.583236 24113 tablet_replica.cc:333] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.583762 24113 raft_consensus.cc:2243] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.584193 24113 raft_consensus.cc:2272] T 4c162b2e111b472b88cd037925284de7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.586354 24113 tablet_replica.cc:333] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.586879 24113 raft_consensus.cc:2243] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 LEADER]: Raft consensus shutting down.
I20260709 13:12:25.587774 24113 raft_consensus.cc:2272] T 120ba483d3f1448e967c504d55e6d5f3 P 9c45b6d917734835a8a37b0f4c789ae6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.590421 24113 tablet_replica.cc:333] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.591419 24113 raft_consensus.cc:2243] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.591938 24113 raft_consensus.cc:2272] T 11cd8a46e76f4bc685aa01885b27cc57 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.594919 24113 tablet_replica.cc:333] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
W20260709 13:12:25.595027  1173 consensus_peers.cc:597] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:25.595752 24113 raft_consensus.cc:2243] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.596380 24113 raft_consensus.cc:2272] T 6aa365e967a941129ed031939342b1d8 P 9c45b6d917734835a8a37b0f4c789ae6 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.599098 24113 tablet_replica.cc:333] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.599920 24113 raft_consensus.cc:2243] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:25.600986 24113 raft_consensus.cc:2272] T 12df324e933f422c9a8ff5d37eb7d4ef P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.604805 24113 tablet_replica.cc:333] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.605590 24113 raft_consensus.cc:2243] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.606225 24113 raft_consensus.cc:2272] T a81738ce3431443d9e332eb931ec3c0b P 9c45b6d917734835a8a37b0f4c789ae6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.608870 24113 tablet_replica.cc:333] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.609673 24113 raft_consensus.cc:2243] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:25.610935 24113 raft_consensus.cc:2272] T cae53ca608e74146a2f89675ddca72a6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.613915 24113 tablet_replica.cc:333] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.614750 24113 raft_consensus.cc:2243] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:25.615837 24113 raft_consensus.cc:2272] T 35cd7c6206834f5dabfca17b22a0605a P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.619026 24113 tablet_replica.cc:333] T 161b9f09b3fb47febfabacf867781986 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.619792 24113 raft_consensus.cc:2243] T 161b9f09b3fb47febfabacf867781986 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.620409 24113 raft_consensus.cc:2272] T 161b9f09b3fb47febfabacf867781986 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.624439 24113 tablet_replica.cc:333] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.626319 24113 raft_consensus.cc:2243] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.626897 24113 raft_consensus.cc:2272] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.630218 24113 tablet_replica.cc:333] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.631126 24113 raft_consensus.cc:2243] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 LEADER]: Raft consensus shutting down.
W20260709 13:12:25.628082  1099 consensus_peers.cc:597] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:25.633459 24113 raft_consensus.cc:2272] T 5cf962b21fa14634af6e810ca29f1bc7 P 9c45b6d917734835a8a37b0f4c789ae6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.636842 24113 tablet_replica.cc:333] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.637645 24113 raft_consensus.cc:2243] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.638139 24113 raft_consensus.cc:2272] T f95cbeb7360c4c61b442e75e35d93e73 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.641115 24113 tablet_replica.cc:333] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
W20260709 13:12:25.641387  1173 consensus_peers.cc:597] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:25.641939 24113 raft_consensus.cc:2243] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.642594 24113 raft_consensus.cc:2272] T e8aead26aca7435abf7d4af1ab2321af P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.648059 24113 tablet_replica.cc:333] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.648828 24113 raft_consensus.cc:2243] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 13:12:25.648983  1099 consensus_peers.cc:597] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:25.649415 24113 raft_consensus.cc:2272] T 4ddfc34a205c43ad81653b5000cbf394 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.652447 24113 tablet_replica.cc:333] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.653270 24113 raft_consensus.cc:2243] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.654115 24113 raft_consensus.cc:2272] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:25.655551  1173 consensus_peers.cc:597] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:25.657008 24113 tablet_replica.cc:333] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.657917 24113 raft_consensus.cc:2243] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Raft consensus shutting down.
W20260709 13:12:25.658279  1027 consensus_peers.cc:597] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:25.659065 24113 raft_consensus.cc:2272] T fa73a73b72b2499fa5219cbf455099b4 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.662047 24113 tablet_replica.cc:333] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.662810 24113 raft_consensus.cc:2243] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:25.664196 24113 raft_consensus.cc:2272] T c2312d768cbf4b3495d5b1574b0b6eb5 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:25.665014  1099 consensus_peers.cc:597] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:25.666502  1175 consensus_peers.cc:597] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:25.668154 24113 tablet_replica.cc:333] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.669013 24113 raft_consensus.cc:2243] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:25.670217 24113 raft_consensus.cc:2272] T 3b917eb6128a4341b3e3288130a65db1 P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.673332 24113 tablet_replica.cc:333] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.674043 24113 raft_consensus.cc:2243] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.674602 24113 raft_consensus.cc:2272] T 240eb8c24e0142c8a23a0afdd5dccda6 P 9c45b6d917734835a8a37b0f4c789ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:25.678583 24113 tablet_replica.cc:333] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6: stopping tablet replica
I20260709 13:12:25.679448 24113 raft_consensus.cc:2243] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:25.680037 24113 raft_consensus.cc:2272] T f0afcfb9359a421b85fabc92b7733d9b P 9c45b6d917734835a8a37b0f4c789ae6 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:25.684547  1099 consensus_peers.cc:597] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:25.691619  1099 consensus_peers.cc:597] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:25.701292  1173 consensus_peers.cc:597] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:25.725556  1101 consensus_peers.cc:597] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:25.774331  1099 consensus_peers.cc:597] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:25.828056  1099 consensus_peers.cc:597] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:25.872754  1101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111) [suppressed 412 similar messages]
W20260709 13:12:25.876324  1101 consensus_peers.cc:597] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:25.883679  1099 consensus_peers.cc:597] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:25.900288  1175 consensus_peers.cc:597] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:25.948118  1175 consensus_peers.cc:597] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:25.966681  1175 consensus_peers.cc:597] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:26.072391  1101 consensus_peers.cc:597] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:26.131409  1101 consensus_peers.cc:597] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:26.201073  1101 consensus_peers.cc:597] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:26.224978  1175 consensus_peers.cc:597] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:26.248371  1175 consensus_peers.cc:597] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:26.258720  1101 consensus_peers.cc:597] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:26.311261  1175 consensus_peers.cc:597] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:26.354055  1175 consensus_peers.cc:597] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:26.373921  1101 consensus_peers.cc:597] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:26.375690  1175 consensus_peers.cc:597] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:26.380649  1175 consensus_peers.cc:597] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:26.403599  1175 consensus_peers.cc:597] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:26.425401  1101 consensus_peers.cc:597] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:26.443007  1175 consensus_peers.cc:597] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:26.512418  1175 consensus_peers.cc:597] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:26.515756  1175 consensus_peers.cc:597] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:12:26.535547  2477 raft_consensus.cc:493] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.536123  2477 raft_consensus.cc:515] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:26.538288  2477 leader_election.cc:290] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:26.544730  1101 leader_election.cc:336] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:26.562192  1101 consensus_peers.cc:597] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:26.567574  1099 leader_election.cc:336] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.568041  1099 leader_election.cc:304] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:26.568672  2477 raft_consensus.cc:2749] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:12:26.604167  1101 consensus_peers.cc:597] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:12:26.612190  2480 raft_consensus.cc:493] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.612711  2480 raft_consensus.cc:515] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
W20260709 13:12:26.618921  1175 leader_election.cc:336] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:26.619421  1173 leader_election.cc:336] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.619596  2480 leader_election.cc:290] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:26.619910  1173 leader_election.cc:304] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:26.620607  2480 raft_consensus.cc:2749] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:26.638690  2480 raft_consensus.cc:493] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.639132  2480 raft_consensus.cc:515] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:26.641034  2480 leader_election.cc:290] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:26.642992  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7780e3bac1204186aefaa6f574717fdb" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
W20260709 13:12:26.644712  1173 leader_election.cc:336] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.645452  1173 leader_election.cc:304] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:26.646030  2480 raft_consensus.cc:2749] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:26.680401  2477 raft_consensus.cc:493] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.680972  2477 raft_consensus.cc:515] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:26.682895  2477 leader_election.cc:290] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:26.685098  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:26.688050  2479 raft_consensus.cc:493] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.688601  2479 raft_consensus.cc:515] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:26.690703  2479 leader_election.cc:290] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:26.694761  2479 raft_consensus.cc:493] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.695293  2479 raft_consensus.cc:515] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
W20260709 13:12:26.697764  1175 leader_election.cc:336] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:26.699152  2479 leader_election.cc:290] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:26.697836  1099 leader_election.cc:336] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.699715  1099 leader_election.cc:304] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:26.700570  2477 raft_consensus.cc:2749] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:12:26.705305  1173 leader_election.cc:336] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.705772  1173 leader_election.cc:304] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:26.706439  2480 raft_consensus.cc:2749] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:12:26.709414  1175 leader_election.cc:336] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:26.709933  2480 raft_consensus.cc:493] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.711571  2480 raft_consensus.cc:515] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
W20260709 13:12:26.712230  1173 leader_election.cc:336] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.712711  1173 leader_election.cc:304] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:26.713605  2480 leader_election.cc:290] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:26.714380  2479 raft_consensus.cc:2749] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:26.721335  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:26.721988  1133 raft_consensus.cc:2468] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 5.
I20260709 13:12:26.728394  2466 raft_consensus.cc:493] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.730394  2466 raft_consensus.cc:515] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:26.732582  2466 leader_election.cc:290] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:26.724584  1173 leader_election.cc:336] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.734475  1173 leader_election.cc:304] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:26.735220  2480 raft_consensus.cc:2804] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:12:26.735586  2480 raft_consensus.cc:493] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.735939  2480 raft_consensus.cc:3060] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:12:26.738570  1101 leader_election.cc:336] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:26.742144  1099 leader_election.cc:336] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.742600  1099 leader_election.cc:304] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:26.743327  2466 raft_consensus.cc:2749] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:26.745216  2466 raft_consensus.cc:493] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.745679  2466 raft_consensus.cc:515] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:26.753983  2466 leader_election.cc:290] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:26.756727  1101 leader_election.cc:336] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:26.765373  2479 raft_consensus.cc:493] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.765934  2479 raft_consensus.cc:515] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:26.767990  2479 leader_election.cc:290] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:26.772439  1175 leader_election.cc:336] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:26.772477  2466 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
W20260709 13:12:26.774747  1099 leader_election.cc:336] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.773888  2466 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:26.775180  1099 leader_election.cc:304] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:26.776289  2477 raft_consensus.cc:2749] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:26.777264  2466 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:26.771816  2480 raft_consensus.cc:515] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:26.780076  2480 leader_election.cc:290] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 6 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
W20260709 13:12:26.783105  1101 leader_election.cc:336] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:26.786067 24113 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 1083)
W20260709 13:12:26.795042  1173 leader_election.cc:336] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.795568  1173 leader_election.cc:304] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:26.796275  2480 raft_consensus.cc:2749] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:12:26.797294  1173 leader_election.cc:336] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.800616  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "120ba483d3f1448e967c504d55e6d5f3" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:26.801213  1133 raft_consensus.cc:3060] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:12:26.805572  2466 raft_consensus.cc:493] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.806155  2466 raft_consensus.cc:515] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:26.809418  2466 leader_election.cc:290] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:26.814448  1133 raft_consensus.cc:2468] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 6.
W20260709 13:12:26.815707  1099 leader_election.cc:336] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.816222  1099 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:26.817999  2466 raft_consensus.cc:2749] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:12:26.818229  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
W20260709 13:12:26.818813  1101 consensus_peers.cc:597] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:12:26.820441  1173 leader_election.cc:304] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:26.821296  2480 raft_consensus.cc:2804] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:12:26.821774  2480 raft_consensus.cc:697] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 6 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:26.822595  2480 consensus_queue.cc:237] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
W20260709 13:12:26.823661  1099 leader_election.cc:336] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.824434  1099 leader_election.cc:304] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 862b41697c74421fbac82c10c245434d, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:26.826052  2466 raft_consensus.cc:2749] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:12:26.836730   880 catalog_manager.cc:5697] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 5 to 6, leader changed from 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 6 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:26.868752  2480 raft_consensus.cc:493] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.869344  2480 raft_consensus.cc:515] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:26.871541  2480 leader_election.cc:290] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:26.878734  1175 leader_election.cc:336] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:26.880672  1173 leader_election.cc:336] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.881072  1173 leader_election.cc:304] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:26.883159  2477 raft_consensus.cc:493] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.883694  2477 raft_consensus.cc:515] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:26.885679  2477 leader_election.cc:290] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:26.886086  2480 raft_consensus.cc:2749] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:26.890750  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7780e3bac1204186aefaa6f574717fdb" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d" is_pre_election: true
I20260709 13:12:26.891363  1203 raft_consensus.cc:2468] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 2.
I20260709 13:12:26.896245  1099 leader_election.cc:304] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
W20260709 13:12:26.897217  1099 leader_election.cc:336] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.896931  2477 raft_consensus.cc:2804] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:12:26.897984  2477 raft_consensus.cc:493] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.898420  2477 raft_consensus.cc:3060] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:26.900231  2466 raft_consensus.cc:493] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.900766  2466 raft_consensus.cc:515] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:26.902825  2466 leader_election.cc:290] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:26.909726  2477 raft_consensus.cc:515] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
W20260709 13:12:26.911727  1101 leader_election.cc:336] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:26.911794  2477 leader_election.cc:290] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), 862b41697c74421fbac82c10c245434d (127.23.140.68:34477)
I20260709 13:12:26.917189  2461 raft_consensus.cc:493] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.917706  2461 raft_consensus.cc:515] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:26.919422  2461 leader_election.cc:290] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:26.924443  1173 leader_election.cc:336] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:26.925442  1101 consensus_peers.cc:597] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:12:26.927296  2393 raft_consensus.cc:493] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.927794  2393 raft_consensus.cc:515] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:26.930779  2393 leader_election.cc:290] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:26.939863  1101 leader_election.cc:336] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:26.947101  1099 leader_election.cc:336] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.947753  1099 leader_election.cc:304] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:26.949536  2393 raft_consensus.cc:2749] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:26.952366  1203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7780e3bac1204186aefaa6f574717fdb" candidate_uuid: "f9d473477839468bbdf000a21d5f9485" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "862b41697c74421fbac82c10c245434d"
I20260709 13:12:26.952965  1203 raft_consensus.cc:3060] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:12:26.955427  1099 leader_election.cc:336] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:26.955929  1099 leader_election.cc:336] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.956257  1099 leader_election.cc:304] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:26.956873  2393 raft_consensus.cc:2749] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:26.957926  2393 raft_consensus.cc:493] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.958360  2393 raft_consensus.cc:515] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:26.960014  1203 raft_consensus.cc:2468] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d473477839468bbdf000a21d5f9485 in term 3.
W20260709 13:12:26.964207  1175 leader_election.cc:336] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:26.964692  1175 leader_election.cc:304] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
W20260709 13:12:26.964946  1101 leader_election.cc:336] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:26.965685  2480 raft_consensus.cc:2749] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:26.960139  2393 leader_election.cc:290] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:26.968027  1175 consensus_peers.cc:597] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:12:26.969456  1099 leader_election.cc:304] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:26.970099  2393 raft_consensus.cc:2804] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:26.970574  2393 raft_consensus.cc:697] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Becoming Leader. State: Replica: f9d473477839468bbdf000a21d5f9485, State: Running, Role: LEADER
I20260709 13:12:26.971428  2393 consensus_queue.cc:237] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:26.976377  2480 raft_consensus.cc:493] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.976922  2480 raft_consensus.cc:515] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:26.978921  2480 leader_election.cc:290] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:26.979811  1099 leader_election.cc:336] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.980268  1099 leader_election.cc:304] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:26.982976  2479 raft_consensus.cc:493] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:26.983251  2477 raft_consensus.cc:2749] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:26.983502  2479 raft_consensus.cc:515] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:26.988576  2479 leader_election.cc:290] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
W20260709 13:12:26.988811  1175 leader_election.cc:336] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:26.991340  1173 leader_election.cc:336] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:26.994706  1173 leader_election.cc:304] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:26.995774  2480 raft_consensus.cc:2749] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:26.998536  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485" is_pre_election: true
I20260709 13:12:26.999264  1133 raft_consensus.cc:2468] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 5.
I20260709 13:12:27.001030  1173 leader_election.cc:304] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 
I20260709 13:12:27.004063  2480 raft_consensus.cc:2804] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:12:27.004671  2480 raft_consensus.cc:493] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:27.005227  2480 raft_consensus.cc:3060] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:12:27.011011   880 catalog_manager.cc:5697] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 reported cstate change: term changed from 2 to 3, leader changed from 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66) to f9d473477839468bbdf000a21d5f9485 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "f9d473477839468bbdf000a21d5f9485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: UNKNOWN } } }
W20260709 13:12:27.004032  1173 leader_election.cc:336] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:27.017217  2477 raft_consensus.cc:493] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:27.017696  2477 raft_consensus.cc:515] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:27.019742  2477 leader_election.cc:290] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:27.027027  1099 leader_election.cc:336] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:27.027724  1101 leader_election.cc:336] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:27.028165  1101 leader_election.cc:304] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:27.028903  2466 raft_consensus.cc:2749] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:27.035555  2480 raft_consensus.cc:515] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
W20260709 13:12:27.043279  1173 leader_election.cc:336] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:27.043803  2393 raft_consensus.cc:493] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:27.044396  2393 raft_consensus.cc:515] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
W20260709 13:12:27.050899  1101 leader_election.cc:336] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:27.052676  2393 leader_election.cc:290] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:27.059087  1099 leader_election.cc:336] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:27.059541  1099 leader_election.cc:304] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:27.062557  2393 raft_consensus.cc:2749] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:27.069453  2480 leader_election.cc:290] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 6 election: Requested vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:27.074025  1133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a260eb172eba4c6db7c0d0fab6a7f7d1" candidate_uuid: "862b41697c74421fbac82c10c245434d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f9d473477839468bbdf000a21d5f9485"
I20260709 13:12:27.074713  1133 raft_consensus.cc:3060] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:12:27.082037  1133 raft_consensus.cc:2468] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862b41697c74421fbac82c10c245434d in term 6.
I20260709 13:12:27.083026  1173 leader_election.cc:304] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862b41697c74421fbac82c10c245434d, f9d473477839468bbdf000a21d5f9485; no voters: 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:27.083787  2480 raft_consensus.cc:2804] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:12:27.084275  2480 raft_consensus.cc:697] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 6 LEADER]: Becoming Leader. State: Replica: 862b41697c74421fbac82c10c245434d, State: Running, Role: LEADER
I20260709 13:12:27.085184  2480 consensus_queue.cc:237] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:27.094491  2480 raft_consensus.cc:493] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:27.095106  2480 raft_consensus.cc:515] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:27.099462  2480 leader_election.cc:290] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:27.099121   880 catalog_manager.cc:5697] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d reported cstate change: term changed from 5 to 6, leader changed from 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66) to 862b41697c74421fbac82c10c245434d (127.23.140.68). New cstate: current_term: 6 leader_uuid: "862b41697c74421fbac82c10c245434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } health_report { overall_health: HEALTHY } } }
W20260709 13:12:27.104292  1173 leader_election.cc:336] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:27.104981  1175 leader_election.cc:336] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:27.105386  1175 leader_election.cc:304] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:27.106104  2479 raft_consensus.cc:2749] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:27.243983  1133 raft_consensus.cc:1275] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 6 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:12:27.246207  2480 consensus_queue.cc:1048] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260709 13:12:27.349047  1173 consensus_peers.cc:597] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:27.422720  1099 consensus_peers.cc:597] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:27.507038  1173 consensus_peers.cc:597] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:27.511399  1175 consensus_peers.cc:597] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:27.567879  1134 raft_consensus.cc:1275] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 6 FOLLOWER]: Refusing update from remote peer 862b41697c74421fbac82c10c245434d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:12:27.571449  2461 consensus_queue.cc:1048] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:12:27.620419  1203 raft_consensus.cc:1275] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Refusing update from remote peer f9d473477839468bbdf000a21d5f9485: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:27.626478  2477 consensus_queue.cc:1048] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 13:12:27.709713  1173 consensus_peers.cc:597] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:27.710330  1173 consensus_peers.cc:597] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:27.786504 24113 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 1083)
W20260709 13:12:27.792328  1099 consensus_peers.cc:597] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:27.949247  1099 consensus_peers.cc:597] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:27.999428  1099 consensus_peers.cc:597] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.015643  1101 consensus_peers.cc:597] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:12:28.032472  1173 consensus_peers.cc:597] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.069073  1099 consensus_peers.cc:597] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.076232  1099 consensus_peers.cc:597] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.077486  1173 consensus_peers.cc:597] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.105480  1173 consensus_peers.cc:597] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.114785  1099 consensus_peers.cc:597] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.144773  1101 consensus_peers.cc:597] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:28.148288  1175 consensus_peers.cc:597] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:28.200522  1173 consensus_peers.cc:597] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.203704  1173 consensus_peers.cc:597] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.234252  1173 consensus_peers.cc:597] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.243203  1099 consensus_peers.cc:597] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.317879  1175 consensus_peers.cc:597] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:12:28.340076  2461 raft_consensus.cc:493] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:28.340698  2461 raft_consensus.cc:515] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:28.342837  2461 leader_election.cc:290] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:28.350528  1173 leader_election.cc:336] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:28.351217  1175 leader_election.cc:336] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:28.351575  1175 leader_election.cc:304] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:28.352236  2461 raft_consensus.cc:2749] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:12:28.364648  1099 consensus_peers.cc:597] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.379386  1099 consensus_peers.cc:597] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.379846  1173 consensus_peers.cc:597] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.408954  1099 consensus_peers.cc:597] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.405165  1173 consensus_peers.cc:597] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.430344  1175 consensus_peers.cc:597] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:12:28.471091  1175 consensus_peers.cc:597] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:12:28.504232  1175 consensus_peers.cc:597] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:28.509084  1099 consensus_peers.cc:597] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.513432  1099 consensus_peers.cc:597] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.523917  1099 consensus_peers.cc:597] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:28.581157  1099 consensus_peers.cc:597] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:28.586025  2461 raft_consensus.cc:493] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:28.586676  2461 raft_consensus.cc:515] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:28.590209  2461 leader_election.cc:290] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:28.597952  1175 leader_election.cc:336] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:28.600404  1173 leader_election.cc:336] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:28.600929  1173 leader_election.cc:304] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:28.607198  2492 raft_consensus.cc:2749] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:28.641847  2495 raft_consensus.cc:493] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:28.642380  2495 raft_consensus.cc:515] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:28.644407  2495 leader_election.cc:290] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:28.645797  2495 raft_consensus.cc:493] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:28.647295  2495 raft_consensus.cc:515] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
W20260709 13:12:28.649993  1101 leader_election.cc:336] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:28.652952  1101 leader_election.cc:336] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:28.649533  2495 leader_election.cc:290] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:28.660142  1099 leader_election.cc:336] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:28.660647  1099 leader_election.cc:304] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:28.661381  2497 raft_consensus.cc:2749] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:12:28.662361  1099 leader_election.cc:336] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:28.663910  1099 leader_election.cc:304] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:28.665975  2497 raft_consensus.cc:2749] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:12:28.678856  2495 raft_consensus.cc:493] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
W20260709 13:12:28.680648  1101 consensus_peers.cc:597] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:12:28.681145  2495 raft_consensus.cc:515] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:28.683856  2495 leader_election.cc:290] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:28.691275  1101 leader_election.cc:336] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:28.704599  1099 leader_election.cc:336] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:28.705435  1099 leader_election.cc:304] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:28.706171  2495 raft_consensus.cc:2749] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:12:28.740054  1175 consensus_peers.cc:597] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:12:28.768216  1175 consensus_peers.cc:597] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:12:28.770630  2492 raft_consensus.cc:493] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:28.771325  2492 raft_consensus.cc:515] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:28.773593  2492 leader_election.cc:290] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:28.776847  1101 consensus_peers.cc:597] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:28.780314  1101 consensus_peers.cc:597] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:12:28.780453  1173 leader_election.cc:336] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:28.787060 24113 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 1083)
W20260709 13:12:28.788760  1175 leader_election.cc:336] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:28.789373  1175 leader_election.cc:304] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
W20260709 13:12:28.800205  1101 consensus_peers.cc:597] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:12:28.803805  2495 raft_consensus.cc:493] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:28.804430  2495 raft_consensus.cc:515] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:28.806703  2495 leader_election.cc:290] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:28.814546  1099 leader_election.cc:336] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:28.815399  1101 consensus_peers.cc:597] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:12:28.816007  2492 raft_consensus.cc:2749] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:12:28.818764  1175 consensus_peers.cc:597] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:12:28.820719  2492 raft_consensus.cc:493] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:28.822467  2492 raft_consensus.cc:515] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
W20260709 13:12:28.821195  1101 leader_election.cc:336] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:28.824437  1101 leader_election.cc:304] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:28.825210  2495 raft_consensus.cc:2749] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:28.827385  2498 raft_consensus.cc:493] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:28.827934  2498 raft_consensus.cc:515] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
W20260709 13:12:28.829263  1175 consensus_peers.cc:597] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:12:28.830163  2498 leader_election.cc:290] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:28.832163  2492 leader_election.cc:290] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:28.834810  1175 consensus_peers.cc:597] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:12:28.838037  1173 leader_election.cc:336] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:28.840274  1175 leader_election.cc:336] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:28.840754  1175 leader_election.cc:304] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:28.841943  2492 raft_consensus.cc:2749] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:12:28.841718  1175 leader_election.cc:336] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:28.844844  1173 leader_election.cc:336] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:28.845348  1173 leader_election.cc:304] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:28.846117  2498 raft_consensus.cc:2749] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:28.849246  2495 raft_consensus.cc:493] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:28.849781  2495 raft_consensus.cc:515] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:28.851956  2495 leader_election.cc:290] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:28.856415  1099 leader_election.cc:336] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:28.858834  1101 leader_election.cc:336] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:28.859298  1101 leader_election.cc:304] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:28.860054  2495 raft_consensus.cc:2749] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:28.869226  2498 raft_consensus.cc:493] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:28.869808  2498 raft_consensus.cc:515] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:28.873306  2498 leader_election.cc:290] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:28.877002  1175 leader_election.cc:336] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:28.878046  1173 leader_election.cc:336] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:28.878466  1173 leader_election.cc:304] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:28.879428  2492 raft_consensus.cc:2749] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:12:28.965003  1175 consensus_peers.cc:597] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:29.028484  1175 consensus_peers.cc:597] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:12:29.047072  2497 raft_consensus.cc:493] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:29.047602  2497 raft_consensus.cc:515] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
W20260709 13:12:29.048244  1175 consensus_peers.cc:597] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:12:29.050633  2497 leader_election.cc:290] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:29.053436  1101 leader_election.cc:336] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:29.056880  1099 leader_election.cc:336] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:29.057361  1099 leader_election.cc:304] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:29.058089  2497 raft_consensus.cc:2749] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:29.098582  2497 raft_consensus.cc:493] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:29.099166  2497 raft_consensus.cc:515] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
W20260709 13:12:29.105428  1101 consensus_peers.cc:597] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:12:29.108817  2497 leader_election.cc:290] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:29.114017  1099 leader_election.cc:336] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:29.116137  1101 consensus_peers.cc:597] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:29.118381  1101 leader_election.cc:336] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:29.118854  1101 leader_election.cc:304] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:29.121075  2500 raft_consensus.cc:2749] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:29.131896  2492 raft_consensus.cc:493] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:29.133040  2492 raft_consensus.cc:515] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:29.135392  2492 leader_election.cc:290] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:29.138794  1175 leader_election.cc:336] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:29.140620  1173 leader_election.cc:336] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:29.141067  1173 leader_election.cc:304] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:29.141824  2492 raft_consensus.cc:2749] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:29.142344  2498 raft_consensus.cc:493] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:29.142650  2500 raft_consensus.cc:493] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:29.142959  2498 raft_consensus.cc:515] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:29.143219  2500 raft_consensus.cc:515] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:29.145443  2500 leader_election.cc:290] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
I20260709 13:12:29.145541  2498 leader_election.cc:290] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:29.150836  1101 leader_election.cc:336] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:29.152172  1175 leader_election.cc:336] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:29.155786  1099 leader_election.cc:336] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:29.156319  1099 leader_election.cc:304] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:29.157050  2500 raft_consensus.cc:2749] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:12:29.161201  1173 leader_election.cc:336] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:29.161723  1173 leader_election.cc:304] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:29.162914  2492 raft_consensus.cc:2749] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:29.203469  2497 raft_consensus.cc:493] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:29.204034  2497 raft_consensus.cc:515] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } }
I20260709 13:12:29.206249  2497 leader_election.cc:290] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:29.211542  1099 leader_election.cc:336] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:29.213784  1101 leader_election.cc:336] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:29.214218  1101 leader_election.cc:304] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d473477839468bbdf000a21d5f9485; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:29.214962  2500 raft_consensus.cc:2749] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:12:29.242699  1175 consensus_peers.cc:597] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:12:29.297763  1175 consensus_peers.cc:597] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:12:29.389075  1101 consensus_peers.cc:597] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:12:29.394304  1101 consensus_peers.cc:597] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:12:29.412664  1101 consensus_peers.cc:597] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:12:29.789572 24113 tablet_server.cc:196] TabletServer@127.23.140.66:0 shutdown complete.
W20260709 13:12:29.805009  1173 consensus_peers.cc:597] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:29.904326 24113 tablet_server.cc:179] TabletServer@127.23.140.67:0 shutting down...
W20260709 13:12:29.925520  1173 consensus_peers.cc:597] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d -> Peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Couldn't send request to peer f9d473477839468bbdf000a21d5f9485. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:29.950030  1173 consensus_peers.cc:597] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d -> Peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Couldn't send request to peer f9d473477839468bbdf000a21d5f9485. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:29.950552  1175 consensus_peers.cc:597] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:12:29.952533 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:12:29.953428 24113 tablet_replica.cc:333] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:29.954519 24113 raft_consensus.cc:2243] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:12:29.955892 24113 raft_consensus.cc:2272] T f31edd3b8d2546a2a98a8a8d3d038f8b P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:29.958657 24113 tablet_replica.cc:333] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:29.959700 24113 raft_consensus.cc:2243] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:29.960269 24113 raft_consensus.cc:2272] T 09039e63d8db4e06b365f43533071272 P f9d473477839468bbdf000a21d5f9485 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:29.962759 24113 tablet_replica.cc:333] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:29.963373 24113 raft_consensus.cc:2243] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:29.964233 24113 raft_consensus.cc:2272] T 3e9125adfbbd414ca6b40457fc69ad84 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:29.966639 24113 tablet_replica.cc:333] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
W20260709 13:12:29.966629  1173 consensus_peers.cc:597] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d -> Peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Couldn't send request to peer f9d473477839468bbdf000a21d5f9485. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:29.968182 24113 raft_consensus.cc:2243] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:29.968927 24113 raft_consensus.cc:2272] T 75b503b2e43f4996b5f494ab0e65afb0 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:29.971947 24113 tablet_replica.cc:333] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:29.973006 24113 raft_consensus.cc:2243] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:29.973554 24113 raft_consensus.cc:2272] T 0867d95d9806478ba50d324c565446f6 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:29.977025 24113 tablet_replica.cc:333] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:29.978168 24113 raft_consensus.cc:2243] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:29.978760 24113 raft_consensus.cc:2272] T 9eec3e72036941a5b16c3a0157986a93 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:29.982378 24113 tablet_replica.cc:333] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:29.983130 24113 raft_consensus.cc:2243] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:29.983819 24113 raft_consensus.cc:2272] T 47f79cfcb8a04931bc19562b5d2c7754 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:29.983965  1173 consensus_peers.cc:597] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:29.986792 24113 tablet_replica.cc:333] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:29.987540 24113 raft_consensus.cc:2243] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:29.987983 24113 raft_consensus.cc:2272] T a0110653f11d472797f30dc22496cb12 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:29.990294 24113 tablet_replica.cc:333] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:29.991261 24113 raft_consensus.cc:2243] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:29.991825 24113 raft_consensus.cc:2272] T eb722a3043814db7a188e49c9861ceca P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:29.994787 24113 tablet_replica.cc:333] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:29.995488 24113 raft_consensus.cc:2243] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:29.996327 24113 raft_consensus.cc:2272] T fc8c8635d8e54625af9c95ecbcd5653b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:29.998327 24113 tablet_replica.cc:333] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:29.998951 24113 raft_consensus.cc:2243] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:29.999761 24113 raft_consensus.cc:2272] T 5d3595711f1e49dfb415fb2f6720e66d P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.002475 24113 tablet_replica.cc:333] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.003319 24113 raft_consensus.cc:2243] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:12:30.004360 24113 raft_consensus.cc:2272] T 9bbe4fae89eb429b989a7e1d87ab7dcf P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.007166 24113 tablet_replica.cc:333] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.007978 24113 raft_consensus.cc:2243] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:30.008976 24113 raft_consensus.cc:2272] T f267e3e7ffec446c9dfb1cfd18ace091 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.011540 24113 tablet_replica.cc:333] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.012419 24113 raft_consensus.cc:2243] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:30.013485 24113 raft_consensus.cc:2272] T d3fbed7be9014cf5984a19768b6269d4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.016405 24113 tablet_replica.cc:333] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.017257 24113 raft_consensus.cc:2243] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:30.017928 24113 raft_consensus.cc:2272] T 7f70fc428ff6447f8dca2e27ed05683a P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.020798 24113 tablet_replica.cc:333] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
W20260709 13:12:30.019549  1173 consensus_peers.cc:597] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d -> Peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Couldn't send request to peer f9d473477839468bbdf000a21d5f9485. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:30.021874 24113 raft_consensus.cc:2243] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:30.023159 24113 raft_consensus.cc:2272] T 545027a75bf749969ab288105031caff P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.026379 24113 tablet_replica.cc:333] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.027402 24113 raft_consensus.cc:2243] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 5 LEADER]: Raft consensus shutting down.
I20260709 13:12:30.028661 24113 raft_consensus.cc:2272] T 4c162b2e111b472b88cd037925284de7 P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.031853 24113 tablet_replica.cc:333] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.032719 24113 raft_consensus.cc:2243] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:30.033481 24113 raft_consensus.cc:2272] T 120ba483d3f1448e967c504d55e6d5f3 P f9d473477839468bbdf000a21d5f9485 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.036916 24113 tablet_replica.cc:333] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.037947 24113 raft_consensus.cc:2243] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:30.038646 24113 raft_consensus.cc:2272] T f0afcfb9359a421b85fabc92b7733d9b P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.041433 24113 tablet_replica.cc:333] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
W20260709 13:12:30.041985  1173 consensus_peers.cc:597] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:12:30.042447 24113 raft_consensus.cc:2243] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:30.043236 24113 raft_consensus.cc:2272] T b89f56a91e004f408b36c36d7aee0076 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:30.043149  1173 consensus_peers.cc:597] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d -> Peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Couldn't send request to peer f9d473477839468bbdf000a21d5f9485. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:30.046286 24113 tablet_replica.cc:333] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.047281 24113 raft_consensus.cc:2243] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:30.048346 24113 raft_consensus.cc:2272] T 7780e3bac1204186aefaa6f574717fdb P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:30.050727  1173 consensus_peers.cc:597] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d -> Peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Couldn't send request to peer f9d473477839468bbdf000a21d5f9485. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:30.050911 24113 tablet_replica.cc:333] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.051671 24113 raft_consensus.cc:2243] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:30.052142 24113 raft_consensus.cc:2272] T 160ad44adbce4b7f97094b248fd05415 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.054154 24113 tablet_replica.cc:333] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.054665 24113 raft_consensus.cc:2243] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:30.055034 24113 raft_consensus.cc:2272] T a260eb172eba4c6db7c0d0fab6a7f7d1 P f9d473477839468bbdf000a21d5f9485 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.056833 24113 tablet_replica.cc:333] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.057487 24113 raft_consensus.cc:2243] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:30.057914 24113 raft_consensus.cc:2272] T 33ceafb69dc74411898ea99b377d1032 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.060523 24113 tablet_replica.cc:333] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.061283 24113 raft_consensus.cc:2243] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:30.062357 24113 raft_consensus.cc:2272] T e8aead26aca7435abf7d4af1ab2321af P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.064456 24113 tablet_replica.cc:333] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.064972 24113 raft_consensus.cc:2243] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:30.065709 24113 raft_consensus.cc:2272] T 4ddfc34a205c43ad81653b5000cbf394 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.067553 24113 tablet_replica.cc:333] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.068207 24113 raft_consensus.cc:2243] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:30.069073 24113 raft_consensus.cc:2272] T 29dde48ef51c4666a28b97181ec2ee21 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.071100 24113 tablet_replica.cc:333] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.071651 24113 raft_consensus.cc:2243] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:30.072228 24113 raft_consensus.cc:2272] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.074826 24113 tablet_replica.cc:333] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.075410 24113 raft_consensus.cc:2243] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:30.075908 24113 raft_consensus.cc:2272] T ff504aa514be43da8b1aac12ab550daf P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.077970 24113 tablet_replica.cc:333] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.078456 24113 raft_consensus.cc:2243] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:30.078845 24113 raft_consensus.cc:2272] T fa73a73b72b2499fa5219cbf455099b4 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.080642 24113 tablet_replica.cc:333] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.081187 24113 raft_consensus.cc:2243] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:30.081943 24113 raft_consensus.cc:2272] T 161b9f09b3fb47febfabacf867781986 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.084038 24113 tablet_replica.cc:333] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.084760 24113 raft_consensus.cc:2243] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:30.085747 24113 raft_consensus.cc:2272] T 6de2d5263c9a4dfebfe53a9eb7bc0bba P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.088389 24113 tablet_replica.cc:333] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.089088 24113 raft_consensus.cc:2243] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:30.089658 24113 raft_consensus.cc:2272] T 5cf962b21fa14634af6e810ca29f1bc7 P f9d473477839468bbdf000a21d5f9485 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.092226 24113 tablet_replica.cc:333] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.092936 24113 raft_consensus.cc:2243] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:30.093788 24113 raft_consensus.cc:2272] T f95cbeb7360c4c61b442e75e35d93e73 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.095917 24113 tablet_replica.cc:333] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.096429 24113 raft_consensus.cc:2243] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:30.096840 24113 raft_consensus.cc:2272] T a81738ce3431443d9e332eb931ec3c0b P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.098783 24113 tablet_replica.cc:333] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.099295 24113 raft_consensus.cc:2243] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:30.099700 24113 raft_consensus.cc:2272] T 69355f8224b14b23be2cf00be247206c P f9d473477839468bbdf000a21d5f9485 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.101589 24113 tablet_replica.cc:333] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.102103 24113 raft_consensus.cc:2243] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:30.102471 24113 raft_consensus.cc:2272] T 3b917eb6128a4341b3e3288130a65db1 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.104569 24113 tablet_replica.cc:333] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.105229 24113 raft_consensus.cc:2243] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:30.105726 24113 raft_consensus.cc:2272] T c216d860ee834d4b8f77c8c39f0dba94 P f9d473477839468bbdf000a21d5f9485 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.108146 24113 tablet_replica.cc:333] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.108820 24113 raft_consensus.cc:2243] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:30.109777 24113 raft_consensus.cc:2272] T 219d22690fa145adb5b04192c1fc16bd P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.112257 24113 tablet_replica.cc:333] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.112757 24113 raft_consensus.cc:2243] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:30.113520 24113 raft_consensus.cc:2272] T 483b49446c4e4a23b780921d7fabfbdf P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.115764 24113 tablet_replica.cc:333] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.116489 24113 raft_consensus.cc:2243] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:30.117419 24113 raft_consensus.cc:2272] T 90e13f7e16774e68bca79e3f141ed6bc P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:30.118566  1173 consensus_peers.cc:597] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d -> Peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Couldn't send request to peer f9d473477839468bbdf000a21d5f9485. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:30.119726 24113 tablet_replica.cc:333] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.120432 24113 raft_consensus.cc:2243] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:30.120870 24113 raft_consensus.cc:2272] T e038f78f3e9540399aeda1f38cbea3fd P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.123028 24113 tablet_replica.cc:333] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.123508 24113 raft_consensus.cc:2243] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:30.124382 24113 raft_consensus.cc:2272] T 39dcc0888dc64bff9ce20b35e4bc8f89 P f9d473477839468bbdf000a21d5f9485 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:30.126385 24113 tablet_replica.cc:333] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485: stopping tablet replica
I20260709 13:12:30.126858 24113 raft_consensus.cc:2243] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:30.127607 24113 raft_consensus.cc:2272] T 240eb8c24e0142c8a23a0afdd5dccda6 P f9d473477839468bbdf000a21d5f9485 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:30.134009  1173 consensus_peers.cc:597] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d -> Peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Couldn't send request to peer f9d473477839468bbdf000a21d5f9485. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:30.158325  1173 consensus_peers.cc:597] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d -> Peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Couldn't send request to peer f9d473477839468bbdf000a21d5f9485. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:30.174170  1173 consensus_peers.cc:597] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d -> Peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Couldn't send request to peer f9d473477839468bbdf000a21d5f9485. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:30.227981  1173 consensus_peers.cc:597] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d -> Peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Couldn't send request to peer f9d473477839468bbdf000a21d5f9485. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.67:33869: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:30.231604  1173 consensus_peers.cc:597] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:30.261308  1173 consensus_peers.cc:597] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d -> Peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Couldn't send request to peer f9d473477839468bbdf000a21d5f9485. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.67:33869: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:30.295253  1173 consensus_peers.cc:597] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d -> Peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Couldn't send request to peer f9d473477839468bbdf000a21d5f9485. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.67:33869: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:30.319792  1173 consensus_peers.cc:597] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d -> Peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Couldn't send request to peer f9d473477839468bbdf000a21d5f9485. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.67:33869: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:30.373426  2498 raft_consensus.cc:493] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:30.374018  2498 raft_consensus.cc:515] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:30.376281  2498 leader_election.cc:290] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:30.379734  1173 leader_election.cc:336] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:30.380175  1175 leader_election.cc:336] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:30.380615  1175 leader_election.cc:304] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:30.381333  2498 raft_consensus.cc:2749] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:30.392166  2492 raft_consensus.cc:493] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:30.392670  2492 raft_consensus.cc:515] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:30.394687  2492 leader_election.cc:290] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:30.397927  1173 leader_election.cc:336] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:30.398500  1175 leader_election.cc:336] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:30.398878  1175 leader_election.cc:304] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:30.399569  2492 raft_consensus.cc:2749] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:12:30.477201  1173 consensus_peers.cc:597] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:30.543885  1173 consensus_peers.cc:597] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:30.713449  1173 consensus_peers.cc:597] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:12:30.714994  2498 raft_consensus.cc:493] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:30.715554  2498 raft_consensus.cc:515] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:30.717573  2498 leader_election.cc:290] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:30.722134  1173 leader_election.cc:336] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:30.723213  1175 leader_election.cc:336] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:30.723628  1175 leader_election.cc:304] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:30.724300  2498 raft_consensus.cc:2749] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:12:30.724478  1175 consensus_peers.cc:597] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:12:30.726676  1173 consensus_peers.cc:597] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:30.737102  1175 consensus_peers.cc:597] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:12:30.767767  1173 consensus_peers.cc:597] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:30.802929  1173 consensus_peers.cc:597] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:12:30.833323  2498 raft_consensus.cc:493] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:30.833931  2498 raft_consensus.cc:515] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:30.838071  2498 leader_election.cc:290] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:30.847157  1175 leader_election.cc:336] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:30.847757  1175 consensus_peers.cc:597] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:12:30.852447  1173 leader_election.cc:336] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:30.852779  1173 leader_election.cc:304] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:30.853456  2498 raft_consensus.cc:2749] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:12:30.874212  1173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.140.67:33869: connect: Connection refused (error 111) [suppressed 611 similar messages]
W20260709 13:12:30.892527  1173 consensus_peers.cc:597] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:30.922045  1173 consensus_peers.cc:597] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d -> Peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Couldn't send request to peer 9c45b6d917734835a8a37b0f4c789ae6. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:12:30.984458  2492 raft_consensus.cc:493] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:30.985052  2492 raft_consensus.cc:515] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:30.987372  2492 leader_election.cc:290] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
W20260709 13:12:30.991276  1173 leader_election.cc:336] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:30.991931  1173 leader_election.cc:336] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Network error: Client connection negotiation failed: client connection to 127.23.140.67:33869: connect: Connection refused (error 111)
I20260709 13:12:30.992254  1173 leader_election.cc:304] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:30.992830  2492 raft_consensus.cc:2749] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:12:31.002322  1175 consensus_peers.cc:597] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:12:31.008874  2492 raft_consensus.cc:493] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:31.009573  2492 raft_consensus.cc:515] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:31.011853  2492 leader_election.cc:290] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
W20260709 13:12:31.015520  1173 leader_election.cc:336] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Network error: Client connection negotiation failed: client connection to 127.23.140.67:33869: connect: Connection refused (error 111)
W20260709 13:12:31.016032  1175 leader_election.cc:336] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:31.016448  1175 leader_election.cc:304] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:31.017208  2498 raft_consensus.cc:2749] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:31.027392  2498 raft_consensus.cc:493] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:31.027804  2498 raft_consensus.cc:515] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:31.029476  2498 leader_election.cc:290] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:31.032722  1175 leader_election.cc:336] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
W20260709 13:12:31.033893  1173 leader_election.cc:336] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:31.034309  1173 leader_election.cc:304] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:31.034993  2498 raft_consensus.cc:2749] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:31.045967  2492 raft_consensus.cc:493] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:31.046525  2492 raft_consensus.cc:515] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:31.048380  2492 leader_election.cc:290] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
W20260709 13:12:31.052443  1173 leader_election.cc:336] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:31.053118  1173 leader_election.cc:336] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Network error: Client connection negotiation failed: client connection to 127.23.140.67:33869: connect: Connection refused (error 111)
I20260709 13:12:31.053531  1173 leader_election.cc:304] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:31.054216  2492 raft_consensus.cc:2749] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:31.071662  2492 raft_consensus.cc:493] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:31.072135  2492 raft_consensus.cc:515] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:31.073958  2492 leader_election.cc:290] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
W20260709 13:12:31.078639  1173 leader_election.cc:336] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Network error: Client connection negotiation failed: client connection to 127.23.140.67:33869: connect: Connection refused (error 111)
W20260709 13:12:31.079245  1173 leader_election.cc:336] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:31.079555  1173 leader_election.cc:304] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:31.080120  2492 raft_consensus.cc:2749] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:31.084324  2492 raft_consensus.cc:493] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:31.084723  2492 raft_consensus.cc:515] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:31.086602  2492 leader_election.cc:290] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:31.089735  2498 raft_consensus.cc:493] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:31.090233  2498 raft_consensus.cc:515] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
W20260709 13:12:31.092240  1175 leader_election.cc:336] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:31.092379  2498 leader_election.cc:290] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
W20260709 13:12:31.096403  1173 leader_election.cc:336] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Network error: Client connection negotiation failed: client connection to 127.23.140.67:33869: connect: Connection refused (error 111)
I20260709 13:12:31.096746  1173 leader_election.cc:304] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:31.097404  2498 raft_consensus.cc:2749] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:12:31.099385  1173 leader_election.cc:336] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:31.099953  1173 leader_election.cc:336] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Network error: Client connection negotiation failed: client connection to 127.23.140.67:33869: connect: Connection refused (error 111)
I20260709 13:12:31.100258  1173 leader_election.cc:304] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:31.100793  2498 raft_consensus.cc:2749] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:31.101318  2492 raft_consensus.cc:493] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:31.101897  2492 raft_consensus.cc:515] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:31.103957  2492 leader_election.cc:290] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
W20260709 13:12:31.107715  1173 leader_election.cc:336] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Network error: Client connection negotiation failed: client connection to 127.23.140.67:33869: connect: Connection refused (error 111)
W20260709 13:12:31.108203  1175 leader_election.cc:336] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:31.108541  1175 leader_election.cc:304] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:31.109071  2492 raft_consensus.cc:2749] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:12:31.113075  1175 consensus_peers.cc:597] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:12:31.161370  2492 raft_consensus.cc:493] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:31.161900  2492 raft_consensus.cc:515] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:31.163913  2492 leader_election.cc:290] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
W20260709 13:12:31.167265  1173 leader_election.cc:336] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:31.167976  1173 leader_election.cc:336] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Network error: Client connection negotiation failed: client connection to 127.23.140.67:33869: connect: Connection refused (error 111)
I20260709 13:12:31.168344  1173 leader_election.cc:304] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:31.168915  2492 raft_consensus.cc:2749] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:31.178748 24113 tablet_server.cc:196] TabletServer@127.23.140.67:0 shutdown complete.
I20260709 13:12:31.178750  2492 raft_consensus.cc:493] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:31.179520  2492 raft_consensus.cc:515] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:31.181587  2492 leader_election.cc:290] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:31.186024  1173 leader_election.cc:336] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:31.187206  1175 leader_election.cc:336] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:31.187625  1175 leader_election.cc:304] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:31.188319  2498 raft_consensus.cc:2749] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:12:31.190347  1175 consensus_peers.cc:597] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:12:31.197417  1175 consensus_peers.cc:597] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:12:31.227797  1175 consensus_peers.cc:597] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:12:31.245164  2498 raft_consensus.cc:493] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:31.245749  2498 raft_consensus.cc:515] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:31.247854  2498 leader_election.cc:290] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:31.248441  2498 raft_consensus.cc:493] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c45b6d917734835a8a37b0f4c789ae6)
I20260709 13:12:31.248935  2498 raft_consensus.cc:515] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } } peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } }
I20260709 13:12:31.251225  2498 leader_election.cc:290] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721), 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097)
W20260709 13:12:31.253734  1173 leader_election.cc:336] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:31.254750  1173 leader_election.cc:336] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Network error: Client connection negotiation failed: client connection to 127.23.140.67:33869: connect: Connection refused (error 111)
I20260709 13:12:31.255119  1173 leader_election.cc:304] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
W20260709 13:12:31.255944  1175 leader_election.cc:336] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:31.256858  2498 raft_consensus.cc:2749] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:12:31.258306  1173 leader_election.cc:336] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:31.258648  1173 leader_election.cc:304] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, 9c45b6d917734835a8a37b0f4c789ae6
I20260709 13:12:31.259258  2498 raft_consensus.cc:2749] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:12:31.260468 24113 tablet_server.cc:179] TabletServer@127.23.140.68:0 shutting down...
I20260709 13:12:31.265352  2498 raft_consensus.cc:493] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:31.265856  2498 raft_consensus.cc:515] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:31.267441  2498 leader_election.cc:290] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
W20260709 13:12:31.271075  1173 leader_election.cc:336] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
W20260709 13:12:31.271843  1173 leader_election.cc:336] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Network error: Client connection negotiation failed: client connection to 127.23.140.67:33869: connect: Connection refused (error 111)
I20260709 13:12:31.272269  1173 leader_election.cc:304] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:31.273010  2498 raft_consensus.cc:2749] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:12:31.287889  1175 consensus_peers.cc:597] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d -> Peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Couldn't send request to peer 294145900dd3496cb4a1d722613e1558. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:12:31.306090  2498 raft_consensus.cc:493] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:31.306802  2498 raft_consensus.cc:515] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b6d917734835a8a37b0f4c789ae6" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 44097 } } peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } }
I20260709 13:12:31.308974  2498 leader_election.cc:290] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097), f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869)
I20260709 13:12:31.310184 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:12:31.311286 24113 tablet_replica.cc:333] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.312158 24113 raft_consensus.cc:2243] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Raft consensus shutting down.
W20260709 13:12:31.313225  1173 leader_election.cc:336] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c45b6d917734835a8a37b0f4c789ae6 (127.23.140.66:44097): Network error: Client connection negotiation failed: client connection to 127.23.140.66:44097: connect: Connection refused (error 111)
I20260709 13:12:31.313222 24113 raft_consensus.cc:2272] T ff504aa514be43da8b1aac12ab550daf P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:31.313876  1173 leader_election.cc:336] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Network error: Client connection negotiation failed: client connection to 127.23.140.67:33869: connect: Connection refused (error 111)
I20260709 13:12:31.314199  1173 leader_election.cc:304] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 9c45b6d917734835a8a37b0f4c789ae6, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:31.314761  2498 raft_consensus.cc:2749] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:31.315862 24113 tablet_replica.cc:333] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.316514 24113 raft_consensus.cc:2243] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.316942 24113 raft_consensus.cc:2272] T 21783150bb0b4141be797e841c3f46fa P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.318928 24113 tablet_replica.cc:333] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.319515 24113 raft_consensus.cc:2243] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.319932 24113 raft_consensus.cc:2272] T 8acf44a40c084675a4f076a7acbda99d P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.321770 24113 tablet_replica.cc:333] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.322392 24113 raft_consensus.cc:2243] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.323175 24113 raft_consensus.cc:2272] T 75b503b2e43f4996b5f494ab0e65afb0 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.324956 24113 tablet_replica.cc:333] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.325500 24113 raft_consensus.cc:2243] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 6 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.326267 24113 raft_consensus.cc:2272] T a260eb172eba4c6db7c0d0fab6a7f7d1 P 862b41697c74421fbac82c10c245434d [term 6 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.327706  2498 raft_consensus.cc:493] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:31.328238 24113 tablet_replica.cc:333] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.328084  2498 raft_consensus.cc:515] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:31.328899 24113 raft_consensus.cc:2243] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.329687 24113 raft_consensus.cc:2272] T 9b75f9192e7246d2a560fd091fb68606 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.329973  2498 leader_election.cc:290] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:31.332525 24113 tablet_replica.cc:333] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.333381 24113 raft_consensus.cc:2243] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Raft consensus shutting down.
W20260709 13:12:31.333559  1173 leader_election.cc:336] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Network error: Client connection negotiation failed: client connection to 127.23.140.67:33869: connect: Connection refused (error 111)
I20260709 13:12:31.334344 24113 raft_consensus.cc:2272] T 9eec3e72036941a5b16c3a0157986a93 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:31.334369  1175 leader_election.cc:336] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:31.335002  1175 leader_election.cc:304] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:31.335557  2498 raft_consensus.cc:2749] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:31.336889 24113 tablet_replica.cc:333] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.337625 24113 raft_consensus.cc:2243] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.338593 24113 raft_consensus.cc:2272] T a0110653f11d472797f30dc22496cb12 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.341184 24113 tablet_replica.cc:333] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.341928 24113 raft_consensus.cc:2243] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.342468 24113 raft_consensus.cc:2272] T 7780e3bac1204186aefaa6f574717fdb P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.344364 24113 tablet_replica.cc:333] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.344898 24113 raft_consensus.cc:2243] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 6 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.345682 24113 raft_consensus.cc:2272] T 6aa365e967a941129ed031939342b1d8 P 862b41697c74421fbac82c10c245434d [term 6 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.347507 24113 tablet_replica.cc:333] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.348021 24113 raft_consensus.cc:2243] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.348428 24113 raft_consensus.cc:2272] T 9bbe4fae89eb429b989a7e1d87ab7dcf P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.350131 24113 tablet_replica.cc:333] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.350759 24113 raft_consensus.cc:2243] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.351176 24113 raft_consensus.cc:2272] T f267e3e7ffec446c9dfb1cfd18ace091 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.353027 24113 tablet_replica.cc:333] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.353581 24113 raft_consensus.cc:2243] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.354290 24113 raft_consensus.cc:2272] T 11adc51543f7421cb3140972d9f0a424 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.356077 24113 tablet_replica.cc:333] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.356617 24113 raft_consensus.cc:2243] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.357067 24113 raft_consensus.cc:2272] T d3fbed7be9014cf5984a19768b6269d4 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.358963 24113 tablet_replica.cc:333] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.359478 24113 raft_consensus.cc:2243] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.360198 24113 raft_consensus.cc:2272] T e038f78f3e9540399aeda1f38cbea3fd P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.362145 24113 tablet_replica.cc:333] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.362679 24113 raft_consensus.cc:2243] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.363389 24113 raft_consensus.cc:2272] T 7f70fc428ff6447f8dca2e27ed05683a P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.365219 24113 tablet_replica.cc:333] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.365773 24113 raft_consensus.cc:2243] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.366168 24113 raft_consensus.cc:2272] T 545027a75bf749969ab288105031caff P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.367825 24113 tablet_replica.cc:333] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.368350 24113 raft_consensus.cc:2243] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.368749 24113 raft_consensus.cc:2272] T f6f69ac309b6488b81b409d785a2d465 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.370536 24113 tablet_replica.cc:333] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.371040 24113 raft_consensus.cc:2243] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.371423 24113 raft_consensus.cc:2272] T e8aead26aca7435abf7d4af1ab2321af P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.373543 24113 tablet_replica.cc:333] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.374204 24113 raft_consensus.cc:2243] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.374732 24113 raft_consensus.cc:2272] T 4ddfc34a205c43ad81653b5000cbf394 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.377064 24113 tablet_replica.cc:333] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.377662 24113 raft_consensus.cc:2243] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.378046 24113 raft_consensus.cc:2272] T 29dde48ef51c4666a28b97181ec2ee21 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.379905 24113 tablet_replica.cc:333] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.380424 24113 raft_consensus.cc:2243] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.381207 24113 raft_consensus.cc:2272] T 05ca8b4123ce4ab3a2e36c3b422e8af5 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.382905 24113 tablet_replica.cc:333] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.383422 24113 raft_consensus.cc:2243] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.384213 24113 raft_consensus.cc:2272] T 8f1971394c7d40bfa460d909b6f47438 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.386225 24113 tablet_replica.cc:333] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.386858 24113 raft_consensus.cc:2243] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.387544 24113 raft_consensus.cc:2272] T 91c2241524a24f7e911295c67f13bd26 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.389412 24113 tablet_replica.cc:333] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.389937 24113 raft_consensus.cc:2243] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.390658 24113 raft_consensus.cc:2272] T f0afcfb9359a421b85fabc92b7733d9b P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.392565 24113 tablet_replica.cc:333] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.393051 24113 raft_consensus.cc:2243] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.393821 24113 raft_consensus.cc:2272] T b89f56a91e004f408b36c36d7aee0076 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.395638 24113 tablet_replica.cc:333] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.396127 24113 raft_consensus.cc:2243] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.396806 24113 raft_consensus.cc:2272] T 7526fd343f7d40528597242eb03386f8 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.398460 24113 tablet_replica.cc:333] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.399042 24113 raft_consensus.cc:2243] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.399477 24113 raft_consensus.cc:2272] T f50253bf21654ca6937a357eb070beda P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.401296 24113 tablet_replica.cc:333] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.401799 24113 raft_consensus.cc:2243] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 6 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.402525 24113 raft_consensus.cc:2272] T 120ba483d3f1448e967c504d55e6d5f3 P 862b41697c74421fbac82c10c245434d [term 6 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.404233 24113 tablet_replica.cc:333] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.404733 24113 raft_consensus.cc:2243] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.405151 24113 raft_consensus.cc:2272] T 12df324e933f422c9a8ff5d37eb7d4ef P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.405601  2498 raft_consensus.cc:493] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f9d473477839468bbdf000a21d5f9485)
I20260709 13:12:31.406076  2498 raft_consensus.cc:515] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d473477839468bbdf000a21d5f9485" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 33869 } } peers { permanent_uuid: "862b41697c74421fbac82c10c245434d" member_type: VOTER last_known_addr { host: "127.23.140.68" port: 34477 } } peers { permanent_uuid: "294145900dd3496cb4a1d722613e1558" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 43721 } }
I20260709 13:12:31.407572 24113 tablet_replica.cc:333] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.408486 24113 raft_consensus.cc:2243] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.408277  2498 leader_election.cc:290] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869), 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721)
I20260709 13:12:31.409502 24113 raft_consensus.cc:2272] T a81738ce3431443d9e332eb931ec3c0b P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:31.411801  1173 leader_election.cc:336] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f9d473477839468bbdf000a21d5f9485 (127.23.140.67:33869): Network error: Client connection negotiation failed: client connection to 127.23.140.67:33869: connect: Connection refused (error 111)
I20260709 13:12:31.412014 24113 tablet_replica.cc:333] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
W20260709 13:12:31.412858  1175 leader_election.cc:336] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 294145900dd3496cb4a1d722613e1558 (127.23.140.65:43721): Network error: Client connection negotiation failed: client connection to 127.23.140.65:43721: connect: Connection refused (error 111)
I20260709 13:12:31.412868 24113 raft_consensus.cc:2243] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.413224  1175 leader_election.cc:304] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 862b41697c74421fbac82c10c245434d; no voters: 294145900dd3496cb4a1d722613e1558, f9d473477839468bbdf000a21d5f9485
I20260709 13:12:31.413533 24113 raft_consensus.cc:2272] T cae53ca608e74146a2f89675ddca72a6 P 862b41697c74421fbac82c10c245434d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.413805  2498 raft_consensus.cc:2749] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:12:31.415715 24113 tablet_replica.cc:333] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.416252 24113 raft_consensus.cc:2243] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.416635 24113 raft_consensus.cc:2272] T fc8c8635d8e54625af9c95ecbcd5653b P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.418362 24113 tablet_replica.cc:333] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.418803 24113 raft_consensus.cc:2243] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.419139 24113 raft_consensus.cc:2272] T 5d3595711f1e49dfb415fb2f6720e66d P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.420847 24113 tablet_replica.cc:333] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.421350 24113 raft_consensus.cc:2243] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.421744 24113 raft_consensus.cc:2272] T 219d22690fa145adb5b04192c1fc16bd P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.423568 24113 tablet_replica.cc:333] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.424080 24113 raft_consensus.cc:2243] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.424829 24113 raft_consensus.cc:2272] T 11cd8a46e76f4bc685aa01885b27cc57 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.426694 24113 tablet_replica.cc:333] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.427217 24113 raft_consensus.cc:2243] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.427644 24113 raft_consensus.cc:2272] T 39dcc0888dc64bff9ce20b35e4bc8f89 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.429456 24113 tablet_replica.cc:333] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.429939 24113 raft_consensus.cc:2243] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.430366 24113 raft_consensus.cc:2272] T 35cd7c6206834f5dabfca17b22a0605a P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.432229 24113 tablet_replica.cc:333] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.432767 24113 raft_consensus.cc:2243] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 5 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.433588 24113 raft_consensus.cc:2272] T 69355f8224b14b23be2cf00be247206c P 862b41697c74421fbac82c10c245434d [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.435509 24113 tablet_replica.cc:333] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.435986 24113 raft_consensus.cc:2243] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.436404 24113 raft_consensus.cc:2272] T 3e9125adfbbd414ca6b40457fc69ad84 P 862b41697c74421fbac82c10c245434d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.438189 24113 tablet_replica.cc:333] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.438668 24113 raft_consensus.cc:2243] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.439051 24113 raft_consensus.cc:2272] T c2312d768cbf4b3495d5b1574b0b6eb5 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.440856 24113 tablet_replica.cc:333] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.441363 24113 raft_consensus.cc:2243] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.441761 24113 raft_consensus.cc:2272] T f95cbeb7360c4c61b442e75e35d93e73 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.443506 24113 tablet_replica.cc:333] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.443986 24113 raft_consensus.cc:2243] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.444698 24113 raft_consensus.cc:2272] T c216d860ee834d4b8f77c8c39f0dba94 P 862b41697c74421fbac82c10c245434d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.446468 24113 tablet_replica.cc:333] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d: stopping tablet replica
I20260709 13:12:31.446910 24113 raft_consensus.cc:2243] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:31.447295 24113 raft_consensus.cc:2272] T 240eb8c24e0142c8a23a0afdd5dccda6 P 862b41697c74421fbac82c10c245434d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.509986 24113 tablet_server.cc:196] TabletServer@127.23.140.68:0 shutdown complete.
I20260709 13:12:31.572438 24113 master.cc:562] Master@127.23.140.126:33289 shutting down...
I20260709 13:12:31.600198 24113 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:31.600787 24113 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:31.601162 24113 tablet_replica.cc:333] T 00000000000000000000000000000000 P 154f6da223eb4abba85302feda9ead4f: stopping tablet replica
I20260709 13:12:31.624912 24113 master.cc:584] Master@127.23.140.126:33289 shutdown complete.
[       OK ] LeaderRebalancerTest.RestartTserver (42518 ms)
[ RUN      ] LeaderRebalancerTest.TestMaintenanceMode
I20260709 13:12:31.716711 24113 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.140.126:37955
I20260709 13:12:31.718183 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:12:31.723822  2505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:31.723832  2504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:31.724650  2507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:12:31.726286 24113 server_base.cc:1061] running on GCE node
I20260709 13:12:31.727429 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:12:31.727648 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:12:31.727806 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602751727792 us; error 0 us; skew 500 ppm
I20260709 13:12:31.728323 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:12:31.730816 24113 webserver.cc:533] Webserver started at http://127.23.140.126:33965/ using document root <none> and password file <none>
I20260709 13:12:31.731310 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:12:31.731510 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:12:31.731784 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:12:31.732852 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/master-0-root/instance:
uuid: "b5f8c43a369f4475b21ff07bce488069"
format_stamp: "Formatted at 2026-07-09 13:12:31 on dist-test-slave-k73n"
I20260709 13:12:31.737468 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.000s
I20260709 13:12:31.740689  2512 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:31.741429 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.001s
I20260709 13:12:31.741719 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/master-0-root
uuid: "b5f8c43a369f4475b21ff07bce488069"
format_stamp: "Formatted at 2026-07-09 13:12:31 on dist-test-slave-k73n"
I20260709 13:12:31.741961 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:12:31.772121 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:12:31.773340 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:12:31.810410 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.126:37955
I20260709 13:12:31.810505  2563 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.126:37955 every 8 connection(s)
I20260709 13:12:31.814224  2564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:31.824494  2564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069: Bootstrap starting.
I20260709 13:12:31.829185  2564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:31.833213  2564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069: No bootstrap required, opened a new log
I20260709 13:12:31.835327  2564 raft_consensus.cc:359] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f8c43a369f4475b21ff07bce488069" member_type: VOTER }
I20260709 13:12:31.835678  2564 raft_consensus.cc:385] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:31.835861  2564 raft_consensus.cc:740] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5f8c43a369f4475b21ff07bce488069, State: Initialized, Role: FOLLOWER
I20260709 13:12:31.836352  2564 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f8c43a369f4475b21ff07bce488069" member_type: VOTER }
I20260709 13:12:31.836750  2564 raft_consensus.cc:399] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:12:31.836954  2564 raft_consensus.cc:493] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:12:31.837194  2564 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:31.841596  2564 raft_consensus.cc:515] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f8c43a369f4475b21ff07bce488069" member_type: VOTER }
I20260709 13:12:31.842041  2564 leader_election.cc:304] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5f8c43a369f4475b21ff07bce488069; no voters: 
I20260709 13:12:31.843214  2564 leader_election.cc:290] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:12:31.843487  2567 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:31.844895  2567 raft_consensus.cc:697] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [term 1 LEADER]: Becoming Leader. State: Replica: b5f8c43a369f4475b21ff07bce488069, State: Running, Role: LEADER
I20260709 13:12:31.845592  2567 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f8c43a369f4475b21ff07bce488069" member_type: VOTER }
I20260709 13:12:31.846292  2564 sys_catalog.cc:565] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:12:31.849733  2569 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5f8c43a369f4475b21ff07bce488069. Latest consensus state: current_term: 1 leader_uuid: "b5f8c43a369f4475b21ff07bce488069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f8c43a369f4475b21ff07bce488069" member_type: VOTER } }
I20260709 13:12:31.849817  2568 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5f8c43a369f4475b21ff07bce488069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f8c43a369f4475b21ff07bce488069" member_type: VOTER } }
I20260709 13:12:31.850576  2569 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [sys.catalog]: This master's current role is: LEADER
I20260709 13:12:31.850903  2568 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [sys.catalog]: This master's current role is: LEADER
I20260709 13:12:31.858970  2573 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:12:31.866869  2573 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:12:31.867889 24113 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:12:31.876176  2573 catalog_manager.cc:1383] Generated new cluster ID: 3268a81420e34c2fb6dd77cd7c6da789
I20260709 13:12:31.876479  2573 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:12:31.906708  2573 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:12:31.907979  2573 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:12:31.931257  2573 catalog_manager.cc:6070] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069: Generated new TSK 0
I20260709 13:12:31.931918  2573 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:12:31.998764 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:12:32.004822  2585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:32.005821  2586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:12:32.010144 24113 server_base.cc:1061] running on GCE node
W20260709 13:12:32.010174  2588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:12:32.011358 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:12:32.011567 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:12:32.011740 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602752011721 us; error 0 us; skew 500 ppm
I20260709 13:12:32.012285 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:12:32.014564 24113 webserver.cc:533] Webserver started at http://127.23.140.65:39607/ using document root <none> and password file <none>
I20260709 13:12:32.015056 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:12:32.015257 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:12:32.015518 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:12:32.016551 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-0-root/instance:
uuid: "fcffaa88967548fdb390c736bf217b09"
format_stamp: "Formatted at 2026-07-09 13:12:32 on dist-test-slave-k73n"
I20260709 13:12:32.020939 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 13:12:32.024137  2593 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:32.024914 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 13:12:32.025336 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-0-root
uuid: "fcffaa88967548fdb390c736bf217b09"
format_stamp: "Formatted at 2026-07-09 13:12:32 on dist-test-slave-k73n"
I20260709 13:12:32.025666 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:12:32.051198 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:12:32.052456 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:12:32.053982 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:12:32.056348 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:12:32.056582 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:32.056818 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:12:32.057015 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:32.096607 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.65:37697
I20260709 13:12:32.096728  2655 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.65:37697 every 8 connection(s)
I20260709 13:12:32.101280 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:12:32.108934  2660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:32.110059  2661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:32.113867  2663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:12:32.114112 24113 server_base.cc:1061] running on GCE node
I20260709 13:12:32.114854  2656 heartbeater.cc:344] Connected to a master server at 127.23.140.126:37955
I20260709 13:12:32.115270  2656 heartbeater.cc:461] Registering TS with master...
I20260709 13:12:32.115621 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:12:32.115871 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:12:32.116046 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602752116014 us; error 0 us; skew 500 ppm
I20260709 13:12:32.116055  2656 heartbeater.cc:507] Master 127.23.140.126:37955 requested a full tablet report, sending...
I20260709 13:12:32.116729 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:12:32.118570  2529 ts_manager.cc:194] Registered new tserver with Master: fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697)
I20260709 13:12:32.120250  2529 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48432
I20260709 13:12:32.121524 24113 webserver.cc:533] Webserver started at http://127.23.140.66:39889/ using document root <none> and password file <none>
I20260709 13:12:32.122098 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:12:32.122295 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:12:32.122570 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:12:32.123641 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-1-root/instance:
uuid: "bf69c09edabf4c568ab360ffa1755c90"
format_stamp: "Formatted at 2026-07-09 13:12:32 on dist-test-slave-k73n"
I20260709 13:12:32.128270 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 13:12:32.131526  2668 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:32.132206 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 13:12:32.132503 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-1-root
uuid: "bf69c09edabf4c568ab360ffa1755c90"
format_stamp: "Formatted at 2026-07-09 13:12:32 on dist-test-slave-k73n"
I20260709 13:12:32.132782 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:12:32.149628 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:12:32.150820 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:12:32.152237 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:12:32.154421 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:12:32.154589 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:32.154860 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:12:32.155017 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:32.196113 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.66:38355
I20260709 13:12:32.196225  2730 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.66:38355 every 8 connection(s)
I20260709 13:12:32.201498 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:12:32.208590  2734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:32.211087  2735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:12:32.212217  2731 heartbeater.cc:344] Connected to a master server at 127.23.140.126:37955
I20260709 13:12:32.212762  2731 heartbeater.cc:461] Registering TS with master...
I20260709 13:12:32.213253 24113 server_base.cc:1061] running on GCE node
I20260709 13:12:32.213795  2731 heartbeater.cc:507] Master 127.23.140.126:37955 requested a full tablet report, sending...
W20260709 13:12:32.214218  2737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:12:32.215453 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:12:32.215703 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:12:32.215901 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602752215884 us; error 0 us; skew 500 ppm
I20260709 13:12:32.216316  2529 ts_manager.cc:194] Registered new tserver with Master: bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
I20260709 13:12:32.216425 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:12:32.217831  2529 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48436
I20260709 13:12:32.219343 24113 webserver.cc:533] Webserver started at http://127.23.140.67:37165/ using document root <none> and password file <none>
I20260709 13:12:32.219873 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:12:32.220083 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:12:32.220342 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:12:32.221556 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-2-root/instance:
uuid: "154ab5545ca94930adbad11a6e38fd7b"
format_stamp: "Formatted at 2026-07-09 13:12:32 on dist-test-slave-k73n"
I20260709 13:12:32.225829 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20260709 13:12:32.229031  2742 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:32.229806 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260709 13:12:32.230095 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-2-root
uuid: "154ab5545ca94930adbad11a6e38fd7b"
format_stamp: "Formatted at 2026-07-09 13:12:32 on dist-test-slave-k73n"
I20260709 13:12:32.230386 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:12:32.251418 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:12:32.252599 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:12:32.253906 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:12:32.256331 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:12:32.256557 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:32.256791 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:12:32.256960 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:32.298631 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.67:41839
I20260709 13:12:32.298741  2804 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.67:41839 every 8 connection(s)
I20260709 13:12:32.312152  2805 heartbeater.cc:344] Connected to a master server at 127.23.140.126:37955
I20260709 13:12:32.312564  2805 heartbeater.cc:461] Registering TS with master...
I20260709 13:12:32.313306  2805 heartbeater.cc:507] Master 127.23.140.126:37955 requested a full tablet report, sending...
I20260709 13:12:32.315191  2529 ts_manager.cc:194] Registered new tserver with Master: 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:32.315311 24113 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013593332s
I20260709 13:12:32.316566  2529 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48444
I20260709 13:12:32.341907  2529 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48452:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:12:32.344852  2529 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:12:32.431006  2695 tablet_service.cc:1511] Processing CreateTablet for tablet 3a14307fa2c84dce941e556353a3f429 (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 13:12:32.431520  2694 tablet_service.cc:1511] Processing CreateTablet for tablet 93c8084adca149bcbf244d636791e9b2 (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 13:12:32.431922  2696 tablet_service.cc:1511] Processing CreateTablet for tablet 794a8d3f15854076abde75e5626686b0 (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 13:12:32.432610  2695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a14307fa2c84dce941e556353a3f429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.433341  2694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93c8084adca149bcbf244d636791e9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.434206  2696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 794a8d3f15854076abde75e5626686b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.436852  2693 tablet_service.cc:1511] Processing CreateTablet for tablet 281527dc3eb54d01b4c21684b06cdb75 (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 13:12:32.438620  2693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 281527dc3eb54d01b4c21684b06cdb75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.440192  2770 tablet_service.cc:1511] Processing CreateTablet for tablet 794a8d3f15854076abde75e5626686b0 (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 13:12:32.440616  2691 tablet_service.cc:1511] Processing CreateTablet for tablet 4116b417de684d87aed71c2ec197c7eb (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 13:12:32.441592  2770 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 794a8d3f15854076abde75e5626686b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.442057  2691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4116b417de684d87aed71c2ec197c7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.444551  2692 tablet_service.cc:1511] Processing CreateTablet for tablet ca9c690004a0496ebd5c127f5d82e22a (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 13:12:32.446283  2692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca9c690004a0496ebd5c127f5d82e22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.448181  2621 tablet_service.cc:1511] Processing CreateTablet for tablet 794a8d3f15854076abde75e5626686b0 (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 13:12:32.449071  2769 tablet_service.cc:1511] Processing CreateTablet for tablet 3a14307fa2c84dce941e556353a3f429 (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 13:12:32.450217  2621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 794a8d3f15854076abde75e5626686b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.450945  2769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a14307fa2c84dce941e556353a3f429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.453435  2620 tablet_service.cc:1511] Processing CreateTablet for tablet 3a14307fa2c84dce941e556353a3f429 (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 13:12:32.455355  2620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a14307fa2c84dce941e556353a3f429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.454959  2767 tablet_service.cc:1511] Processing CreateTablet for tablet 281527dc3eb54d01b4c21684b06cdb75 (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 13:12:32.456820  2767 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 281527dc3eb54d01b4c21684b06cdb75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.458700  2766 tablet_service.cc:1511] Processing CreateTablet for tablet ca9c690004a0496ebd5c127f5d82e22a (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 13:12:32.453502  2768 tablet_service.cc:1511] Processing CreateTablet for tablet 93c8084adca149bcbf244d636791e9b2 (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 13:12:32.461201  2619 tablet_service.cc:1511] Processing CreateTablet for tablet 93c8084adca149bcbf244d636791e9b2 (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 13:12:32.463047  2619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93c8084adca149bcbf244d636791e9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.464313  2765 tablet_service.cc:1511] Processing CreateTablet for tablet 4116b417de684d87aed71c2ec197c7eb (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 13:12:32.467821  2766 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca9c690004a0496ebd5c127f5d82e22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.469333  2765 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4116b417de684d87aed71c2ec197c7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.470799  2768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93c8084adca149bcbf244d636791e9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.479082  2690 tablet_service.cc:1511] Processing CreateTablet for tablet 90a4191a9d844e9daed17f0fca5cb8dc (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 13:12:32.482725  2689 tablet_service.cc:1511] Processing CreateTablet for tablet cbd31a8e68cb4d21974cd529b2fb2212 (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 13:12:32.485867  2688 tablet_service.cc:1511] Processing CreateTablet for tablet 47cba406eeea448d92432c1ecb287268 (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 13:12:32.487874  2618 tablet_service.cc:1511] Processing CreateTablet for tablet 281527dc3eb54d01b4c21684b06cdb75 (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 13:12:32.489285  2618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 281527dc3eb54d01b4c21684b06cdb75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.488960  2687 tablet_service.cc:1511] Processing CreateTablet for tablet 57abe29d17c14a28b9854d11c1b90c8b (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 13:12:32.492046  2764 tablet_service.cc:1511] Processing CreateTablet for tablet 90a4191a9d844e9daed17f0fca5cb8dc (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 13:12:32.498183  2762 tablet_service.cc:1511] Processing CreateTablet for tablet 47cba406eeea448d92432c1ecb287268 (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 13:12:32.499598  2762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47cba406eeea448d92432c1ecb287268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.495394  2763 tablet_service.cc:1511] Processing CreateTablet for tablet cbd31a8e68cb4d21974cd529b2fb2212 (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 13:12:32.503034  2761 tablet_service.cc:1511] Processing CreateTablet for tablet 57abe29d17c14a28b9854d11c1b90c8b (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 13:12:32.503087  2763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd31a8e68cb4d21974cd529b2fb2212. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.505214  2761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57abe29d17c14a28b9854d11c1b90c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.506835  2617 tablet_service.cc:1511] Processing CreateTablet for tablet ca9c690004a0496ebd5c127f5d82e22a (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 13:12:32.508281  2617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca9c690004a0496ebd5c127f5d82e22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.510845  2764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90a4191a9d844e9daed17f0fca5cb8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.481003  2690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90a4191a9d844e9daed17f0fca5cb8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.511849  2616 tablet_service.cc:1511] Processing CreateTablet for tablet 4116b417de684d87aed71c2ec197c7eb (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 13:12:32.513556  2689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd31a8e68cb4d21974cd529b2fb2212. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.513607  2616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4116b417de684d87aed71c2ec197c7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.515203  2687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57abe29d17c14a28b9854d11c1b90c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.516578  2688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47cba406eeea448d92432c1ecb287268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.516383  2615 tablet_service.cc:1511] Processing CreateTablet for tablet 90a4191a9d844e9daed17f0fca5cb8dc (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 13:12:32.518126  2615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90a4191a9d844e9daed17f0fca5cb8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.592757  2613 tablet_service.cc:1511] Processing CreateTablet for tablet 47cba406eeea448d92432c1ecb287268 (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 13:12:32.594272  2613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47cba406eeea448d92432c1ecb287268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.596239  2614 tablet_service.cc:1511] Processing CreateTablet for tablet cbd31a8e68cb4d21974cd529b2fb2212 (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 13:12:32.597332  2612 tablet_service.cc:1511] Processing CreateTablet for tablet 57abe29d17c14a28b9854d11c1b90c8b (DEFAULT_TABLE table=test-workload [id=954c35ce7edb4bada44739b3ebfd0054]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 13:12:32.598634  2614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd31a8e68cb4d21974cd529b2fb2212. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.601821  2612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57abe29d17c14a28b9854d11c1b90c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:32.648146  2820 tablet_bootstrap.cc:492] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90: Bootstrap starting.
I20260709 13:12:32.667316  2821 tablet_bootstrap.cc:492] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
I20260709 13:12:32.704104  2821 tablet_bootstrap.cc:654] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:32.721581  2820 tablet_bootstrap.cc:654] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:32.743788  2821 tablet_bootstrap.cc:492] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09: No bootstrap required, opened a new log
I20260709 13:12:32.744444  2821 ts_tablet_manager.cc:1403] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.077s	user 0.012s	sys 0.000s
I20260709 13:12:32.756809  2821 raft_consensus.cc:359] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:32.758023  2821 raft_consensus.cc:385] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:32.758518  2821 raft_consensus.cc:740] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:32.759706  2821 consensus_queue.cc:260] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:32.771219  2822 tablet_bootstrap.cc:492] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b: Bootstrap starting.
I20260709 13:12:32.772149  2656 heartbeater.cc:499] Master 127.23.140.126:37955 was elected leader, sending a full tablet report...
I20260709 13:12:32.772192  2821 ts_tablet_manager.cc:1434] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.027s	user 0.008s	sys 0.000s
I20260709 13:12:32.773365  2820 tablet_bootstrap.cc:492] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90: No bootstrap required, opened a new log
I20260709 13:12:32.773891  2820 ts_tablet_manager.cc:1403] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90: Time spent bootstrapping tablet: real 0.127s	user 0.013s	sys 0.000s
I20260709 13:12:32.775270  2821 tablet_bootstrap.cc:492] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
I20260709 13:12:32.776901  2820 raft_consensus.cc:359] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:32.777715  2820 raft_consensus.cc:385] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:32.778015  2820 raft_consensus.cc:740] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Initialized, Role: FOLLOWER
I20260709 13:12:32.779088  2822 tablet_bootstrap.cc:654] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:32.778847  2820 consensus_queue.cc:260] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:32.782455  2821 tablet_bootstrap.cc:654] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:32.802897  2820 ts_tablet_manager.cc:1434] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90: Time spent starting tablet: real 0.029s	user 0.019s	sys 0.009s
I20260709 13:12:32.803478  2731 heartbeater.cc:499] Master 127.23.140.126:37955 was elected leader, sending a full tablet report...
I20260709 13:12:32.804059  2820 tablet_bootstrap.cc:492] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90: Bootstrap starting.
I20260709 13:12:32.805735  2821 tablet_bootstrap.cc:492] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09: No bootstrap required, opened a new log
I20260709 13:12:32.806107  2822 tablet_bootstrap.cc:492] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b: No bootstrap required, opened a new log
I20260709 13:12:32.806305  2821 ts_tablet_manager.cc:1403] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.031s	user 0.018s	sys 0.012s
I20260709 13:12:32.806603  2822 ts_tablet_manager.cc:1403] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b: Time spent bootstrapping tablet: real 0.036s	user 0.008s	sys 0.021s
I20260709 13:12:32.809561  2822 raft_consensus.cc:359] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:32.810267  2822 raft_consensus.cc:385] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:32.810475  2820 tablet_bootstrap.cc:654] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:32.810551  2822 raft_consensus.cc:740] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Initialized, Role: FOLLOWER
I20260709 13:12:32.811717  2822 consensus_queue.cc:260] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:32.811985  2821 raft_consensus.cc:359] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:32.812846  2821 raft_consensus.cc:385] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:32.813356  2821 raft_consensus.cc:740] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:32.814640  2821 consensus_queue.cc:260] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:32.817890  2821 ts_tablet_manager.cc:1434] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 13:12:32.819233  2821 tablet_bootstrap.cc:492] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
I20260709 13:12:32.822522  2820 tablet_bootstrap.cc:492] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90: No bootstrap required, opened a new log
I20260709 13:12:32.823107  2820 ts_tablet_manager.cc:1403] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90: Time spent bootstrapping tablet: real 0.019s	user 0.011s	sys 0.007s
I20260709 13:12:32.826141  2820 raft_consensus.cc:359] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } }
I20260709 13:12:32.827023  2820 raft_consensus.cc:385] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:32.827399  2820 raft_consensus.cc:740] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Initialized, Role: FOLLOWER
I20260709 13:12:32.828061  2821 tablet_bootstrap.cc:654] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:32.828254  2820 consensus_queue.cc:260] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } }
I20260709 13:12:32.830888  2820 ts_tablet_manager.cc:1434] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:12:32.831983  2820 tablet_bootstrap.cc:492] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90: Bootstrap starting.
I20260709 13:12:32.844621  2820 tablet_bootstrap.cc:654] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90: Neither blocks nor log segments found. Creating new log.
W20260709 13:12:32.851207  2657 tablet.cc:2406] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 13:12:32.855626  2805 heartbeater.cc:499] Master 127.23.140.126:37955 was elected leader, sending a full tablet report...
I20260709 13:12:32.856362  2822 ts_tablet_manager.cc:1434] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b: Time spent starting tablet: real 0.049s	user 0.019s	sys 0.030s
I20260709 13:12:32.857776  2822 tablet_bootstrap.cc:492] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b: Bootstrap starting.
W20260709 13:12:32.859629  2577 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:12:32.862376  2826 raft_consensus.cc:493] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:32.863070  2826 raft_consensus.cc:515] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:32.866391  2822 tablet_bootstrap.cc:654] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:32.871413  2826 leader_election.cc:290] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:32.878253  2825 raft_consensus.cc:493] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:32.878966  2825 raft_consensus.cc:515] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:32.880828  2822 tablet_bootstrap.cc:492] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b: No bootstrap required, opened a new log
I20260709 13:12:32.881577  2822 ts_tablet_manager.cc:1403] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b: Time spent bootstrapping tablet: real 0.024s	user 0.011s	sys 0.010s
I20260709 13:12:32.885664  2822 raft_consensus.cc:359] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } }
I20260709 13:12:32.886802  2822 raft_consensus.cc:385] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:32.887252  2822 raft_consensus.cc:740] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Initialized, Role: FOLLOWER
I20260709 13:12:32.887377  2825 leader_election.cc:290] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:32.888309  2822 consensus_queue.cc:260] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } }
I20260709 13:12:32.899402  2822 ts_tablet_manager.cc:1434] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b: Time spent starting tablet: real 0.017s	user 0.010s	sys 0.003s
I20260709 13:12:32.900844  2822 tablet_bootstrap.cc:492] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b: Bootstrap starting.
I20260709 13:12:32.909590  2822 tablet_bootstrap.cc:654] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:32.922204  2821 tablet_bootstrap.cc:492] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09: No bootstrap required, opened a new log
I20260709 13:12:32.922441  2822 tablet_bootstrap.cc:492] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b: No bootstrap required, opened a new log
I20260709 13:12:32.922917  2821 ts_tablet_manager.cc:1403] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.104s	user 0.036s	sys 0.055s
I20260709 13:12:32.923172  2822 ts_tablet_manager.cc:1403] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b: Time spent bootstrapping tablet: real 0.023s	user 0.013s	sys 0.007s
I20260709 13:12:32.926445  2822 raft_consensus.cc:359] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:32.927325  2822 raft_consensus.cc:385] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:32.927687  2822 raft_consensus.cc:740] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Initialized, Role: FOLLOWER
I20260709 13:12:32.928685  2822 consensus_queue.cc:260] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:32.931399  2822 ts_tablet_manager.cc:1434] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:12:32.931069  2821 raft_consensus.cc:359] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:32.932488  2821 raft_consensus.cc:385] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:32.932744  2822 tablet_bootstrap.cc:492] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b: Bootstrap starting.
I20260709 13:12:32.932977  2821 raft_consensus.cc:740] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:32.935755  2821 consensus_queue.cc:260] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:32.940165  2822 tablet_bootstrap.cc:654] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:32.944314  2821 ts_tablet_manager.cc:1434] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.021s	user 0.010s	sys 0.001s
I20260709 13:12:32.945914  2821 tablet_bootstrap.cc:492] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
I20260709 13:12:32.951699  2829 raft_consensus.cc:493] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:32.952386  2829 raft_consensus.cc:515] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } }
I20260709 13:12:32.953294  2822 tablet_bootstrap.cc:492] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b: No bootstrap required, opened a new log
I20260709 13:12:32.953853  2822 ts_tablet_manager.cc:1403] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b: Time spent bootstrapping tablet: real 0.021s	user 0.008s	sys 0.010s
I20260709 13:12:32.955799  2829 leader_election.cc:290] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355), fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697)
I20260709 13:12:32.960001  2821 tablet_bootstrap.cc:654] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:32.974499  2822 raft_consensus.cc:359] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:32.981321  2780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c8084adca149bcbf244d636791e9b2" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b" is_pre_election: true
I20260709 13:12:32.981344  2822 raft_consensus.cc:385] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 13:12:32.983182  2670 leader_election.cc:343] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:32.983814  2822 raft_consensus.cc:740] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Initialized, Role: FOLLOWER
I20260709 13:12:32.984242  2829 raft_consensus.cc:493] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:32.984711  2829 raft_consensus.cc:515] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:32.984804  2822 consensus_queue.cc:260] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:32.986835  2829 leader_election.cc:290] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
I20260709 13:12:32.990073  2822 ts_tablet_manager.cc:1434] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b: Time spent starting tablet: real 0.036s	user 0.007s	sys 0.000s
I20260709 13:12:32.991376  2780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "281527dc3eb54d01b4c21684b06cdb75" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b" is_pre_election: true
I20260709 13:12:32.992506  2825 raft_consensus.cc:493] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:32.995113  2825 raft_consensus.cc:515] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:32.997277  2825 leader_election.cc:290] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:32.999110  2631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c8084adca149bcbf244d636791e9b2" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09" is_pre_election: true
W20260709 13:12:33.001209  2672 leader_election.cc:343] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:33.001804  2672 leader_election.cc:304] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf69c09edabf4c568ab360ffa1755c90; no voters: 154ab5545ca94930adbad11a6e38fd7b, fcffaa88967548fdb390c736bf217b09
I20260709 13:12:33.003736  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "281527dc3eb54d01b4c21684b06cdb75" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90" is_pre_election: true
I20260709 13:12:33.005183  2826 raft_consensus.cc:2749] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:12:32.999169  2822 tablet_bootstrap.cc:492] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b: Bootstrap starting.
W20260709 13:12:33.006417  2596 leader_election.cc:343] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:33.005493  2705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4116b417de684d87aed71c2ec197c7eb" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90" is_pre_election: true
W20260709 13:12:33.008493  2596 leader_election.cc:343] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:33.009997  2820 tablet_bootstrap.cc:492] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90: No bootstrap required, opened a new log
I20260709 13:12:33.010555  2820 ts_tablet_manager.cc:1403] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90: Time spent bootstrapping tablet: real 0.179s	user 0.080s	sys 0.089s
I20260709 13:12:33.011906  2705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "794a8d3f15854076abde75e5626686b0" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90" is_pre_election: true
I20260709 13:12:33.012552  2705 raft_consensus.cc:2468] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 0.
I20260709 13:12:33.013584  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47cba406eeea448d92432c1ecb287268" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90" is_pre_election: true
I20260709 13:12:33.015276  2822 tablet_bootstrap.cc:654] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.015331  2745 leader_election.cc:304] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90; no voters: 
I20260709 13:12:33.016692  2839 raft_consensus.cc:2804] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:33.017069  2839 raft_consensus.cc:493] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:33.017405  2839 raft_consensus.cc:3060] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.025858  2780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4116b417de684d87aed71c2ec197c7eb" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b" is_pre_election: true
W20260709 13:12:33.026270  2595 leader_election.cc:343] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:33.026810  2595 leader_election.cc:304] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fcffaa88967548fdb390c736bf217b09; no voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90
I20260709 13:12:33.027642  2825 raft_consensus.cc:2749] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:12:33.025455  2745 leader_election.cc:343] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:12:33.028148  2595 leader_election.cc:343] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:33.028865  2595 leader_election.cc:304] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fcffaa88967548fdb390c736bf217b09; no voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90
I20260709 13:12:33.031813  2825 raft_consensus.cc:2749] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:12:33.024924  2839 raft_consensus.cc:515] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } }
I20260709 13:12:33.033648  2820 raft_consensus.cc:359] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.034340  2631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "794a8d3f15854076abde75e5626686b0" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09" is_pre_election: true
I20260709 13:12:33.035898  2630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47cba406eeea448d92432c1ecb287268" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09" is_pre_election: true
I20260709 13:12:33.036486  2820 raft_consensus.cc:385] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 13:12:33.036597  2746 leader_election.cc:343] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:33.036857  2820 raft_consensus.cc:740] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Initialized, Role: FOLLOWER
W20260709 13:12:33.037810  2746 leader_election.cc:343] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:33.038316  2746 leader_election.cc:304] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b; no voters: bf69c09edabf4c568ab360ffa1755c90, fcffaa88967548fdb390c736bf217b09
I20260709 13:12:33.038764  2822 tablet_bootstrap.cc:492] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b: No bootstrap required, opened a new log
I20260709 13:12:33.039134  2829 raft_consensus.cc:2749] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:12:33.039316  2822 ts_tablet_manager.cc:1403] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b: Time spent bootstrapping tablet: real 0.040s	user 0.007s	sys 0.015s
I20260709 13:12:33.039558  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "794a8d3f15854076abde75e5626686b0" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
I20260709 13:12:33.040211  2706 raft_consensus.cc:3060] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.040546  2820 consensus_queue.cc:260] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.042793  2820 ts_tablet_manager.cc:1434] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.000s
I20260709 13:12:33.042398  2822 raft_consensus.cc:359] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.043277  2822 raft_consensus.cc:385] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.043650  2822 raft_consensus.cc:740] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.044695  2822 consensus_queue.cc:260] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.045792  2821 tablet_bootstrap.cc:492] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09: No bootstrap required, opened a new log
I20260709 13:12:33.046341  2821 ts_tablet_manager.cc:1403] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.101s	user 0.025s	sys 0.042s
I20260709 13:12:33.047385  2822 ts_tablet_manager.cc:1434] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:12:33.048565  2822 tablet_bootstrap.cc:492] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b: Bootstrap starting.
I20260709 13:12:33.049508  2821 raft_consensus.cc:359] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } }
I20260709 13:12:33.050252  2821 raft_consensus.cc:385] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.050544  2821 raft_consensus.cc:740] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.051326  2821 consensus_queue.cc:260] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } }
I20260709 13:12:33.052079  2706 raft_consensus.cc:2468] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 1.
I20260709 13:12:33.053711  2821 ts_tablet_manager.cc:1434] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.003s
I20260709 13:12:33.054798  2821 tablet_bootstrap.cc:492] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
I20260709 13:12:33.055601  2745 leader_election.cc:304] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90; no voters: 
I20260709 13:12:33.056222  2822 tablet_bootstrap.cc:654] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.056517  2829 raft_consensus.cc:2804] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:33.062039  2821 tablet_bootstrap.cc:654] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.068442  2826 raft_consensus.cc:493] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:33.069013  2826 raft_consensus.cc:515] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.069315  2822 tablet_bootstrap.cc:492] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b: No bootstrap required, opened a new log
I20260709 13:12:33.070189  2822 ts_tablet_manager.cc:1403] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.010s
I20260709 13:12:33.071256  2826 leader_election.cc:290] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:33.074410  2630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4116b417de684d87aed71c2ec197c7eb" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09" is_pre_election: true
I20260709 13:12:33.075184  2630 raft_consensus.cc:2468] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 0.
I20260709 13:12:33.076453  2672 leader_election.cc:304] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf69c09edabf4c568ab360ffa1755c90, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:33.077598  2780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4116b417de684d87aed71c2ec197c7eb" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b" is_pre_election: true
I20260709 13:12:33.077848  2821 tablet_bootstrap.cc:492] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09: No bootstrap required, opened a new log
I20260709 13:12:33.078069  2839 leader_election.cc:290] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 election: Requested vote from peers bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355), fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697)
I20260709 13:12:33.078994  2826 raft_consensus.cc:2804] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 13:12:33.079033  2670 leader_election.cc:343] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:33.079587  2826 raft_consensus.cc:493] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:33.074479  2822 raft_consensus.cc:359] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.079912  2630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "794a8d3f15854076abde75e5626686b0" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:33.080061  2822 raft_consensus.cc:385] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.080116  2821 ts_tablet_manager.cc:1403] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.024s	user 0.008s	sys 0.014s
I20260709 13:12:33.080583  2630 raft_consensus.cc:3060] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.080598  2822 raft_consensus.cc:740] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.079932  2826 raft_consensus.cc:3060] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.080991  2820 tablet_bootstrap.cc:492] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90: Bootstrap starting.
I20260709 13:12:33.083734  2821 raft_consensus.cc:359] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.084515  2821 raft_consensus.cc:385] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.084877  2821 raft_consensus.cc:740] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.085875  2821 consensus_queue.cc:260] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.088456  2820 tablet_bootstrap.cc:654] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.088537  2630 raft_consensus.cc:2468] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 1.
I20260709 13:12:33.088573  2826 raft_consensus.cc:515] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.090848  2829 raft_consensus.cc:697] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 1 LEADER]: Becoming Leader. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Running, Role: LEADER
I20260709 13:12:33.091802  2822 consensus_queue.cc:260] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.091907  2829 consensus_queue.cc:237] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } }
I20260709 13:12:33.094236  2822 ts_tablet_manager.cc:1434] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.000s
I20260709 13:12:33.095350  2822 tablet_bootstrap.cc:492] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b: Bootstrap starting.
I20260709 13:12:33.096736  2821 ts_tablet_manager.cc:1434] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.016s	user 0.000s	sys 0.006s
I20260709 13:12:33.101363  2630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4116b417de684d87aed71c2ec197c7eb" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:33.101873  2822 tablet_bootstrap.cc:654] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.102289  2630 raft_consensus.cc:3060] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.107685  2820 tablet_bootstrap.cc:492] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90: No bootstrap required, opened a new log
I20260709 13:12:33.108229  2820 ts_tablet_manager.cc:1403] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90: Time spent bootstrapping tablet: real 0.027s	user 0.009s	sys 0.002s
I20260709 13:12:33.111292  2820 raft_consensus.cc:359] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.112013  2820 raft_consensus.cc:385] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.112300  2820 raft_consensus.cc:740] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.115765  2821 tablet_bootstrap.cc:492] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
I20260709 13:12:33.115733  2780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4116b417de684d87aed71c2ec197c7eb" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
W20260709 13:12:33.117482  2670 leader_election.cc:343] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:33.118085  2826 leader_election.cc:290] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:33.118350  2822 tablet_bootstrap.cc:492] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b: No bootstrap required, opened a new log
I20260709 13:12:33.118908  2822 ts_tablet_manager.cc:1403] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b: Time spent bootstrapping tablet: real 0.024s	user 0.010s	sys 0.000s
I20260709 13:12:33.120466  2630 raft_consensus.cc:2468] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 1.
I20260709 13:12:33.120580  2528 catalog_manager.cc:5697] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67). New cstate: current_term: 1 leader_uuid: "154ab5545ca94930adbad11a6e38fd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:33.113039  2820 consensus_queue.cc:260] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.121742  2672 leader_election.cc:304] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf69c09edabf4c568ab360ffa1755c90, fcffaa88967548fdb390c736bf217b09; no voters: 154ab5545ca94930adbad11a6e38fd7b
I20260709 13:12:33.121939  2822 raft_consensus.cc:359] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.122717  2822 raft_consensus.cc:385] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.123009  2822 raft_consensus.cc:740] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.123569  2820 ts_tablet_manager.cc:1434] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90: Time spent starting tablet: real 0.015s	user 0.004s	sys 0.004s
I20260709 13:12:33.123783  2822 consensus_queue.cc:260] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.126415  2822 ts_tablet_manager.cc:1434] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:12:33.127046  2820 tablet_bootstrap.cc:492] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90: Bootstrap starting.
I20260709 13:12:33.127481  2822 tablet_bootstrap.cc:492] T 281527dc3eb54d01b4c21684b06cdb75 P 154ab5545ca94930adbad11a6e38fd7b: Bootstrap starting.
I20260709 13:12:33.129647  2821 tablet_bootstrap.cc:654] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.135743  2822 tablet_bootstrap.cc:654] T 281527dc3eb54d01b4c21684b06cdb75 P 154ab5545ca94930adbad11a6e38fd7b: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.140067  2829 raft_consensus.cc:493] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:33.141057  2820 tablet_bootstrap.cc:654] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.140760  2829 raft_consensus.cc:515] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.144018  2826 raft_consensus.cc:2804] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:33.144629  2826 raft_consensus.cc:697] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 1 LEADER]: Becoming Leader. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:33.145467  2826 consensus_queue.cc:237] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.145927  2630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09" is_pre_election: true
I20260709 13:12:33.146263  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90" is_pre_election: true
W20260709 13:12:33.147933  2746 leader_election.cc:343] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:33.148571  2829 leader_election.cc:290] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
W20260709 13:12:33.149705  2745 leader_election.cc:343] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:33.151640  2745 leader_election.cc:304] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b; no voters: bf69c09edabf4c568ab360ffa1755c90, fcffaa88967548fdb390c736bf217b09
I20260709 13:12:33.152874  2848 raft_consensus.cc:2749] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:12:33.161270  2821 tablet_bootstrap.cc:492] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09: No bootstrap required, opened a new log
I20260709 13:12:33.162748  2820 tablet_bootstrap.cc:492] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90: No bootstrap required, opened a new log
I20260709 13:12:33.161828  2527 catalog_manager.cc:5697] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "bf69c09edabf4c568ab360ffa1755c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:33.163160  2822 tablet_bootstrap.cc:492] T 281527dc3eb54d01b4c21684b06cdb75 P 154ab5545ca94930adbad11a6e38fd7b: No bootstrap required, opened a new log
I20260709 13:12:33.163461  2820 ts_tablet_manager.cc:1403] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90: Time spent bootstrapping tablet: real 0.037s	user 0.012s	sys 0.005s
I20260709 13:12:33.163648  2822 ts_tablet_manager.cc:1403] T 281527dc3eb54d01b4c21684b06cdb75 P 154ab5545ca94930adbad11a6e38fd7b: Time spent bootstrapping tablet: real 0.036s	user 0.009s	sys 0.007s
I20260709 13:12:33.161861  2821 ts_tablet_manager.cc:1403] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.046s	user 0.021s	sys 0.016s
I20260709 13:12:33.166563  2820 raft_consensus.cc:359] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.167294  2820 raft_consensus.cc:385] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.167266  2822 raft_consensus.cc:359] T 281527dc3eb54d01b4c21684b06cdb75 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.168081  2820 raft_consensus.cc:740] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.167603  2821 raft_consensus.cc:359] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.168458  2822 raft_consensus.cc:385] T 281527dc3eb54d01b4c21684b06cdb75 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.168723  2821 raft_consensus.cc:385] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.168891  2822 raft_consensus.cc:740] T 281527dc3eb54d01b4c21684b06cdb75 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.169296  2821 raft_consensus.cc:740] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.169039  2820 consensus_queue.cc:260] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.169909  2822 consensus_queue.cc:260] T 281527dc3eb54d01b4c21684b06cdb75 P 154ab5545ca94930adbad11a6e38fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.170347  2821 consensus_queue.cc:260] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.171900  2820 ts_tablet_manager.cc:1434] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:12:33.172052  2822 ts_tablet_manager.cc:1434] T 281527dc3eb54d01b4c21684b06cdb75 P 154ab5545ca94930adbad11a6e38fd7b: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:12:33.172971  2820 tablet_bootstrap.cc:492] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90: Bootstrap starting.
I20260709 13:12:33.173002  2822 tablet_bootstrap.cc:492] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b: Bootstrap starting.
I20260709 13:12:33.180037  2820 tablet_bootstrap.cc:654] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.180145  2822 tablet_bootstrap.cc:654] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.181533  2821 ts_tablet_manager.cc:1434] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.017s	user 0.003s	sys 0.004s
I20260709 13:12:33.187986  2821 tablet_bootstrap.cc:492] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
I20260709 13:12:33.194962  2829 raft_consensus.cc:493] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:33.195720  2829 raft_consensus.cc:515] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.197360  2821 tablet_bootstrap.cc:654] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.199572  2820 tablet_bootstrap.cc:492] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90: No bootstrap required, opened a new log
I20260709 13:12:33.199530  2630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd31a8e68cb4d21974cd529b2fb2212" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09" is_pre_election: true
I20260709 13:12:33.200519  2820 ts_tablet_manager.cc:1403] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90: Time spent bootstrapping tablet: real 0.028s	user 0.017s	sys 0.009s
I20260709 13:12:33.202713  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd31a8e68cb4d21974cd529b2fb2212" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90" is_pre_election: true
W20260709 13:12:33.204319  2745 leader_election.cc:343] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:33.205143  2820 raft_consensus.cc:359] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
W20260709 13:12:33.205706  2746 leader_election.cc:343] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:33.206267  2829 leader_election.cc:290] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
I20260709 13:12:33.206414  2746 leader_election.cc:304] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b; no voters: bf69c09edabf4c568ab360ffa1755c90, fcffaa88967548fdb390c736bf217b09
I20260709 13:12:33.207763  2829 raft_consensus.cc:2749] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:12:33.208019  2820 raft_consensus.cc:385] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.208704  2820 raft_consensus.cc:740] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.210094  2820 consensus_queue.cc:260] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.216096  2822 tablet_bootstrap.cc:492] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b: No bootstrap required, opened a new log
I20260709 13:12:33.216616  2822 ts_tablet_manager.cc:1403] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b: Time spent bootstrapping tablet: real 0.044s	user 0.025s	sys 0.014s
I20260709 13:12:33.217289  2821 tablet_bootstrap.cc:492] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09: No bootstrap required, opened a new log
I20260709 13:12:33.217782  2821 ts_tablet_manager.cc:1403] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.030s	user 0.012s	sys 0.016s
I20260709 13:12:33.219465  2822 raft_consensus.cc:359] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.220192  2822 raft_consensus.cc:385] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.220456  2822 raft_consensus.cc:740] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.220537  2821 raft_consensus.cc:359] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.221354  2821 raft_consensus.cc:385] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.221210  2822 consensus_queue.cc:260] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.221710  2821 raft_consensus.cc:740] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.222621  2821 consensus_queue.cc:260] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.223316  2822 ts_tablet_manager.cc:1434] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:12:33.224478  2822 tablet_bootstrap.cc:492] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b: Bootstrap starting.
I20260709 13:12:33.224900  2821 ts_tablet_manager.cc:1434] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:12:33.226168  2821 tablet_bootstrap.cc:492] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
I20260709 13:12:33.233191  2829 raft_consensus.cc:493] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:33.234905  2821 tablet_bootstrap.cc:654] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.234789  2829 raft_consensus.cc:515] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.236944  2829 leader_election.cc:290] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
I20260709 13:12:33.238410  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57abe29d17c14a28b9854d11c1b90c8b" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90" is_pre_election: true
I20260709 13:12:33.229877  2822 tablet_bootstrap.cc:654] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.239621  2630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57abe29d17c14a28b9854d11c1b90c8b" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09" is_pre_election: true
W20260709 13:12:33.241070  2745 leader_election.cc:343] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:12:33.242389  2746 leader_election.cc:343] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:33.242959  2746 leader_election.cc:304] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b; no voters: bf69c09edabf4c568ab360ffa1755c90, fcffaa88967548fdb390c736bf217b09
I20260709 13:12:33.244441  2829 raft_consensus.cc:2749] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:12:33.249485  2820 ts_tablet_manager.cc:1434] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90: Time spent starting tablet: real 0.048s	user 0.010s	sys 0.000s
I20260709 13:12:33.252076  2829 raft_consensus.cc:493] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:33.252588  2829 raft_consensus.cc:515] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.254422  2822 tablet_bootstrap.cc:492] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b: No bootstrap required, opened a new log
I20260709 13:12:33.254788  2829 leader_election.cc:290] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
I20260709 13:12:33.254904  2822 ts_tablet_manager.cc:1403] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b: Time spent bootstrapping tablet: real 0.031s	user 0.019s	sys 0.000s
I20260709 13:12:33.255478  2630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4116b417de684d87aed71c2ec197c7eb" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09" is_pre_election: true
I20260709 13:12:33.256269  2630 raft_consensus.cc:2393] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 154ab5545ca94930adbad11a6e38fd7b in current term 1: Already voted for candidate bf69c09edabf4c568ab360ffa1755c90 in this term.
I20260709 13:12:33.256403  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4116b417de684d87aed71c2ec197c7eb" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90" is_pre_election: true
I20260709 13:12:33.257884  2820 tablet_bootstrap.cc:492] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90: Bootstrap starting.
I20260709 13:12:33.257728  2822 raft_consensus.cc:359] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.258023  2745 leader_election.cc:304] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b; no voters: bf69c09edabf4c568ab360ffa1755c90, fcffaa88967548fdb390c736bf217b09
I20260709 13:12:33.258477  2822 raft_consensus.cc:385] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.258812  2822 raft_consensus.cc:740] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.259261  2829 raft_consensus.cc:3060] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.259430  2822 consensus_queue.cc:260] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.261425  2822 ts_tablet_manager.cc:1434] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:12:33.263783  2821 tablet_bootstrap.cc:492] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09: No bootstrap required, opened a new log
I20260709 13:12:33.264288  2821 ts_tablet_manager.cc:1403] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.038s	user 0.018s	sys 0.008s
I20260709 13:12:33.265455  2820 tablet_bootstrap.cc:654] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.266969  2821 raft_consensus.cc:359] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.267673  2821 raft_consensus.cc:385] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.267944  2821 raft_consensus.cc:740] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.267992  2829 raft_consensus.cc:2749] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:12:33.269424  2829 raft_consensus.cc:493] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:33.269055  2821 consensus_queue.cc:260] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.270087  2829 raft_consensus.cc:515] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.271811  2821 ts_tablet_manager.cc:1434] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.003s
I20260709 13:12:33.273497  2630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a14307fa2c84dce941e556353a3f429" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09" is_pre_election: true
I20260709 13:12:33.274123  2630 raft_consensus.cc:2468] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 0.
I20260709 13:12:33.274444  2821 tablet_bootstrap.cc:492] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
I20260709 13:12:33.275257  2746 leader_election.cc:304] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:33.276091  2848 raft_consensus.cc:2804] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:33.276417  2848 raft_consensus.cc:493] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:33.276695  2848 raft_consensus.cc:3060] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.277760  2829 leader_election.cc:290] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
I20260709 13:12:33.278863  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a14307fa2c84dce941e556353a3f429" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90" is_pre_election: true
W20260709 13:12:33.280192  2745 leader_election.cc:343] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:33.283262  2821 tablet_bootstrap.cc:654] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.283121  2848 raft_consensus.cc:515] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.284825  2820 tablet_bootstrap.cc:492] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90: No bootstrap required, opened a new log
I20260709 13:12:33.285372  2820 ts_tablet_manager.cc:1403] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90: Time spent bootstrapping tablet: real 0.028s	user 0.010s	sys 0.008s
I20260709 13:12:33.285959  2630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a14307fa2c84dce941e556353a3f429" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:33.286561  2630 raft_consensus.cc:3060] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.288419  2829 raft_consensus.cc:493] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:33.288957  2829 raft_consensus.cc:515] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.290095  2820 raft_consensus.cc:359] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.291109  2820 raft_consensus.cc:385] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.291168  2829 leader_election.cc:290] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
I20260709 13:12:33.291446  2820 raft_consensus.cc:740] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.291788  2631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca9c690004a0496ebd5c127f5d82e22a" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09" is_pre_election: true
I20260709 13:12:33.292404  2631 raft_consensus.cc:2468] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 0.
I20260709 13:12:33.292462  2820 consensus_queue.cc:260] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.292404  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca9c690004a0496ebd5c127f5d82e22a" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90" is_pre_election: true
I20260709 13:12:33.293929  2706 raft_consensus.cc:2468] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 0.
I20260709 13:12:33.293834  2746 leader_election.cc:304] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:33.295727  2821 tablet_bootstrap.cc:492] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09: No bootstrap required, opened a new log
I20260709 13:12:33.296216  2821 ts_tablet_manager.cc:1403] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.024s	user 0.012s	sys 0.004s
I20260709 13:12:33.297344  2829 raft_consensus.cc:2804] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:33.297465  2630 raft_consensus.cc:2468] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 1.
I20260709 13:12:33.298146  2829 raft_consensus.cc:493] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:33.298921  2746 leader_election.cc:304] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:33.299594  2829 raft_consensus.cc:3060] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.299921  2839 raft_consensus.cc:2804] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:33.299015  2848 leader_election.cc:290] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
I20260709 13:12:33.300055  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a14307fa2c84dce941e556353a3f429" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
W20260709 13:12:33.301527  2745 leader_election.cc:343] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:33.301723  2820 ts_tablet_manager.cc:1434] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 13:12:33.298822  2821 raft_consensus.cc:359] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.302928  2821 raft_consensus.cc:385] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.303202  2821 raft_consensus.cc:740] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.303943  2821 consensus_queue.cc:260] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.305624  2820 tablet_bootstrap.cc:492] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90: Bootstrap starting.
I20260709 13:12:33.310330  2821 ts_tablet_manager.cc:1434] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 13:12:33.311335  2821 tablet_bootstrap.cc:492] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
I20260709 13:12:33.312994  2820 tablet_bootstrap.cc:654] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.314657  2854 raft_consensus.cc:493] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:33.315172  2854 raft_consensus.cc:515] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.317075  2821 tablet_bootstrap.cc:654] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.316597  2829 raft_consensus.cc:515] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.317368  2854 leader_election.cc:290] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:33.319926  2780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "281527dc3eb54d01b4c21684b06cdb75" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b" is_pre_election: true
I20260709 13:12:33.320314  2631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca9c690004a0496ebd5c127f5d82e22a" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:33.320832  2780 raft_consensus.cc:2468] T 281527dc3eb54d01b4c21684b06cdb75 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 0.
I20260709 13:12:33.321450  2829 leader_election.cc:290] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
I20260709 13:12:33.321568  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca9c690004a0496ebd5c127f5d82e22a" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
I20260709 13:12:33.320917  2631 raft_consensus.cc:3060] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.322322  2706 raft_consensus.cc:3060] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.322400  2839 raft_consensus.cc:697] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [term 1 LEADER]: Becoming Leader. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Running, Role: LEADER
I20260709 13:12:33.323460  2839 consensus_queue.cc:237] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.322274  2670 leader_election.cc:304] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90; no voters: 
I20260709 13:12:33.327301  2825 raft_consensus.cc:493] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:33.327461  2854 raft_consensus.cc:2804] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:33.327951  2854 raft_consensus.cc:493] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:33.327821  2825 raft_consensus.cc:515] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.328308  2854 raft_consensus.cc:3060] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.329919  2825 leader_election.cc:290] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:33.331739  2630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "281527dc3eb54d01b4c21684b06cdb75" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09" is_pre_election: true
I20260709 13:12:33.331871  2780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b" is_pre_election: true
I20260709 13:12:33.332360  2705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90" is_pre_election: true
I20260709 13:12:33.332553  2780 raft_consensus.cc:2468] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcffaa88967548fdb390c736bf217b09 in term 0.
I20260709 13:12:33.333007  2705 raft_consensus.cc:2468] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcffaa88967548fdb390c736bf217b09 in term 0.
I20260709 13:12:33.333802  2595 leader_election.cc:304] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:33.335230  2825 raft_consensus.cc:2804] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:33.335589  2825 raft_consensus.cc:493] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:33.335902  2825 raft_consensus.cc:3060] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.337687  2631 raft_consensus.cc:2468] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 1.
I20260709 13:12:33.338872  2746 leader_election.cc:304] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:33.339747  2829 raft_consensus.cc:2804] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:33.340241  2829 raft_consensus.cc:697] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 1 LEADER]: Becoming Leader. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Running, Role: LEADER
I20260709 13:12:33.341089  2829 consensus_queue.cc:237] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.343822  2706 raft_consensus.cc:2468] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 1.
I20260709 13:12:33.347276  2821 tablet_bootstrap.cc:492] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09: No bootstrap required, opened a new log
I20260709 13:12:33.347774  2821 ts_tablet_manager.cc:1403] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.037s	user 0.007s	sys 0.003s
I20260709 13:12:33.332571  2630 raft_consensus.cc:2468] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 0.
I20260709 13:12:33.350459  2821 raft_consensus.cc:359] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.351239  2821 raft_consensus.cc:385] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.351583  2821 raft_consensus.cc:740] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.348812  2854 raft_consensus.cc:515] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.354957  2630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "281527dc3eb54d01b4c21684b06cdb75" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:33.355537  2630 raft_consensus.cc:3060] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.355857  2825 raft_consensus.cc:515] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.358796  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
I20260709 13:12:33.359380  2706 raft_consensus.cc:3060] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.363042  2854 leader_election.cc:290] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:33.364276  2780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "281527dc3eb54d01b4c21684b06cdb75" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
I20260709 13:12:33.365116  2780 raft_consensus.cc:3060] T 281527dc3eb54d01b4c21684b06cdb75 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.367647  2630 raft_consensus.cc:2468] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 1.
I20260709 13:12:33.368885  2672 leader_election.cc:304] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf69c09edabf4c568ab360ffa1755c90, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:33.369772  2854 raft_consensus.cc:2804] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:33.370990  2854 raft_consensus.cc:697] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [term 1 LEADER]: Becoming Leader. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:33.371814  2854 consensus_queue.cc:237] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.372351  2780 raft_consensus.cc:2468] T 281527dc3eb54d01b4c21684b06cdb75 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 1.
I20260709 13:12:33.378371  2825 leader_election.cc:290] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 election: Requested vote from peers bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:33.378710  2779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
I20260709 13:12:33.379356  2779 raft_consensus.cc:3060] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.382383  2527 catalog_manager.cc:5697] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67). New cstate: current_term: 1 leader_uuid: "154ab5545ca94930adbad11a6e38fd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:33.384384  2706 raft_consensus.cc:2468] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcffaa88967548fdb390c736bf217b09 in term 1.
I20260709 13:12:33.386431  2821 consensus_queue.cc:260] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.387161  2779 raft_consensus.cc:2468] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcffaa88967548fdb390c736bf217b09 in term 1.
I20260709 13:12:33.388507  2595 leader_election.cc:304] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:33.389386  2821 ts_tablet_manager.cc:1434] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.041s	user 0.008s	sys 0.002s
I20260709 13:12:33.393604  2825 raft_consensus.cc:2804] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:33.394367  2825 raft_consensus.cc:697] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 1 LEADER]: Becoming Leader. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Running, Role: LEADER
I20260709 13:12:33.395412  2825 consensus_queue.cc:237] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.399485  2820 tablet_bootstrap.cc:492] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90: No bootstrap required, opened a new log
I20260709 13:12:33.400103  2820 ts_tablet_manager.cc:1403] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90: Time spent bootstrapping tablet: real 0.095s	user 0.025s	sys 0.026s
I20260709 13:12:33.401057  2528 catalog_manager.cc:5697] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67). New cstate: current_term: 1 leader_uuid: "154ab5545ca94930adbad11a6e38fd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:33.409914  2820 raft_consensus.cc:359] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.410742  2820 raft_consensus.cc:385] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.411069  2820 raft_consensus.cc:740] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.413451  2820 consensus_queue.cc:260] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.415000  2820 ts_tablet_manager.cc:1434] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90: Time spent starting tablet: real 0.015s	user 0.004s	sys 0.000s
I20260709 13:12:33.415853  2820 tablet_bootstrap.cc:492] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90: Bootstrap starting.
I20260709 13:12:33.419142  2527 catalog_manager.cc:5697] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcffaa88967548fdb390c736bf217b09 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "fcffaa88967548fdb390c736bf217b09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:33.421136  2820 tablet_bootstrap.cc:654] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.423023  2528 catalog_manager.cc:5697] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "bf69c09edabf4c568ab360ffa1755c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:33.426636  2820 tablet_bootstrap.cc:492] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90: No bootstrap required, opened a new log
I20260709 13:12:33.427150  2820 ts_tablet_manager.cc:1403] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90: Time spent bootstrapping tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 13:12:33.429978  2820 raft_consensus.cc:359] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.430495  2820 raft_consensus.cc:385] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.430783  2820 raft_consensus.cc:740] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.431456  2820 consensus_queue.cc:260] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.433355  2820 ts_tablet_manager.cc:1434] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:12:33.434540  2820 tablet_bootstrap.cc:492] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90: Bootstrap starting.
I20260709 13:12:33.439857  2820 tablet_bootstrap.cc:654] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:33.444226  2820 tablet_bootstrap.cc:492] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90: No bootstrap required, opened a new log
I20260709 13:12:33.444710  2820 ts_tablet_manager.cc:1403] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90: Time spent bootstrapping tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 13:12:33.447448  2820 raft_consensus.cc:359] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.448113  2820 raft_consensus.cc:385] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:33.448349  2820 raft_consensus.cc:740] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Initialized, Role: FOLLOWER
I20260709 13:12:33.448961  2820 consensus_queue.cc:260] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.450778  2820 ts_tablet_manager.cc:1434] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:12:33.493062  2854 raft_consensus.cc:493] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:33.495550  2876 raft_consensus.cc:493] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:33.498454  2876 raft_consensus.cc:515] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.493435  2825 raft_consensus.cc:493] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:33.503850  2825 raft_consensus.cc:515] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.505602  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c8084adca149bcbf244d636791e9b2" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90" is_pre_election: true
I20260709 13:12:33.507287  2706 raft_consensus.cc:2468] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcffaa88967548fdb390c736bf217b09 in term 0.
I20260709 13:12:33.508044  2779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c8084adca149bcbf244d636791e9b2" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b" is_pre_election: true
I20260709 13:12:33.504820  2876 leader_election.cc:290] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:33.509512  2779 raft_consensus.cc:2468] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcffaa88967548fdb390c736bf217b09 in term 0.
I20260709 13:12:33.509617  2825 leader_election.cc:290] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:33.513447  2595 leader_election.cc:304] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:33.513890  2779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47cba406eeea448d92432c1ecb287268" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b" is_pre_election: true
I20260709 13:12:33.513820  2854 raft_consensus.cc:515] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.515007  2779 raft_consensus.cc:2468] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcffaa88967548fdb390c736bf217b09 in term 0.
I20260709 13:12:33.515349  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47cba406eeea448d92432c1ecb287268" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90" is_pre_election: true
I20260709 13:12:33.515998  2706 raft_consensus.cc:2468] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcffaa88967548fdb390c736bf217b09 in term 0.
I20260709 13:12:33.516449  2595 leader_election.cc:304] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:33.516959  2825 raft_consensus.cc:2804] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:33.517474  2825 raft_consensus.cc:493] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:33.517597  2854 leader_election.cc:290] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:33.517908  2825 raft_consensus.cc:3060] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.518550  2630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47cba406eeea448d92432c1ecb287268" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09" is_pre_election: true
I20260709 13:12:33.518958  2876 raft_consensus.cc:2804] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:33.519476  2876 raft_consensus.cc:493] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:33.519847  2876 raft_consensus.cc:3060] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.520045  2779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47cba406eeea448d92432c1ecb287268" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b" is_pre_election: true
I20260709 13:12:33.520824  2779 raft_consensus.cc:2468] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 0.
I20260709 13:12:33.522574  2670 leader_election.cc:304] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90; no voters: 
I20260709 13:12:33.529354  2854 raft_consensus.cc:2804] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:33.529791  2854 raft_consensus.cc:493] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:33.530141  2854 raft_consensus.cc:3060] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.530282  2825 raft_consensus.cc:515] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.533711  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c8084adca149bcbf244d636791e9b2" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
I20260709 13:12:33.534385  2706 raft_consensus.cc:3060] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.536209  2876 raft_consensus.cc:515] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.537789  2630 raft_consensus.cc:2393] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bf69c09edabf4c568ab360ffa1755c90 in current term 1: Already voted for candidate fcffaa88967548fdb390c736bf217b09 in this term.
I20260709 13:12:33.542038  2825 leader_election.cc:290] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 election: Requested vote from peers bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:33.542516  2854 raft_consensus.cc:515] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.543268  2779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c8084adca149bcbf244d636791e9b2" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
I20260709 13:12:33.543941  2779 raft_consensus.cc:3060] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.544906  2854 leader_election.cc:290] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:33.546083  2706 raft_consensus.cc:2468] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcffaa88967548fdb390c736bf217b09 in term 1.
I20260709 13:12:33.546504  2631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47cba406eeea448d92432c1ecb287268" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:33.546800  2780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47cba406eeea448d92432c1ecb287268" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
I20260709 13:12:33.547372  2780 raft_consensus.cc:3060] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.547544  2596 leader_election.cc:304] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf69c09edabf4c568ab360ffa1755c90, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:33.548873  2825 raft_consensus.cc:2804] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:33.553270  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47cba406eeea448d92432c1ecb287268" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
I20260709 13:12:33.554490  2778 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47cba406eeea448d92432c1ecb287268" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
I20260709 13:12:33.555779  2706 raft_consensus.cc:2393] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fcffaa88967548fdb390c736bf217b09 in current term 1: Already voted for candidate bf69c09edabf4c568ab360ffa1755c90 in this term.
I20260709 13:12:33.555447  2779 raft_consensus.cc:2468] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcffaa88967548fdb390c736bf217b09 in term 1.
I20260709 13:12:33.557453  2596 leader_election.cc:304] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fcffaa88967548fdb390c736bf217b09; no voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90
I20260709 13:12:33.558426  2780 raft_consensus.cc:2468] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 1.
I20260709 13:12:33.553050  2876 leader_election.cc:290] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 election: Requested vote from peers bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:33.560820  2876 raft_consensus.cc:2749] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:12:33.560642  2881 raft_consensus.cc:493] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:33.562121  2881 raft_consensus.cc:515] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.563678  2825 raft_consensus.cc:697] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 1 LEADER]: Becoming Leader. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Running, Role: LEADER
I20260709 13:12:33.565196  2825 consensus_queue.cc:237] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.566679  2779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd31a8e68cb4d21974cd529b2fb2212" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b" is_pre_election: true
I20260709 13:12:33.567507  2779 raft_consensus.cc:2468] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcffaa88967548fdb390c736bf217b09 in term 0.
I20260709 13:12:33.568981  2595 leader_election.cc:304] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:33.570036  2876 raft_consensus.cc:2804] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:33.570716  2881 leader_election.cc:290] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:33.570478  2876 raft_consensus.cc:493] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:33.573000  2876 raft_consensus.cc:3060] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.573678  2670 leader_election.cc:304] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90; no voters: fcffaa88967548fdb390c736bf217b09
I20260709 13:12:33.573829  2839 consensus_queue.cc:1048] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:12:33.574820  2854 raft_consensus.cc:2804] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:33.575392  2854 raft_consensus.cc:697] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 1 LEADER]: Becoming Leader. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:33.577409  2854 consensus_queue.cc:237] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.582108  2876 raft_consensus.cc:515] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.585271  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd31a8e68cb4d21974cd529b2fb2212" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90" is_pre_election: true
I20260709 13:12:33.586048  2706 raft_consensus.cc:2468] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcffaa88967548fdb390c736bf217b09 in term 0.
I20260709 13:12:33.586177  2780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd31a8e68cb4d21974cd529b2fb2212" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
I20260709 13:12:33.587141  2780 raft_consensus.cc:3060] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.587020  2528 catalog_manager.cc:5697] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcffaa88967548fdb390c736bf217b09 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "fcffaa88967548fdb390c736bf217b09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:33.588934  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd31a8e68cb4d21974cd529b2fb2212" candidate_uuid: "fcffaa88967548fdb390c736bf217b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
I20260709 13:12:33.589584  2706 raft_consensus.cc:3060] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.594542  2876 leader_election.cc:290] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 election: Requested vote from peers bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:33.600499  2780 raft_consensus.cc:2468] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcffaa88967548fdb390c736bf217b09 in term 1.
I20260709 13:12:33.602015  2595 leader_election.cc:304] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:33.602916  2876 raft_consensus.cc:2804] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:33.603401  2876 raft_consensus.cc:697] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 1 LEADER]: Becoming Leader. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Running, Role: LEADER
I20260709 13:12:33.604161  2706 raft_consensus.cc:2468] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcffaa88967548fdb390c736bf217b09 in term 1.
I20260709 13:12:33.604398  2876 consensus_queue.cc:237] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.616951  2854 consensus_queue.cc:1048] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 13:12:33.620180  2528 catalog_manager.cc:5697] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "bf69c09edabf4c568ab360ffa1755c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:33.619669  2527 catalog_manager.cc:5697] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcffaa88967548fdb390c736bf217b09 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "fcffaa88967548fdb390c736bf217b09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:33.630924  2870 consensus_queue.cc:1048] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:12:33.632911  2854 raft_consensus.cc:493] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:33.633708  2854 raft_consensus.cc:515] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.637208  2854 leader_election.cc:290] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:33.637974  2826 consensus_queue.cc:1048] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:12:33.639660  2779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57abe29d17c14a28b9854d11c1b90c8b" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b" is_pre_election: true
I20260709 13:12:33.640477  2779 raft_consensus.cc:2468] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 0.
I20260709 13:12:33.642019  2670 leader_election.cc:304] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90; no voters: 
I20260709 13:12:33.644449  2631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57abe29d17c14a28b9854d11c1b90c8b" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09" is_pre_election: true
I20260709 13:12:33.643878  2826 raft_consensus.cc:2804] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:33.645568  2631 raft_consensus.cc:2468] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 0.
I20260709 13:12:33.646188  2826 raft_consensus.cc:493] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:33.646641  2826 raft_consensus.cc:3060] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.666023  2826 raft_consensus.cc:515] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.671407  2630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57abe29d17c14a28b9854d11c1b90c8b" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:33.672350  2630 raft_consensus.cc:3060] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.673074  2780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57abe29d17c14a28b9854d11c1b90c8b" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
I20260709 13:12:33.674005  2780 raft_consensus.cc:3060] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.681277  2826 leader_election.cc:290] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:33.687433  2780 raft_consensus.cc:2468] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 1.
I20260709 13:12:33.689275  2670 leader_election.cc:304] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90; no voters: 
I20260709 13:12:33.697381  2897 raft_consensus.cc:2804] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:33.702708  2897 raft_consensus.cc:697] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 1 LEADER]: Becoming Leader. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:33.704282  2897 consensus_queue.cc:237] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.715389  2630 raft_consensus.cc:2468] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 1.
I20260709 13:12:33.731628  2529 catalog_manager.cc:5697] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "bf69c09edabf4c568ab360ffa1755c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:33.742976  2870 consensus_queue.cc:1048] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:33.756379  2870 consensus_queue.cc:1048] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:12:33.765241 24113 ts_manager.cc:295] Set tserver state for fcffaa88967548fdb390c736bf217b09 to MAINTENANCE_MODE
I20260709 13:12:33.766541 24113 tablet_server.cc:179] TabletServer@127.23.140.65:0 shutting down...
I20260709 13:12:33.804454 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:12:33.805480 24113 tablet_replica.cc:333] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09: stopping tablet replica
I20260709 13:12:33.806474 24113 raft_consensus.cc:2243] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:33.806913 24113 raft_consensus.cc:2272] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:33.809598 24113 tablet_replica.cc:333] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09: stopping tablet replica
I20260709 13:12:33.810357 24113 raft_consensus.cc:2243] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:33.811053 24113 pending_rounds.cc:70] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09: Trying to abort 1 pending ops.
I20260709 13:12:33.811237 24113 pending_rounds.cc:77] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7305636878614470656 op_type: NO_OP noop_request { }
I20260709 13:12:33.811564 24113 raft_consensus.cc:2889] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:12:33.811851 24113 raft_consensus.cc:2272] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:33.814493 24113 tablet_replica.cc:333] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09: stopping tablet replica
I20260709 13:12:33.815093 24113 raft_consensus.cc:2243] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:33.815663 24113 pending_rounds.cc:70] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09: Trying to abort 1 pending ops.
I20260709 13:12:33.815860 24113 pending_rounds.cc:77] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7305636878777344000 op_type: NO_OP noop_request { }
I20260709 13:12:33.816114 24113 raft_consensus.cc:2889] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:12:33.816387 24113 raft_consensus.cc:2272] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:33.818847 24113 tablet_replica.cc:333] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09: stopping tablet replica
I20260709 13:12:33.819526 24113 raft_consensus.cc:2243] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:33.820325 24113 pending_rounds.cc:70] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09: Trying to abort 1 pending ops.
I20260709 13:12:33.820567 24113 pending_rounds.cc:77] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7305636877924728832 op_type: NO_OP noop_request { }
I20260709 13:12:33.820854 24113 raft_consensus.cc:2889] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:12:33.821202 24113 raft_consensus.cc:2272] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:33.823747 24113 tablet_replica.cc:333] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09: stopping tablet replica
I20260709 13:12:33.824340 24113 raft_consensus.cc:2243] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:33.824806 24113 raft_consensus.cc:2272] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:33.827039 24113 tablet_replica.cc:333] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09: stopping tablet replica
I20260709 13:12:33.827620 24113 raft_consensus.cc:2243] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:33.828055 24113 raft_consensus.cc:2272] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:33.830466 24113 tablet_replica.cc:333] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09: stopping tablet replica
I20260709 13:12:33.831036 24113 raft_consensus.cc:2243] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:33.831458 24113 raft_consensus.cc:2272] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:33.834035 24113 tablet_replica.cc:333] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09: stopping tablet replica
I20260709 13:12:33.834764 24113 raft_consensus.cc:2243] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:33.835294 24113 raft_consensus.cc:2272] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:33.837882 24113 tablet_replica.cc:333] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09: stopping tablet replica
I20260709 13:12:33.838424 24113 raft_consensus.cc:2243] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:33.838838 24113 raft_consensus.cc:2272] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:33.841217 24113 tablet_replica.cc:333] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09: stopping tablet replica
I20260709 13:12:33.841747 24113 raft_consensus.cc:2243] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:33.842222 24113 raft_consensus.cc:2272] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:33.845875  2894 raft_consensus.cc:493] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:33.846612  2897 consensus_queue.cc:1048] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:12:33.846625  2894 raft_consensus.cc:515] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:33.849648  2894 leader_election.cc:290] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:33.851437  2779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a14307fa2c84dce941e556353a3f429" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b" is_pre_election: true
W20260709 13:12:33.860925  2577 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 13:12:33.861191  2672 leader_election.cc:336] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:12:33.861913  2672 leader_election.cc:304] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf69c09edabf4c568ab360ffa1755c90; no voters: 154ab5545ca94930adbad11a6e38fd7b, fcffaa88967548fdb390c736bf217b09
I20260709 13:12:33.863420  2854 raft_consensus.cc:2749] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:12:33.866678  2672 consensus_peers.cc:597] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:33.881287  2746 consensus_peers.cc:597] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:33.888098 24113 tablet_server.cc:196] TabletServer@127.23.140.65:0 shutdown complete.
I20260709 13:12:33.903151  2706 raft_consensus.cc:3060] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:33.910708  2848 consensus_queue.cc:1048] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:12:34.062960  2672 consensus_peers.cc:597] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:34.087498  2746 consensus_peers.cc:597] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:34.129530  2897 consensus_queue.cc:1048] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:12:34.157696  2672 consensus_peers.cc:597] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:34.178009  2897 consensus_queue.cc:1048] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:12:34.190527  2672 consensus_peers.cc:597] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:34.212139  2746 consensus_peers.cc:597] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:34.861991  2577 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:12:34.980912  2897 raft_consensus.cc:493] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:34.981539  2897 raft_consensus.cc:515] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:34.983238  2897 leader_election.cc:290] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:34.984285  2780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b" is_pre_election: true
I20260709 13:12:34.984933  2780 raft_consensus.cc:2468] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 1.
I20260709 13:12:34.986064  2670 leader_election.cc:304] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90; no voters: 
W20260709 13:12:34.986708  2672 leader_election.cc:336] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111)
I20260709 13:12:34.986832  2897 raft_consensus.cc:2804] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:34.987218  2897 raft_consensus.cc:493] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:34.987543  2897 raft_consensus.cc:3060] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:34.992868  2897 raft_consensus.cc:515] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:34.994438  2897 leader_election.cc:290] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 2 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:34.995513  2780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
I20260709 13:12:34.996133  2780 raft_consensus.cc:3060] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:12:34.997804  2672 leader_election.cc:336] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111)
I20260709 13:12:35.001557  2780 raft_consensus.cc:2468] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 2.
I20260709 13:12:35.002596  2670 leader_election.cc:304] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90; no voters: fcffaa88967548fdb390c736bf217b09
I20260709 13:12:35.003361  2897 raft_consensus.cc:2804] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:35.003823  2897 raft_consensus.cc:697] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Becoming Leader. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:35.004698  2897 consensus_queue.cc:237] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:35.011837  2528 catalog_manager.cc:5697] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 reported cstate change: term changed from 1 to 2, leader changed from fcffaa88967548fdb390c736bf217b09 (127.23.140.65) to bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "bf69c09edabf4c568ab360ffa1755c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:35.516971  2780 raft_consensus.cc:1275] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Refusing update from remote peer bf69c09edabf4c568ab360ffa1755c90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:12:35.518750  2897 consensus_queue.cc:1048] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:12:35.539212  2672 consensus_peers.cc:597] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:35.658618  2897 raft_consensus.cc:493] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:35.659108  2897 raft_consensus.cc:515] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:35.661098  2897 leader_election.cc:290] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:35.662024  2779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd31a8e68cb4d21974cd529b2fb2212" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b" is_pre_election: true
I20260709 13:12:35.662662  2779 raft_consensus.cc:2468] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 1.
I20260709 13:12:35.663667  2670 leader_election.cc:304] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90; no voters: 
I20260709 13:12:35.664407  2897 raft_consensus.cc:2804] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:35.664765  2897 raft_consensus.cc:493] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:35.665094  2897 raft_consensus.cc:3060] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:12:35.664937  2672 leader_election.cc:336] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111)
I20260709 13:12:35.670997  2897 raft_consensus.cc:515] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:35.672638  2897 leader_election.cc:290] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 2 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:35.673656  2779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd31a8e68cb4d21974cd529b2fb2212" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
I20260709 13:12:35.674270  2779 raft_consensus.cc:3060] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:12:35.675945  2672 leader_election.cc:336] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111)
I20260709 13:12:35.679563  2779 raft_consensus.cc:2468] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 2.
I20260709 13:12:35.680466  2670 leader_election.cc:304] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90; no voters: fcffaa88967548fdb390c736bf217b09
I20260709 13:12:35.681097  2897 raft_consensus.cc:2804] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:35.681602  2897 raft_consensus.cc:697] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Becoming Leader. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:35.682408  2897 consensus_queue.cc:237] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:35.688711  2529 catalog_manager.cc:5697] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 reported cstate change: term changed from 1 to 2, leader changed from fcffaa88967548fdb390c736bf217b09 (127.23.140.65) to bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "bf69c09edabf4c568ab360ffa1755c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:35.768539  2870 raft_consensus.cc:493] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:35.768987  2870 raft_consensus.cc:515] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:35.770780  2870 leader_election.cc:290] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
I20260709 13:12:35.771910  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c8084adca149bcbf244d636791e9b2" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90" is_pre_election: true
I20260709 13:12:35.772620  2706 raft_consensus.cc:2468] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 1.
I20260709 13:12:35.773885  2745 leader_election.cc:304] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90; no voters: 
W20260709 13:12:35.774376  2746 leader_election.cc:336] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111)
I20260709 13:12:35.774561  2870 raft_consensus.cc:2804] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:12:35.774982  2870 raft_consensus.cc:493] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:35.775306  2870 raft_consensus.cc:3060] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:35.781563  2870 raft_consensus.cc:515] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:35.783027  2870 leader_election.cc:290] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 2 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
I20260709 13:12:35.784117  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c8084adca149bcbf244d636791e9b2" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
I20260709 13:12:35.784768  2706 raft_consensus.cc:3060] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:12:35.786625  2746 leader_election.cc:336] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111)
I20260709 13:12:35.790165  2706 raft_consensus.cc:2468] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 2.
I20260709 13:12:35.791163  2745 leader_election.cc:304] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90; no voters: fcffaa88967548fdb390c736bf217b09
I20260709 13:12:35.791846  2870 raft_consensus.cc:2804] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:35.793150  2870 raft_consensus.cc:697] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [term 2 LEADER]: Becoming Leader. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Running, Role: LEADER
I20260709 13:12:35.794026  2870 consensus_queue.cc:237] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:35.801258  2529 catalog_manager.cc:5697] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b reported cstate change: term changed from 1 to 2, leader changed from fcffaa88967548fdb390c736bf217b09 (127.23.140.65) to 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67). New cstate: current_term: 2 leader_uuid: "154ab5545ca94930adbad11a6e38fd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: HEALTHY } } }
W20260709 13:12:35.862874  2577 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 13:12:35.963989  2672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111) [suppressed 105 similar messages]
W20260709 13:12:36.092280  2672 consensus_peers.cc:597] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:36.268934  2779 raft_consensus.cc:1275] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Refusing update from remote peer bf69c09edabf4c568ab360ffa1755c90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:12:36.270131  2917 consensus_queue.cc:1048] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:36.312928  2706 raft_consensus.cc:1275] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Refusing update from remote peer 154ab5545ca94930adbad11a6e38fd7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:12:36.314108  2870 consensus_queue.cc:1048] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:12:36.326669  2746 consensus_peers.cc:597] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:36.536573  2746 consensus_peers.cc:597] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:36.583854  2672 consensus_peers.cc:597] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:36.621493  2672 consensus_peers.cc:597] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:36.660455  2672 consensus_peers.cc:597] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:36.688783  2746 consensus_peers.cc:597] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:36.775553  2746 consensus_peers.cc:597] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:36.828150  2672 consensus_peers.cc:597] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:36.863879  2577 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 13:12:37.864945  2577 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 13:12:38.106134  2672 consensus_peers.cc:597] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:38.618816  2746 consensus_peers.cc:597] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:12:38.736529  2672 consensus_peers.cc:597] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:12:38.865921  2577 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 13:12:38.890462  2672 consensus_peers.cc:597] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:12:38.919706 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:12:38.926671  2932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:38.928033  2933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:38.931017  2935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:12:38.933899 24113 server_base.cc:1061] running on GCE node
I20260709 13:12:38.936035 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:12:38.936242 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:12:38.936450 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602758936436 us; error 0 us; skew 500 ppm
I20260709 13:12:38.937053 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:12:38.939762 24113 webserver.cc:533] Webserver started at http://127.23.140.65:39607/ using document root <none> and password file <none>
I20260709 13:12:38.940302 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:12:38.940483 24113 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 13:12:38.944428 24113 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 13:12:38.949750  2940 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
W20260709 13:12:38.950270  2672 consensus_peers.cc:597] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:12:38.950801 24113 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.004s
I20260709 13:12:38.951144 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-0-root
uuid: "fcffaa88967548fdb390c736bf217b09"
format_stamp: "Formatted at 2026-07-09 13:12:32 on dist-test-slave-k73n"
I20260709 13:12:38.951535 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602412999163-24113-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:12:38.983673 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:12:38.985049 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:12:38.986682 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:12:38.990756  2947 ts_tablet_manager.cc:548] Loading tablet metadata (0/10 complete)
W20260709 13:12:39.015285  2746 consensus_peers.cc:597] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:12:39.039696 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (10 total tablets, 10 live tablets)
I20260709 13:12:39.040066 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.050s	user 0.002s	sys 0.000s
I20260709 13:12:39.040356 24113 ts_tablet_manager.cc:600] Registering tablets (0/10 complete)
I20260709 13:12:39.045356  2947 tablet_bootstrap.cc:492] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
I20260709 13:12:39.059937  2947 tablet_bootstrap.cc:492] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 13:12:39.060724  2947 tablet_bootstrap.cc:492] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09: Bootstrap complete.
I20260709 13:12:39.061350  2947 ts_tablet_manager.cc:1403] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.016s	user 0.012s	sys 0.004s
I20260709 13:12:39.063449  2947 raft_consensus.cc:359] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.064116  2947 raft_consensus.cc:740] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:39.064792  2947 consensus_queue.cc:260] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.070004  2947 ts_tablet_manager.cc:1434] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260709 13:12:39.070837  2947 tablet_bootstrap.cc:492] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
W20260709 13:12:39.071676  2672 consensus_peers.cc:597] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:12:39.082697 24113 ts_tablet_manager.cc:616] Registered 10 tablets
I20260709 13:12:39.082697  2947 tablet_bootstrap.cc:492] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:39.083184 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.043s	user 0.043s	sys 0.000s
I20260709 13:12:39.083716  2947 tablet_bootstrap.cc:492] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09: Bootstrap complete.
I20260709 13:12:39.084365  2947 ts_tablet_manager.cc:1403] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.014s	user 0.009s	sys 0.003s
I20260709 13:12:39.086546  2947 raft_consensus.cc:359] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.087087  2947 raft_consensus.cc:740] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:39.087764  2947 consensus_queue.cc:260] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.089421  2947 ts_tablet_manager.cc:1434] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20260709 13:12:39.090091  2947 tablet_bootstrap.cc:492] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
I20260709 13:12:39.102273  2947 tablet_bootstrap.cc:492] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:39.103003  2947 tablet_bootstrap.cc:492] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09: Bootstrap complete.
I20260709 13:12:39.103586  2947 ts_tablet_manager.cc:1403] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.014s	user 0.006s	sys 0.004s
I20260709 13:12:39.106120  2947 raft_consensus.cc:359] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.106693  2947 raft_consensus.cc:740] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:39.107563  2947 consensus_queue.cc:260] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.109761  2947 ts_tablet_manager.cc:1434] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 13:12:39.110730  2947 tablet_bootstrap.cc:492] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
I20260709 13:12:39.126698  2947 tablet_bootstrap.cc:492] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:39.127558  2947 tablet_bootstrap.cc:492] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09: Bootstrap complete.
I20260709 13:12:39.128250  2947 ts_tablet_manager.cc:1403] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.018s	user 0.014s	sys 0.000s
I20260709 13:12:39.131318  2947 raft_consensus.cc:359] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.132117  2947 raft_consensus.cc:740] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:39.132926  2947 consensus_queue.cc:260] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.135594  2947 ts_tablet_manager.cc:1434] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.007s	user 0.009s	sys 0.000s
I20260709 13:12:39.136439  2947 tablet_bootstrap.cc:492] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
W20260709 13:12:39.138514  2746 consensus_peers.cc:597] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:12:39.139711 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.65:37697
I20260709 13:12:39.139815  3008 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.65:37697 every 8 connection(s)
I20260709 13:12:39.149955 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:12:39.195914  2947 tablet_bootstrap.cc:492] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 13:12:39.196982  2947 tablet_bootstrap.cc:492] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09: Bootstrap complete.
I20260709 13:12:39.197597  2947 ts_tablet_manager.cc:1403] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.061s	user 0.044s	sys 0.015s
I20260709 13:12:39.199558  2947 raft_consensus.cc:359] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.200776  2947 raft_consensus.cc:740] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:39.201093  3009 heartbeater.cc:344] Connected to a master server at 127.23.140.126:37955
I20260709 13:12:39.201536  3009 heartbeater.cc:461] Registering TS with master...
I20260709 13:12:39.201536  2947 consensus_queue.cc:260] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.202610  3009 heartbeater.cc:507] Master 127.23.140.126:37955 requested a full tablet report, sending...
I20260709 13:12:39.203725  2947 ts_tablet_manager.cc:1434] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 13:12:39.204571  2947 tablet_bootstrap.cc:492] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
I20260709 13:12:39.209676  2529 ts_manager.cc:194] Re-registered known tserver with Master: fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697)
I20260709 13:12:39.213080  2983 raft_consensus.cc:3060] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:39.223111  2706 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "57abe29d17c14a28b9854d11c1b90c8b"
dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
mode: GRACEFUL
new_leader_uuid: "154ab5545ca94930adbad11a6e38fd7b"
 from {username='slave'} at 127.0.0.1:51890
I20260709 13:12:39.224131  2706 raft_consensus.cc:606] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 1 LEADER]: Received request to transfer leadership to TS 154ab5545ca94930adbad11a6e38fd7b
I20260709 13:12:39.228699  2706 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc"
dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
mode: GRACEFUL
new_leader_uuid: "154ab5545ca94930adbad11a6e38fd7b"
 from {username='slave'} at 127.0.0.1:51890
I20260709 13:12:39.229545  2706 raft_consensus.cc:606] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Received request to transfer leadership to TS 154ab5545ca94930adbad11a6e38fd7b
I20260709 13:12:39.231144 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:12:39.231842  2983 pending_rounds.cc:85] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09: Aborting all ops after (but not including) 0
I20260709 13:12:39.232105 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:12:39.232232  2983 pending_rounds.cc:107] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 13:12:39.232815  2983 raft_consensus.cc:2889] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:12:39.233090 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:12:39.234877  2947 tablet_bootstrap.cc:492] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:39.236069  2947 tablet_bootstrap.cc:492] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09: Bootstrap complete.
I20260709 13:12:39.236856  2947 ts_tablet_manager.cc:1403] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.032s	user 0.013s	sys 0.015s
I20260709 13:12:39.240127  2947 raft_consensus.cc:359] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.242975  2529 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49254
I20260709 13:12:39.241009  2947 raft_consensus.cc:740] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:39.245137  2947 consensus_queue.cc:260] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.246307  3022 raft_consensus.cc:993] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90: : Instructing follower 154ab5545ca94930adbad11a6e38fd7b to start an election
I20260709 13:12:39.246873  3022 raft_consensus.cc:1081] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 1 LEADER]: Signalling peer 154ab5545ca94930adbad11a6e38fd7b to start an election
I20260709 13:12:39.247071  2947 ts_tablet_manager.cc:1434] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20260709 13:12:39.247253  3009 heartbeater.cc:499] Master 127.23.140.126:37955 was elected leader, sending a full tablet report...
I20260709 13:12:39.248067  2947 tablet_bootstrap.cc:492] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
I20260709 13:12:39.249215  2779 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "57abe29d17c14a28b9854d11c1b90c8b"
dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
 from {username='slave'} at 127.0.0.1:51216
I20260709 13:12:39.250056  2779 raft_consensus.cc:493] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:39.250504  2779 raft_consensus.cc:3060] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:39.258105  2779 raft_consensus.cc:515] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.260180  2947 tablet_bootstrap.cc:492] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 13:12:39.261165  2947 tablet_bootstrap.cc:492] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09: Bootstrap complete.
I20260709 13:12:39.261039  2983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57abe29d17c14a28b9854d11c1b90c8b" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:39.261967  2947 ts_tablet_manager.cc:1403] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.014s	user 0.010s	sys 0.003s
I20260709 13:12:39.262037  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57abe29d17c14a28b9854d11c1b90c8b" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
W20260709 13:12:39.262616  2746 leader_election.cc:343] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:39.263064  2779 leader_election.cc:290] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 2 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
I20260709 13:12:39.262658  2706 raft_consensus.cc:3055] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:39.263649  2706 raft_consensus.cc:740] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:39.264565  2947 raft_consensus.cc:359] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.264457  2706 consensus_queue.cc:260] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.265347  2947 raft_consensus.cc:740] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:39.265856  2706 raft_consensus.cc:3060] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:39.265975  2947 consensus_queue.cc:260] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.267565  2947 ts_tablet_manager.cc:1434] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20260709 13:12:39.268179  2947 tablet_bootstrap.cc:492] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
I20260709 13:12:39.271957  2706 raft_consensus.cc:2468] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 2.
I20260709 13:12:39.273180  2745 leader_election.cc:304] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90; no voters: fcffaa88967548fdb390c736bf217b09
I20260709 13:12:39.274030  2930 raft_consensus.cc:2804] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:39.275303  2930 raft_consensus.cc:697] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 2 LEADER]: Becoming Leader. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Running, Role: LEADER
I20260709 13:12:39.276120  2930 consensus_queue.cc:237] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.282174  2947 tablet_bootstrap.cc:492] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:39.283041  2947 tablet_bootstrap.cc:492] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09: Bootstrap complete.
I20260709 13:12:39.283584  2947 ts_tablet_manager.cc:1403] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.016s	user 0.013s	sys 0.000s
I20260709 13:12:39.284416  2529 catalog_manager.cc:5697] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b reported cstate change: term changed from 1 to 2, leader changed from bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66) to 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67). New cstate: current_term: 2 leader_uuid: "154ab5545ca94930adbad11a6e38fd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:39.285322  2947 raft_consensus.cc:359] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.285853  2947 raft_consensus.cc:740] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:39.286427  2947 consensus_queue.cc:260] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.288255  2947 ts_tablet_manager.cc:1434] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20260709 13:12:39.289016  2947 tablet_bootstrap.cc:492] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
I20260709 13:12:39.302206  2947 tablet_bootstrap.cc:492] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:39.302970  2947 tablet_bootstrap.cc:492] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09: Bootstrap complete.
I20260709 13:12:39.303483  2947 ts_tablet_manager.cc:1403] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.015s	user 0.012s	sys 0.000s
I20260709 13:12:39.305346  2947 raft_consensus.cc:359] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } }
I20260709 13:12:39.305783  2947 raft_consensus.cc:740] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:39.306257  2947 consensus_queue.cc:260] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } }
I20260709 13:12:39.307915  2947 ts_tablet_manager.cc:1434] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20260709 13:12:39.308669  2947 tablet_bootstrap.cc:492] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09: Bootstrap starting.
I20260709 13:12:39.320773  2947 tablet_bootstrap.cc:492] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:12:39.321480  2947 tablet_bootstrap.cc:492] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09: Bootstrap complete.
I20260709 13:12:39.322017  2947 ts_tablet_manager.cc:1403] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09: Time spent bootstrapping tablet: real 0.014s	user 0.012s	sys 0.000s
I20260709 13:12:39.323755  2947 raft_consensus.cc:359] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.324258  2947 raft_consensus.cc:740] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcffaa88967548fdb390c736bf217b09, State: Initialized, Role: FOLLOWER
I20260709 13:12:39.324816  2947 consensus_queue.cc:260] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.326537  2947 ts_tablet_manager.cc:1434] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20260709 13:12:39.335325  2983 raft_consensus.cc:3060] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:39.340703  2983 pending_rounds.cc:85] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09: Aborting all ops after (but not including) 0
I20260709 13:12:39.340898  2983 pending_rounds.cc:107] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 13:12:39.341073  2983 raft_consensus.cc:2889] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:12:39.495762  2983 raft_consensus.cc:3060] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:39.501279  2983 pending_rounds.cc:85] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09: Aborting all ops after (but not including) 0
I20260709 13:12:39.501557  2983 pending_rounds.cc:107] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 13:12:39.501808  2983 raft_consensus.cc:2889] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:12:39.680122  2706 raft_consensus.cc:1275] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Refusing update from remote peer 154ab5545ca94930adbad11a6e38fd7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:39.681571  3039 consensus_queue.cc:1048] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:39.683840  3022 raft_consensus.cc:993] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90: : Instructing follower 154ab5545ca94930adbad11a6e38fd7b to start an election
I20260709 13:12:39.684337  2897 raft_consensus.cc:1081] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Signalling peer 154ab5545ca94930adbad11a6e38fd7b to start an election
I20260709 13:12:39.685774  2779 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc"
dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
 from {username='slave'} at 127.0.0.1:51216
I20260709 13:12:39.686391  2779 raft_consensus.cc:493] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:39.686753  2779 raft_consensus.cc:3060] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:39.709982  2982 raft_consensus.cc:3060] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:39.716019  2779 raft_consensus.cc:515] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.722602  2982 raft_consensus.cc:1275] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Refusing update from remote peer 154ab5545ca94930adbad11a6e38fd7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:39.724337  3039 consensus_queue.cc:1048] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:39.724967  2982 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:39.726351  2982 raft_consensus.cc:3060] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:39.725816  2779 leader_election.cc:290] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 3 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
I20260709 13:12:39.727016  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
I20260709 13:12:39.729718  2706 raft_consensus.cc:3055] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:39.730156  2706 raft_consensus.cc:740] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:39.731127  2706 consensus_queue.cc:260] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.735514  2706 raft_consensus.cc:3060] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:39.741811  2706 raft_consensus.cc:2468] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 3.
I20260709 13:12:39.742585  2982 raft_consensus.cc:2468] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 3.
I20260709 13:12:39.742700  2745 leader_election.cc:304] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90; no voters: 
I20260709 13:12:39.743690  3039 raft_consensus.cc:2804] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:39.744150  3039 raft_consensus.cc:697] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 3 LEADER]: Becoming Leader. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Running, Role: LEADER
I20260709 13:12:39.745062  3039 consensus_queue.cc:237] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:39.753629  2529 catalog_manager.cc:5697] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b reported cstate change: term changed from 2 to 3, leader changed from bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66) to 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67). New cstate: current_term: 3 leader_uuid: "154ab5545ca94930adbad11a6e38fd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: HEALTHY } } }
W20260709 13:12:39.866890  2577 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:12:40.245185 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:12:40.253374  2705 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4116b417de684d87aed71c2ec197c7eb"
dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
mode: GRACEFUL
new_leader_uuid: "154ab5545ca94930adbad11a6e38fd7b"
 from {username='slave'} at 127.0.0.1:51890
I20260709 13:12:40.254014  2705 raft_consensus.cc:606] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 1 LEADER]: Received request to transfer leadership to TS 154ab5545ca94930adbad11a6e38fd7b
I20260709 13:12:40.266299  2779 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc"
dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
mode: GRACEFUL
new_leader_uuid: "bf69c09edabf4c568ab360ffa1755c90"
 from {username='slave'} at 127.0.0.1:42776
I20260709 13:12:40.266858  2779 raft_consensus.cc:606] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 3 LEADER]: Received request to transfer leadership to TS bf69c09edabf4c568ab360ffa1755c90
I20260709 13:12:40.269354  2779 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ca9c690004a0496ebd5c127f5d82e22a"
dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
mode: GRACEFUL
new_leader_uuid: "bf69c09edabf4c568ab360ffa1755c90"
 from {username='slave'} at 127.0.0.1:42776
I20260709 13:12:40.269878  2779 raft_consensus.cc:606] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 1 LEADER]: Received request to transfer leadership to TS bf69c09edabf4c568ab360ffa1755c90
I20260709 13:12:40.270891 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:12:40.271337 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:12:40.271853 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:12:40.284401  2705 raft_consensus.cc:1275] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Refusing update from remote peer 154ab5545ca94930adbad11a6e38fd7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:40.285722  2982 raft_consensus.cc:1275] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Refusing update from remote peer 154ab5545ca94930adbad11a6e38fd7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:40.285913  2930 consensus_queue.cc:1048] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:40.287175  3039 consensus_queue.cc:1048] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:40.301707  3025 raft_consensus.cc:993] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b: : Instructing follower bf69c09edabf4c568ab360ffa1755c90 to start an election
I20260709 13:12:40.303833  3039 raft_consensus.cc:1081] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 3 LEADER]: Signalling peer bf69c09edabf4c568ab360ffa1755c90 to start an election
I20260709 13:12:40.305855  2705 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc"
dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
 from {username='slave'} at 127.0.0.1:49738
I20260709 13:12:40.306483  2705 raft_consensus.cc:493] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:40.306867  2705 raft_consensus.cc:3060] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:40.314079  2705 raft_consensus.cc:515] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:40.314877  2706 raft_consensus.cc:1240] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 4 FOLLOWER]: Rejecting Update request from peer 154ab5545ca94930adbad11a6e38fd7b for earlier term 3. Current term is 4. Ops: []
I20260709 13:12:40.315757  2705 leader_election.cc:290] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 4 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:40.316668  2983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:40.316876  2779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
I20260709 13:12:40.317458  2983 raft_consensus.cc:3060] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:40.317543  2779 raft_consensus.cc:3055] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:12:40.318005  2779 raft_consensus.cc:740] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Running, Role: LEADER
I20260709 13:12:40.318321  3052 consensus_queue.cc:1059] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:12:40.319528  2779 consensus_queue.cc:260] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:40.320807  2779 raft_consensus.cc:3060] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:40.324697  2983 raft_consensus.cc:2468] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 4.
I20260709 13:12:40.326038  2672 leader_election.cc:304] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf69c09edabf4c568ab360ffa1755c90, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:40.326831  3022 raft_consensus.cc:2804] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:12:40.327359  3022 raft_consensus.cc:697] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 4 LEADER]: Becoming Leader. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:40.327937  2779 raft_consensus.cc:2468] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 4.
W20260709 13:12:40.328454  3025 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260709 13:12:40.328234  3022 consensus_queue.cc:237] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:40.335412  2529 catalog_manager.cc:5697] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 reported cstate change: term changed from 3 to 4, leader changed from 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67) to bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66). New cstate: current_term: 4 leader_uuid: "bf69c09edabf4c568ab360ffa1755c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:40.481652  3052 raft_consensus.cc:993] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b: : Instructing follower bf69c09edabf4c568ab360ffa1755c90 to start an election
I20260709 13:12:40.482064  3025 raft_consensus.cc:1081] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 1 LEADER]: Signalling peer bf69c09edabf4c568ab360ffa1755c90 to start an election
I20260709 13:12:40.483439  2706 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ca9c690004a0496ebd5c127f5d82e22a"
dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
 from {username='slave'} at 127.0.0.1:49738
I20260709 13:12:40.483839  2706 raft_consensus.cc:493] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:40.484053  2706 raft_consensus.cc:3060] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:40.488224  2706 raft_consensus.cc:515] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:40.489570  2706 leader_election.cc:290] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 2 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:40.490383  2983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca9c690004a0496ebd5c127f5d82e22a" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:40.490774  2779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca9c690004a0496ebd5c127f5d82e22a" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
I20260709 13:12:40.491076  2983 raft_consensus.cc:3060] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:40.491381  2779 raft_consensus.cc:3055] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:40.491703  2779 raft_consensus.cc:740] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Running, Role: LEADER
I20260709 13:12:40.492599  2779 consensus_queue.cc:260] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:40.493854  2779 raft_consensus.cc:3060] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:40.498147  2983 raft_consensus.cc:2468] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 2.
I20260709 13:12:40.499378  2672 leader_election.cc:304] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf69c09edabf4c568ab360ffa1755c90, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:40.500020  3022 raft_consensus.cc:2804] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:40.500499  3022 raft_consensus.cc:697] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Becoming Leader. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:40.500672  2779 raft_consensus.cc:2468] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 2.
I20260709 13:12:40.501435  3022 consensus_queue.cc:237] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:40.508013  2529 catalog_manager.cc:5697] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 reported cstate change: term changed from 1 to 2, leader changed from 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67) to bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "bf69c09edabf4c568ab360ffa1755c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:40.657902  3045 raft_consensus.cc:993] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90: : Instructing follower 154ab5545ca94930adbad11a6e38fd7b to start an election
I20260709 13:12:40.658417  3022 raft_consensus.cc:1081] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 1 LEADER]: Signalling peer 154ab5545ca94930adbad11a6e38fd7b to start an election
I20260709 13:12:40.659860  2779 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4116b417de684d87aed71c2ec197c7eb"
dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
 from {username='slave'} at 127.0.0.1:51216
I20260709 13:12:40.660332  2779 raft_consensus.cc:493] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:40.660629  2779 raft_consensus.cc:3060] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:40.665390  2779 raft_consensus.cc:515] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:40.666970  2779 leader_election.cc:290] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 2 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
I20260709 13:12:40.668109  2983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4116b417de684d87aed71c2ec197c7eb" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:40.668129  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4116b417de684d87aed71c2ec197c7eb" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
I20260709 13:12:40.668905  2983 raft_consensus.cc:3060] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:40.668991  2706 raft_consensus.cc:3055] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:40.669523  2706 raft_consensus.cc:740] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:40.670219  2706 consensus_queue.cc:260] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:40.671308  2706 raft_consensus.cc:3060] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:40.674918  2983 raft_consensus.cc:2468] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 2.
I20260709 13:12:40.675832  2706 raft_consensus.cc:2468] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 2.
I20260709 13:12:40.676056  2746 leader_election.cc:304] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:40.676945  3025 raft_consensus.cc:2804] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:40.677439  3025 raft_consensus.cc:697] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [term 2 LEADER]: Becoming Leader. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Running, Role: LEADER
I20260709 13:12:40.678149  3025 consensus_queue.cc:237] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:40.684592  2529 catalog_manager.cc:5697] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b reported cstate change: term changed from 1 to 2, leader changed from bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66) to 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67). New cstate: current_term: 2 leader_uuid: "154ab5545ca94930adbad11a6e38fd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:40.708571  2779 raft_consensus.cc:1275] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 4 FOLLOWER]: Refusing update from remote peer bf69c09edabf4c568ab360ffa1755c90: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:12:40.709816  3022 consensus_queue.cc:1048] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:40.718957  2983 raft_consensus.cc:1275] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 4 FOLLOWER]: Refusing update from remote peer bf69c09edabf4c568ab360ffa1755c90: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:12:40.720471  3045 consensus_queue.cc:1048] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 13:12:40.867858  2577 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:12:40.913005  2983 raft_consensus.cc:1275] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Refusing update from remote peer bf69c09edabf4c568ab360ffa1755c90: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:40.914484  3045 consensus_queue.cc:1048] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:40.920253  2779 raft_consensus.cc:1275] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Refusing update from remote peer bf69c09edabf4c568ab360ffa1755c90: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:40.922084  3022 consensus_queue.cc:1048] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:41.123783  2706 raft_consensus.cc:1275] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Refusing update from remote peer 154ab5545ca94930adbad11a6e38fd7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:41.124867  3025 consensus_queue.cc:1048] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:41.135459  2983 raft_consensus.cc:1275] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Refusing update from remote peer 154ab5545ca94930adbad11a6e38fd7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:41.137218  3052 consensus_queue.cc:1048] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:41.278180 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:12:41.285326  2706 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "47cba406eeea448d92432c1ecb287268"
dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
mode: GRACEFUL
new_leader_uuid: "154ab5545ca94930adbad11a6e38fd7b"
 from {username='slave'} at 127.0.0.1:51890
I20260709 13:12:41.285787  2706 raft_consensus.cc:606] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 1 LEADER]: Received request to transfer leadership to TS 154ab5545ca94930adbad11a6e38fd7b
I20260709 13:12:41.288260  2779 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "57abe29d17c14a28b9854d11c1b90c8b"
dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
mode: GRACEFUL
new_leader_uuid: "bf69c09edabf4c568ab360ffa1755c90"
 from {username='slave'} at 127.0.0.1:42776
I20260709 13:12:41.288779  2779 raft_consensus.cc:606] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 2 LEADER]: Received request to transfer leadership to TS bf69c09edabf4c568ab360ffa1755c90
I20260709 13:12:41.291034  2706 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc"
dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
mode: GRACEFUL
new_leader_uuid: "154ab5545ca94930adbad11a6e38fd7b"
 from {username='slave'} at 127.0.0.1:51890
I20260709 13:12:41.291499  2706 raft_consensus.cc:606] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 4 LEADER]: Received request to transfer leadership to TS 154ab5545ca94930adbad11a6e38fd7b
I20260709 13:12:41.292483 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:12:41.292981 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:12:41.293601 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:12:41.318152  3059 raft_consensus.cc:993] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90: : Instructing follower 154ab5545ca94930adbad11a6e38fd7b to start an election
I20260709 13:12:41.318743  3062 raft_consensus.cc:1081] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 4 LEADER]: Signalling peer 154ab5545ca94930adbad11a6e38fd7b to start an election
I20260709 13:12:41.320277  2779 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc"
dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
 from {username='slave'} at 127.0.0.1:51216
I20260709 13:12:41.320695  2779 raft_consensus.cc:493] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:41.320914  2779 raft_consensus.cc:3060] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:12:41.325248  2779 raft_consensus.cc:515] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:41.326522  2779 leader_election.cc:290] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 5 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
I20260709 13:12:41.327436  2983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:41.327531  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90a4191a9d844e9daed17f0fca5cb8dc" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
I20260709 13:12:41.328153  2983 raft_consensus.cc:3060] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:12:41.328275  2706 raft_consensus.cc:3055] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:12:41.328639  2706 raft_consensus.cc:740] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:41.329288  2706 consensus_queue.cc:260] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:41.330137  2706 raft_consensus.cc:3060] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:12:41.333093  2983 raft_consensus.cc:2468] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 5.
I20260709 13:12:41.334496  2746 leader_election.cc:304] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:41.334925  2706 raft_consensus.cc:2468] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 5.
I20260709 13:12:41.335399  3052 raft_consensus.cc:2804] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:12:41.335896  3052 raft_consensus.cc:697] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 5 LEADER]: Becoming Leader. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Running, Role: LEADER
I20260709 13:12:41.336623  3052 consensus_queue.cc:237] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:41.342401  2528 catalog_manager.cc:5697] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b reported cstate change: term changed from 4 to 5, leader changed from bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66) to 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67). New cstate: current_term: 5 leader_uuid: "154ab5545ca94930adbad11a6e38fd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:41.671073  3065 raft_consensus.cc:993] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b: : Instructing follower bf69c09edabf4c568ab360ffa1755c90 to start an election
I20260709 13:12:41.671532  3052 raft_consensus.cc:1081] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 2 LEADER]: Signalling peer bf69c09edabf4c568ab360ffa1755c90 to start an election
I20260709 13:12:41.672904  2706 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "57abe29d17c14a28b9854d11c1b90c8b"
dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
 from {username='slave'} at 127.0.0.1:49738
I20260709 13:12:41.673413  2706 raft_consensus.cc:493] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:41.673684  2706 raft_consensus.cc:3060] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:41.678076  2706 raft_consensus.cc:515] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:41.679503  2706 leader_election.cc:290] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 3 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:41.680361  2983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57abe29d17c14a28b9854d11c1b90c8b" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:41.680545  2779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57abe29d17c14a28b9854d11c1b90c8b" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
I20260709 13:12:41.681020  2983 raft_consensus.cc:3060] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:41.681221  2779 raft_consensus.cc:3055] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:41.681635  2779 raft_consensus.cc:740] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Running, Role: LEADER
I20260709 13:12:41.682339  2779 consensus_queue.cc:260] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:41.683502  2779 raft_consensus.cc:3060] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:41.686700  2983 raft_consensus.cc:2468] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 3.
I20260709 13:12:41.687724  2672 leader_election.cc:304] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf69c09edabf4c568ab360ffa1755c90, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:41.688395  3062 raft_consensus.cc:2804] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:41.688794  3062 raft_consensus.cc:697] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 3 LEADER]: Becoming Leader. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:41.688951  2779 raft_consensus.cc:2468] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 3.
I20260709 13:12:41.689558  3062 consensus_queue.cc:237] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:41.695793  2528 catalog_manager.cc:5697] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 reported cstate change: term changed from 2 to 3, leader changed from 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67) to bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66). New cstate: current_term: 3 leader_uuid: "bf69c09edabf4c568ab360ffa1755c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:41.727018  3059 raft_consensus.cc:993] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90: : Instructing follower 154ab5545ca94930adbad11a6e38fd7b to start an election
I20260709 13:12:41.727442  3062 raft_consensus.cc:1081] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 1 LEADER]: Signalling peer 154ab5545ca94930adbad11a6e38fd7b to start an election
I20260709 13:12:41.728853  2779 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "47cba406eeea448d92432c1ecb287268"
dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
 from {username='slave'} at 127.0.0.1:51216
I20260709 13:12:41.729382  2779 raft_consensus.cc:493] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:41.729697  2779 raft_consensus.cc:3060] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:41.730335  2983 raft_consensus.cc:1275] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 5 FOLLOWER]: Refusing update from remote peer 154ab5545ca94930adbad11a6e38fd7b: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:12:41.731628  3052 consensus_queue.cc:1048] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:12:41.734808  2779 raft_consensus.cc:515] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:41.737573  2779 leader_election.cc:290] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 2 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
I20260709 13:12:41.737622  2982 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47cba406eeea448d92432c1ecb287268" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:41.738354  2982 raft_consensus.cc:3060] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:41.739605  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47cba406eeea448d92432c1ecb287268" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
I20260709 13:12:41.740156  2706 raft_consensus.cc:3055] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:41.740448  2706 raft_consensus.cc:740] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:41.741144  2706 consensus_queue.cc:260] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:41.742122  2706 raft_consensus.cc:3060] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:41.745692  2982 raft_consensus.cc:2468] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 2.
I20260709 13:12:41.747033  2746 leader_election.cc:304] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:41.747938  3052 raft_consensus.cc:2804] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:41.750578  2706 raft_consensus.cc:2468] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 2.
I20260709 13:12:41.751566  3052 raft_consensus.cc:697] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 2 LEADER]: Becoming Leader. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Running, Role: LEADER
I20260709 13:12:41.752646  3052 consensus_queue.cc:237] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:41.756306  2706 raft_consensus.cc:1275] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 5 FOLLOWER]: Refusing update from remote peer 154ab5545ca94930adbad11a6e38fd7b: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:12:41.761914  3077 consensus_queue.cc:1048] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:12:41.770141  2528 catalog_manager.cc:5697] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b reported cstate change: term changed from 1 to 2, leader changed from bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66) to 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67). New cstate: current_term: 2 leader_uuid: "154ab5545ca94930adbad11a6e38fd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: HEALTHY } } }
W20260709 13:12:41.868803  2577 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:12:42.141448  2779 raft_consensus.cc:1275] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Refusing update from remote peer bf69c09edabf4c568ab360ffa1755c90: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:42.142668  3062 consensus_queue.cc:1048] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:42.151137  2980 raft_consensus.cc:1275] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Refusing update from remote peer bf69c09edabf4c568ab360ffa1755c90: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:42.152721  3062 consensus_queue.cc:1048] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:42.209432  2980 raft_consensus.cc:1275] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Refusing update from remote peer 154ab5545ca94930adbad11a6e38fd7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:42.210691  3076 consensus_queue.cc:1048] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:42.219851  2706 raft_consensus.cc:1275] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Refusing update from remote peer 154ab5545ca94930adbad11a6e38fd7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:42.222668  3076 consensus_queue.cc:1048] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:42.299716 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:12:42.306807  2779 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "47cba406eeea448d92432c1ecb287268"
dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
mode: GRACEFUL
new_leader_uuid: "bf69c09edabf4c568ab360ffa1755c90"
 from {username='slave'} at 127.0.0.1:42776
I20260709 13:12:42.307267  2779 raft_consensus.cc:606] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 2 LEADER]: Received request to transfer leadership to TS bf69c09edabf4c568ab360ffa1755c90
I20260709 13:12:42.309531  2779 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "794a8d3f15854076abde75e5626686b0"
dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
mode: GRACEFUL
new_leader_uuid: "bf69c09edabf4c568ab360ffa1755c90"
 from {username='slave'} at 127.0.0.1:42776
I20260709 13:12:42.309914  2779 raft_consensus.cc:606] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 1 LEADER]: Received request to transfer leadership to TS bf69c09edabf4c568ab360ffa1755c90
I20260709 13:12:42.312156  2706 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ca9c690004a0496ebd5c127f5d82e22a"
dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
mode: GRACEFUL
new_leader_uuid: "154ab5545ca94930adbad11a6e38fd7b"
 from {username='slave'} at 127.0.0.1:51890
I20260709 13:12:42.312716  2706 raft_consensus.cc:606] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Received request to transfer leadership to TS 154ab5545ca94930adbad11a6e38fd7b
I20260709 13:12:42.313864 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:12:42.314294 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:12:42.314934 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:12:42.636154  3059 raft_consensus.cc:993] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90: : Instructing follower 154ab5545ca94930adbad11a6e38fd7b to start an election
I20260709 13:12:42.636686  3062 raft_consensus.cc:1081] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Signalling peer 154ab5545ca94930adbad11a6e38fd7b to start an election
I20260709 13:12:42.637977  2779 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ca9c690004a0496ebd5c127f5d82e22a"
dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
 from {username='slave'} at 127.0.0.1:51216
I20260709 13:12:42.638389  2779 raft_consensus.cc:493] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:42.638607  2779 raft_consensus.cc:3060] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:42.642863  2779 raft_consensus.cc:515] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:42.644379  2779 leader_election.cc:290] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 3 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
I20260709 13:12:42.645092  2980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca9c690004a0496ebd5c127f5d82e22a" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:42.645394  2706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca9c690004a0496ebd5c127f5d82e22a" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
I20260709 13:12:42.645689  2980 raft_consensus.cc:3060] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:42.645951  2706 raft_consensus.cc:3055] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:42.646250  2706 raft_consensus.cc:740] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:42.646790  2706 consensus_queue.cc:260] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:42.647610  2706 raft_consensus.cc:3060] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:42.651564  2980 raft_consensus.cc:2468] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 3.
I20260709 13:12:42.652535  2706 raft_consensus.cc:2468] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 3.
I20260709 13:12:42.653712  2745 leader_election.cc:304] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, bf69c09edabf4c568ab360ffa1755c90; no voters: 
I20260709 13:12:42.654515  3076 raft_consensus.cc:2804] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:42.654999  3076 raft_consensus.cc:697] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 3 LEADER]: Becoming Leader. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Running, Role: LEADER
I20260709 13:12:42.656140  3076 consensus_queue.cc:237] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:42.664956  2528 catalog_manager.cc:5697] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b reported cstate change: term changed from 2 to 3, leader changed from bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66) to 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67). New cstate: current_term: 3 leader_uuid: "154ab5545ca94930adbad11a6e38fd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:42.696834  3065 raft_consensus.cc:993] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b: : Instructing follower bf69c09edabf4c568ab360ffa1755c90 to start an election
I20260709 13:12:42.697350  3065 raft_consensus.cc:1081] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 2 LEADER]: Signalling peer bf69c09edabf4c568ab360ffa1755c90 to start an election
I20260709 13:12:42.698971  2706 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "47cba406eeea448d92432c1ecb287268"
dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
 from {username='slave'} at 127.0.0.1:49738
I20260709 13:12:42.699491  2706 raft_consensus.cc:493] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:42.699822  2706 raft_consensus.cc:3060] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:42.704161  2706 raft_consensus.cc:515] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:42.705647  2706 leader_election.cc:290] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 3 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:42.706370  2980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47cba406eeea448d92432c1ecb287268" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:42.706609  2779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47cba406eeea448d92432c1ecb287268" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
I20260709 13:12:42.706949  2980 raft_consensus.cc:3060] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:42.707185  2779 raft_consensus.cc:3055] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:42.707494  2779 raft_consensus.cc:740] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Running, Role: LEADER
I20260709 13:12:42.708104  2779 consensus_queue.cc:260] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:42.708961  2779 raft_consensus.cc:3060] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:42.711865  2980 raft_consensus.cc:2468] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 3.
I20260709 13:12:42.712901  2672 leader_election.cc:304] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf69c09edabf4c568ab360ffa1755c90, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:42.713254  2779 raft_consensus.cc:2468] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 3.
I20260709 13:12:42.713768  3062 raft_consensus.cc:2804] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:42.714228  3062 raft_consensus.cc:697] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 3 LEADER]: Becoming Leader. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:42.714897  3062 consensus_queue.cc:237] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:42.720923  2528 catalog_manager.cc:5697] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 reported cstate change: term changed from 2 to 3, leader changed from 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67) to bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66). New cstate: current_term: 3 leader_uuid: "bf69c09edabf4c568ab360ffa1755c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:42.787338  3076 raft_consensus.cc:993] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b: : Instructing follower bf69c09edabf4c568ab360ffa1755c90 to start an election
I20260709 13:12:42.787807  3065 raft_consensus.cc:1081] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 1 LEADER]: Signalling peer bf69c09edabf4c568ab360ffa1755c90 to start an election
I20260709 13:12:42.789120  2706 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "794a8d3f15854076abde75e5626686b0"
dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
 from {username='slave'} at 127.0.0.1:49738
I20260709 13:12:42.789593  2706 raft_consensus.cc:493] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:42.789873  2706 raft_consensus.cc:3060] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:42.794255  2706 raft_consensus.cc:515] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } }
I20260709 13:12:42.796025  2706 leader_election.cc:290] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 2 election: Requested vote from peers 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839), fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697)
I20260709 13:12:42.796415  2779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "794a8d3f15854076abde75e5626686b0" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
I20260709 13:12:42.796708  2980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "794a8d3f15854076abde75e5626686b0" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:42.797044  2779 raft_consensus.cc:3055] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:42.797444  2980 raft_consensus.cc:3060] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:42.797454  2779 raft_consensus.cc:740] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Running, Role: LEADER
I20260709 13:12:42.798317  2779 consensus_queue.cc:260] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } }
I20260709 13:12:42.799196  2779 raft_consensus.cc:3060] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:42.802493  2980 raft_consensus.cc:2468] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 2.
I20260709 13:12:42.803340  2779 raft_consensus.cc:2468] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 2.
I20260709 13:12:42.803575  2672 leader_election.cc:304] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf69c09edabf4c568ab360ffa1755c90, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:42.804369  3062 raft_consensus.cc:2804] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:42.804756  3062 raft_consensus.cc:697] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Becoming Leader. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:42.805430  3062 consensus_queue.cc:237] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } }
I20260709 13:12:42.812112  2528 catalog_manager.cc:5697] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 reported cstate change: term changed from 1 to 2, leader changed from 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67) to bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "bf69c09edabf4c568ab360ffa1755c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } }
W20260709 13:12:42.869863  2577 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:12:43.058917  2706 raft_consensus.cc:1275] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Refusing update from remote peer 154ab5545ca94930adbad11a6e38fd7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:43.060098  3065 consensus_queue.cc:1048] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:43.069926  2980 raft_consensus.cc:1275] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Refusing update from remote peer 154ab5545ca94930adbad11a6e38fd7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:43.071425  3076 consensus_queue.cc:1048] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:43.209147  2779 raft_consensus.cc:1275] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Refusing update from remote peer bf69c09edabf4c568ab360ffa1755c90: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:43.210320  3085 consensus_queue.cc:1048] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:43.217242  2980 raft_consensus.cc:1275] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Refusing update from remote peer bf69c09edabf4c568ab360ffa1755c90: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:43.218657  3085 consensus_queue.cc:1048] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:43.225200  2983 raft_consensus.cc:1275] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Refusing update from remote peer bf69c09edabf4c568ab360ffa1755c90: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:43.226711  3062 consensus_queue.cc:1048] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:43.236896  2779 raft_consensus.cc:1275] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Refusing update from remote peer bf69c09edabf4c568ab360ffa1755c90: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:43.239701  3062 consensus_queue.cc:1048] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:43.321868 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:12:43.328856  2705 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "794a8d3f15854076abde75e5626686b0"
dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
mode: GRACEFUL
new_leader_uuid: "154ab5545ca94930adbad11a6e38fd7b"
 from {username='slave'} at 127.0.0.1:51890
I20260709 13:12:43.329479  2705 raft_consensus.cc:606] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Received request to transfer leadership to TS 154ab5545ca94930adbad11a6e38fd7b
I20260709 13:12:43.331831  2779 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ca9c690004a0496ebd5c127f5d82e22a"
dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
mode: GRACEFUL
new_leader_uuid: "bf69c09edabf4c568ab360ffa1755c90"
 from {username='slave'} at 127.0.0.1:42776
I20260709 13:12:43.332281  2779 raft_consensus.cc:606] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 3 LEADER]: Received request to transfer leadership to TS bf69c09edabf4c568ab360ffa1755c90
I20260709 13:12:43.334498  2705 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cbd31a8e68cb4d21974cd529b2fb2212"
dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
mode: GRACEFUL
new_leader_uuid: "154ab5545ca94930adbad11a6e38fd7b"
 from {username='slave'} at 127.0.0.1:51890
I20260709 13:12:43.334997  2705 raft_consensus.cc:606] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Received request to transfer leadership to TS 154ab5545ca94930adbad11a6e38fd7b
I20260709 13:12:43.336018 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:12:43.336491 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:12:43.337038 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:12:43.498114  3076 raft_consensus.cc:993] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b: : Instructing follower bf69c09edabf4c568ab360ffa1755c90 to start an election
I20260709 13:12:43.498684  3075 raft_consensus.cc:1081] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 3 LEADER]: Signalling peer bf69c09edabf4c568ab360ffa1755c90 to start an election
I20260709 13:12:43.500315  2705 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ca9c690004a0496ebd5c127f5d82e22a"
dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
 from {username='slave'} at 127.0.0.1:49738
I20260709 13:12:43.500908  2705 raft_consensus.cc:493] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:43.501368  2705 raft_consensus.cc:3060] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:43.506821  2705 raft_consensus.cc:515] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:43.508821  2705 leader_election.cc:290] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 4 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67:41839)
I20260709 13:12:43.509539  2983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca9c690004a0496ebd5c127f5d82e22a" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:43.509807  2780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca9c690004a0496ebd5c127f5d82e22a" candidate_uuid: "bf69c09edabf4c568ab360ffa1755c90" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
I20260709 13:12:43.510232  2983 raft_consensus.cc:3060] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:43.510346  2780 raft_consensus.cc:3055] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:12:43.510795  2780 raft_consensus.cc:740] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Running, Role: LEADER
I20260709 13:12:43.511615  2780 consensus_queue.cc:260] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:43.512880  2780 raft_consensus.cc:3060] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:12:43.518038  2983 raft_consensus.cc:2468] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 4.
I20260709 13:12:43.544759  2672 leader_election.cc:304] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf69c09edabf4c568ab360ffa1755c90, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:43.547302  3085 raft_consensus.cc:2804] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:12:43.549384  3085 raft_consensus.cc:697] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 4 LEADER]: Becoming Leader. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:43.551342  3085 consensus_queue.cc:237] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:43.562566  2780 raft_consensus.cc:2468] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf69c09edabf4c568ab360ffa1755c90 in term 4.
I20260709 13:12:43.564410  2527 catalog_manager.cc:5697] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 reported cstate change: term changed from 3 to 4, leader changed from 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67) to bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66). New cstate: current_term: 4 leader_uuid: "bf69c09edabf4c568ab360ffa1755c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:43.642176  3100 raft_consensus.cc:993] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90: : Instructing follower 154ab5545ca94930adbad11a6e38fd7b to start an election
I20260709 13:12:43.642709  3085 raft_consensus.cc:1081] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Signalling peer 154ab5545ca94930adbad11a6e38fd7b to start an election
I20260709 13:12:43.644496  2780 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "794a8d3f15854076abde75e5626686b0"
dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
 from {username='slave'} at 127.0.0.1:51216
I20260709 13:12:43.645190  2780 raft_consensus.cc:493] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:43.645555  2780 raft_consensus.cc:3060] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:43.651885  2780 raft_consensus.cc:515] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } }
I20260709 13:12:43.654024  2780 leader_election.cc:290] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 3 election: Requested vote from peers bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355), fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697)
I20260709 13:12:43.655400  2983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "794a8d3f15854076abde75e5626686b0" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:43.655573  2705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "794a8d3f15854076abde75e5626686b0" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
I20260709 13:12:43.656232  2983 raft_consensus.cc:3060] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:43.656337  2705 raft_consensus.cc:3055] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:43.656837  2705 raft_consensus.cc:740] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:43.657825  2705 consensus_queue.cc:260] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } }
I20260709 13:12:43.659569  2705 raft_consensus.cc:3060] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:43.663511  2983 raft_consensus.cc:2468] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 3.
I20260709 13:12:43.665076  2746 leader_election.cc:304] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:43.666067  3076 raft_consensus.cc:2804] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:43.666616  3076 raft_consensus.cc:697] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 3 LEADER]: Becoming Leader. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Running, Role: LEADER
I20260709 13:12:43.668251  3076 consensus_queue.cc:237] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } }
I20260709 13:12:43.669305  3100 raft_consensus.cc:993] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90: : Instructing follower 154ab5545ca94930adbad11a6e38fd7b to start an election
I20260709 13:12:43.669950  3085 raft_consensus.cc:1081] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Signalling peer 154ab5545ca94930adbad11a6e38fd7b to start an election
I20260709 13:12:43.670176  2705 raft_consensus.cc:2468] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 3.
I20260709 13:12:43.671877  2780 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cbd31a8e68cb4d21974cd529b2fb2212"
dest_uuid: "154ab5545ca94930adbad11a6e38fd7b"
 from {username='slave'} at 127.0.0.1:51216
I20260709 13:12:43.672667  2780 raft_consensus.cc:493] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:43.673149  2780 raft_consensus.cc:3060] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:43.678215  2527 catalog_manager.cc:5697] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b reported cstate change: term changed from 2 to 3, leader changed from bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66) to 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67). New cstate: current_term: 3 leader_uuid: "154ab5545ca94930adbad11a6e38fd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:43.680116  2780 raft_consensus.cc:515] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:43.682192  2780 leader_election.cc:290] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 3 election: Requested vote from peers fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697), bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355)
I20260709 13:12:43.682955  2983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd31a8e68cb4d21974cd529b2fb2212" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "fcffaa88967548fdb390c736bf217b09"
I20260709 13:12:43.683306  2705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd31a8e68cb4d21974cd529b2fb2212" candidate_uuid: "154ab5545ca94930adbad11a6e38fd7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "bf69c09edabf4c568ab360ffa1755c90"
I20260709 13:12:43.683867  2983 raft_consensus.cc:3060] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:43.683954  2705 raft_consensus.cc:3055] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:43.684441  2705 raft_consensus.cc:740] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bf69c09edabf4c568ab360ffa1755c90, State: Running, Role: LEADER
I20260709 13:12:43.685503  2705 consensus_queue.cc:260] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:43.687222  2705 raft_consensus.cc:3060] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:43.692009  2983 raft_consensus.cc:2468] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 3.
I20260709 13:12:43.692586  2705 raft_consensus.cc:2468] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154ab5545ca94930adbad11a6e38fd7b in term 3.
I20260709 13:12:43.693334  2746 leader_election.cc:304] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154ab5545ca94930adbad11a6e38fd7b, fcffaa88967548fdb390c736bf217b09; no voters: 
I20260709 13:12:43.694249  3076 raft_consensus.cc:2804] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:43.694697  3076 raft_consensus.cc:697] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 3 LEADER]: Becoming Leader. State: Replica: 154ab5545ca94930adbad11a6e38fd7b, State: Running, Role: LEADER
I20260709 13:12:43.695678  3076 consensus_queue.cc:237] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } }
I20260709 13:12:43.702188  2527 catalog_manager.cc:5697] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b reported cstate change: term changed from 2 to 3, leader changed from bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66) to 154ab5545ca94930adbad11a6e38fd7b (127.23.140.67). New cstate: current_term: 3 leader_uuid: "154ab5545ca94930adbad11a6e38fd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 } health_report { overall_health: HEALTHY } } }
W20260709 13:12:43.871196  2577 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:12:44.057483  2983 raft_consensus.cc:1275] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 4 FOLLOWER]: Refusing update from remote peer bf69c09edabf4c568ab360ffa1755c90: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:12:44.058880  3062 consensus_queue.cc:1048] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:12:44.071182  2780 raft_consensus.cc:1275] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 4 FOLLOWER]: Refusing update from remote peer bf69c09edabf4c568ab360ffa1755c90: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:12:44.072783  3062 consensus_queue.cc:1048] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "154ab5545ca94930adbad11a6e38fd7b" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:12:44.147851  2705 raft_consensus.cc:1275] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Refusing update from remote peer 154ab5545ca94930adbad11a6e38fd7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:44.149080  3076 consensus_queue.cc:1048] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:44.166103  2983 raft_consensus.cc:1275] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Refusing update from remote peer 154ab5545ca94930adbad11a6e38fd7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:44.169009  3118 consensus_queue.cc:1048] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:12:44.313812  2980 raft_consensus.cc:1275] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Refusing update from remote peer 154ab5545ca94930adbad11a6e38fd7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:44.315338  2705 raft_consensus.cc:1275] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Refusing update from remote peer 154ab5545ca94930adbad11a6e38fd7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:12:44.315222  3118 consensus_queue.cc:1048] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcffaa88967548fdb390c736bf217b09" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:44.316704  3118 consensus_queue.cc:1048] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf69c09edabf4c568ab360ffa1755c90" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:44.343012 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:12:44.374752 24113 tablet_server.cc:179] TabletServer@127.23.140.65:37697 shutting down...
W20260709 13:12:44.386135  2672 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 65 similar messages]
W20260709 13:12:44.395380  2672 consensus_peers.cc:597] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:44.410830 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260709 13:12:44.410837  2746 consensus_peers.cc:597] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:44.412134 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:44.413389 24113 raft_consensus.cc:2243] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.413908 24113 raft_consensus.cc:2272] T 57abe29d17c14a28b9854d11c1b90c8b P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.416553 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:44.417292 24113 raft_consensus.cc:2243] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.417711 24113 raft_consensus.cc:2272] T cbd31a8e68cb4d21974cd529b2fb2212 P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.420642 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:44.421352 24113 raft_consensus.cc:2243] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.421828 24113 raft_consensus.cc:2272] T 4116b417de684d87aed71c2ec197c7eb P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.424275 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:44.424870 24113 raft_consensus.cc:2243] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.425442 24113 raft_consensus.cc:2272] T 90a4191a9d844e9daed17f0fca5cb8dc P fcffaa88967548fdb390c736bf217b09 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.428534 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:44.429375 24113 raft_consensus.cc:2243] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.429864 24113 raft_consensus.cc:2272] T 794a8d3f15854076abde75e5626686b0 P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.433176 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:44.433766 24113 raft_consensus.cc:2243] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.434434 24113 raft_consensus.cc:2272] T 47cba406eeea448d92432c1ecb287268 P fcffaa88967548fdb390c736bf217b09 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.437011 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:44.437683 24113 raft_consensus.cc:2243] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.438221 24113 raft_consensus.cc:2272] T 3a14307fa2c84dce941e556353a3f429 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.440898 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:44.441535 24113 raft_consensus.cc:2243] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.442025 24113 raft_consensus.cc:2272] T ca9c690004a0496ebd5c127f5d82e22a P fcffaa88967548fdb390c736bf217b09 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.445075 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:44.445717 24113 raft_consensus.cc:2243] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.446261 24113 raft_consensus.cc:2272] T 281527dc3eb54d01b4c21684b06cdb75 P fcffaa88967548fdb390c736bf217b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.448870 24113 tablet_replica.cc:333] stopping tablet replica
I20260709 13:12:44.449468 24113 raft_consensus.cc:2243] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.450009 24113 raft_consensus.cc:2272] T 93c8084adca149bcbf244d636791e9b2 P fcffaa88967548fdb390c736bf217b09 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.474592 24113 tablet_server.cc:196] TabletServer@127.23.140.65:37697 shutdown complete.
I20260709 13:12:44.505656 24113 tablet_server.cc:179] TabletServer@127.23.140.66:0 shutting down...
W20260709 13:12:44.507826  2672 consensus_peers.cc:597] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:44.534144 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:12:44.535435 24113 tablet_replica.cc:333] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90: stopping tablet replica
I20260709 13:12:44.536463 24113 raft_consensus.cc:2243] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 3 LEADER]: Raft consensus shutting down.
W20260709 13:12:44.537788  2745 consensus_peers.cc:597] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b -> Peer bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355): Couldn't send request to peer bf69c09edabf4c568ab360ffa1755c90. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:44.537796 24113 raft_consensus.cc:2272] T 57abe29d17c14a28b9854d11c1b90c8b P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.540597 24113 tablet_replica.cc:333] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90: stopping tablet replica
I20260709 13:12:44.541316 24113 raft_consensus.cc:2243] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.541797 24113 raft_consensus.cc:2272] T 3a14307fa2c84dce941e556353a3f429 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.544127 24113 tablet_replica.cc:333] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90: stopping tablet replica
I20260709 13:12:44.544641 24113 raft_consensus.cc:2243] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.545027 24113 raft_consensus.cc:2272] T cbd31a8e68cb4d21974cd529b2fb2212 P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.547806 24113 tablet_replica.cc:333] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90: stopping tablet replica
I20260709 13:12:44.548400 24113 raft_consensus.cc:2243] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.548918 24113 raft_consensus.cc:2272] T 90a4191a9d844e9daed17f0fca5cb8dc P bf69c09edabf4c568ab360ffa1755c90 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.552253 24113 tablet_replica.cc:333] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90: stopping tablet replica
W20260709 13:12:44.552886  2745 consensus_peers.cc:597] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b -> Peer bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355): Couldn't send request to peer bf69c09edabf4c568ab360ffa1755c90. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:44.552932 24113 raft_consensus.cc:2243] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:44.554100 24113 raft_consensus.cc:2272] T 281527dc3eb54d01b4c21684b06cdb75 P bf69c09edabf4c568ab360ffa1755c90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.557313 24113 tablet_replica.cc:333] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90: stopping tablet replica
I20260709 13:12:44.558043 24113 raft_consensus.cc:2243] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:12:44.558892 24113 raft_consensus.cc:2272] T ca9c690004a0496ebd5c127f5d82e22a P bf69c09edabf4c568ab360ffa1755c90 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:44.560200  2746 consensus_peers.cc:597] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:44.561873 24113 tablet_replica.cc:333] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90: stopping tablet replica
I20260709 13:12:44.562567 24113 raft_consensus.cc:2243] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.563057 24113 raft_consensus.cc:2272] T 4116b417de684d87aed71c2ec197c7eb P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.565411 24113 tablet_replica.cc:333] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90: stopping tablet replica
I20260709 13:12:44.565935 24113 raft_consensus.cc:2243] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:44.566738 24113 raft_consensus.cc:2272] T 47cba406eeea448d92432c1ecb287268 P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.568917 24113 tablet_replica.cc:333] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90: stopping tablet replica
I20260709 13:12:44.569533 24113 raft_consensus.cc:2243] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.569934 24113 raft_consensus.cc:2272] T 794a8d3f15854076abde75e5626686b0 P bf69c09edabf4c568ab360ffa1755c90 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.572412 24113 tablet_replica.cc:333] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90: stopping tablet replica
I20260709 13:12:44.572906 24113 raft_consensus.cc:2243] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.573396 24113 raft_consensus.cc:2272] T 93c8084adca149bcbf244d636791e9b2 P bf69c09edabf4c568ab360ffa1755c90 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.606635 24113 tablet_server.cc:196] TabletServer@127.23.140.66:0 shutdown complete.
W20260709 13:12:44.613615  2745 consensus_peers.cc:597] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b -> Peer bf69c09edabf4c568ab360ffa1755c90 (127.23.140.66:38355): Couldn't send request to peer bf69c09edabf4c568ab360ffa1755c90. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:38355: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:44.630522  2746 consensus_peers.cc:597] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:44.638074 24113 tablet_server.cc:179] TabletServer@127.23.140.67:0 shutting down...
W20260709 13:12:44.659194  2746 consensus_peers.cc:597] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b -> Peer fcffaa88967548fdb390c736bf217b09 (127.23.140.65:37697): Couldn't send request to peer fcffaa88967548fdb390c736bf217b09. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:44.665712 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:12:44.666479 24113 tablet_replica.cc:333] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b: stopping tablet replica
I20260709 13:12:44.667140 24113 raft_consensus.cc:2243] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:44.668097 24113 raft_consensus.cc:2272] T 93c8084adca149bcbf244d636791e9b2 P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.670655 24113 tablet_replica.cc:333] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b: stopping tablet replica
I20260709 13:12:44.671304 24113 raft_consensus.cc:2243] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:44.672283 24113 raft_consensus.cc:2272] T 3a14307fa2c84dce941e556353a3f429 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.674528 24113 tablet_replica.cc:333] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b: stopping tablet replica
I20260709 13:12:44.675098 24113 raft_consensus.cc:2243] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:44.675968 24113 raft_consensus.cc:2272] T 4116b417de684d87aed71c2ec197c7eb P 154ab5545ca94930adbad11a6e38fd7b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.678210 24113 tablet_replica.cc:333] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b: stopping tablet replica
I20260709 13:12:44.678704 24113 raft_consensus.cc:2243] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.679191 24113 raft_consensus.cc:2272] T 57abe29d17c14a28b9854d11c1b90c8b P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.681345 24113 tablet_replica.cc:333] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b: stopping tablet replica
I20260709 13:12:44.681833 24113 raft_consensus.cc:2243] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 5 LEADER]: Raft consensus shutting down.
I20260709 13:12:44.682654 24113 raft_consensus.cc:2272] T 90a4191a9d844e9daed17f0fca5cb8dc P 154ab5545ca94930adbad11a6e38fd7b [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.684724 24113 tablet_replica.cc:333] T 281527dc3eb54d01b4c21684b06cdb75 P 154ab5545ca94930adbad11a6e38fd7b: stopping tablet replica
I20260709 13:12:44.685312 24113 raft_consensus.cc:2243] T 281527dc3eb54d01b4c21684b06cdb75 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.685730 24113 raft_consensus.cc:2272] T 281527dc3eb54d01b4c21684b06cdb75 P 154ab5545ca94930adbad11a6e38fd7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.687800 24113 tablet_replica.cc:333] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b: stopping tablet replica
I20260709 13:12:44.688339 24113 raft_consensus.cc:2243] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.688786 24113 raft_consensus.cc:2272] T ca9c690004a0496ebd5c127f5d82e22a P 154ab5545ca94930adbad11a6e38fd7b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.691309 24113 tablet_replica.cc:333] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b: stopping tablet replica
I20260709 13:12:44.691841 24113 raft_consensus.cc:2243] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:44.692288 24113 raft_consensus.cc:2272] T 47cba406eeea448d92432c1ecb287268 P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.694347 24113 tablet_replica.cc:333] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b: stopping tablet replica
I20260709 13:12:44.694860 24113 raft_consensus.cc:2243] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:44.695618 24113 raft_consensus.cc:2272] T 794a8d3f15854076abde75e5626686b0 P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.698264 24113 tablet_replica.cc:333] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b: stopping tablet replica
I20260709 13:12:44.698788 24113 raft_consensus.cc:2243] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:44.699550 24113 raft_consensus.cc:2272] T cbd31a8e68cb4d21974cd529b2fb2212 P 154ab5545ca94930adbad11a6e38fd7b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.734760 24113 tablet_server.cc:196] TabletServer@127.23.140.67:0 shutdown complete.
I20260709 13:12:44.761896 24113 master.cc:562] Master@127.23.140.126:37955 shutting down...
I20260709 13:12:44.782703 24113 raft_consensus.cc:2243] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:44.783282 24113 raft_consensus.cc:2272] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:44.783622 24113 tablet_replica.cc:333] T 00000000000000000000000000000000 P b5f8c43a369f4475b21ff07bce488069: stopping tablet replica
I20260709 13:12:44.803534 24113 master.cc:584] Master@127.23.140.126:37955 shutdown complete.
[       OK ] LeaderRebalancerTest.TestMaintenanceMode (13131 ms)
[ RUN      ] LeaderRebalancerTest.MultiTableLeaderBalance
I20260709 13:12:44.848296 24113 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.140.126:34333
I20260709 13:12:44.849483 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:12:44.855331  3128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:44.855458  3127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:44.855829  3130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:12:44.856992 24113 server_base.cc:1061] running on GCE node
I20260709 13:12:44.857971 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:12:44.858151 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:12:44.858264 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602764858254 us; error 0 us; skew 500 ppm
I20260709 13:12:44.858762 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:12:44.861024 24113 webserver.cc:533] Webserver started at http://127.23.140.126:46235/ using document root <none> and password file <none>
I20260709 13:12:44.861545 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:12:44.861703 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:12:44.861915 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:12:44.862982 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/master-0-root/instance:
uuid: "dbe21f5caabf41c2abe6770f1e2780af"
format_stamp: "Formatted at 2026-07-09 13:12:44 on dist-test-slave-k73n"
I20260709 13:12:44.867408 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 13:12:44.870615  3135 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:44.871326 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260709 13:12:44.871630 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/master-0-root
uuid: "dbe21f5caabf41c2abe6770f1e2780af"
format_stamp: "Formatted at 2026-07-09 13:12:44 on dist-test-slave-k73n"
I20260709 13:12:44.871896 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:12:44.898063 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:12:44.899276 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:12:44.936816 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.126:34333
I20260709 13:12:44.936919  3186 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.126:34333 every 8 connection(s)
I20260709 13:12:44.940621  3187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:44.952490  3187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af: Bootstrap starting.
I20260709 13:12:44.957525  3187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:44.961633  3187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af: No bootstrap required, opened a new log
I20260709 13:12:44.963718  3187 raft_consensus.cc:359] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe21f5caabf41c2abe6770f1e2780af" member_type: VOTER }
I20260709 13:12:44.964083  3187 raft_consensus.cc:385] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:44.964326  3187 raft_consensus.cc:740] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbe21f5caabf41c2abe6770f1e2780af, State: Initialized, Role: FOLLOWER
I20260709 13:12:44.964872  3187 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe21f5caabf41c2abe6770f1e2780af" member_type: VOTER }
I20260709 13:12:44.965541  3187 raft_consensus.cc:399] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:12:44.965780  3187 raft_consensus.cc:493] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:12:44.966041  3187 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:44.970737  3187 raft_consensus.cc:515] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe21f5caabf41c2abe6770f1e2780af" member_type: VOTER }
I20260709 13:12:44.971235  3187 leader_election.cc:304] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbe21f5caabf41c2abe6770f1e2780af; no voters: 
I20260709 13:12:44.972304  3187 leader_election.cc:290] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:12:44.972703  3190 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:44.974267  3190 raft_consensus.cc:697] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [term 1 LEADER]: Becoming Leader. State: Replica: dbe21f5caabf41c2abe6770f1e2780af, State: Running, Role: LEADER
I20260709 13:12:44.974931  3190 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe21f5caabf41c2abe6770f1e2780af" member_type: VOTER }
I20260709 13:12:44.975157  3187 sys_catalog.cc:565] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:12:44.979938  3192 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbe21f5caabf41c2abe6770f1e2780af. Latest consensus state: current_term: 1 leader_uuid: "dbe21f5caabf41c2abe6770f1e2780af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe21f5caabf41c2abe6770f1e2780af" member_type: VOTER } }
I20260709 13:12:44.980155  3191 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbe21f5caabf41c2abe6770f1e2780af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe21f5caabf41c2abe6770f1e2780af" member_type: VOTER } }
I20260709 13:12:44.980777  3192 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [sys.catalog]: This master's current role is: LEADER
I20260709 13:12:44.980937  3191 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [sys.catalog]: This master's current role is: LEADER
I20260709 13:12:44.983279  3197 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:12:44.988014  3197 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:12:44.991542 24113 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:12:44.996526  3197 catalog_manager.cc:1383] Generated new cluster ID: 10dc5210041d47618014810821710c81
I20260709 13:12:44.996809  3197 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:12:45.024359  3197 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:12:45.025688  3197 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:12:45.038388  3197 catalog_manager.cc:6070] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af: Generated new TSK 0
I20260709 13:12:45.038993  3197 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:12:45.058097 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:12:45.064155  3208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:45.065512  3209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:45.066710  3211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:12:45.067510 24113 server_base.cc:1061] running on GCE node
I20260709 13:12:45.068562 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:12:45.068787 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:12:45.068982 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602765068963 us; error 0 us; skew 500 ppm
I20260709 13:12:45.069589 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:12:45.071899 24113 webserver.cc:533] Webserver started at http://127.23.140.65:40277/ using document root <none> and password file <none>
I20260709 13:12:45.072386 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:12:45.072588 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:12:45.072857 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:12:45.074256 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/ts-0-root/instance:
uuid: "05b65d4b42f94b5c9c3cfc7729165b03"
format_stamp: "Formatted at 2026-07-09 13:12:45 on dist-test-slave-k73n"
I20260709 13:12:45.079085 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 13:12:45.082382  3216 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:45.083109 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 13:12:45.083416 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/ts-0-root
uuid: "05b65d4b42f94b5c9c3cfc7729165b03"
format_stamp: "Formatted at 2026-07-09 13:12:45 on dist-test-slave-k73n"
I20260709 13:12:45.083703 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:12:45.100112 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:12:45.101362 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:12:45.102809 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:12:45.105005 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:12:45.105235 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:45.105423 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:12:45.105559 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:45.144326 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.65:39107
I20260709 13:12:45.144488  3278 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.65:39107 every 8 connection(s)
I20260709 13:12:45.149538 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:12:45.156629  3283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:45.157536  3284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:45.164992  3286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:12:45.165223 24113 server_base.cc:1061] running on GCE node
I20260709 13:12:45.165781  3279 heartbeater.cc:344] Connected to a master server at 127.23.140.126:34333
I20260709 13:12:45.166199  3279 heartbeater.cc:461] Registering TS with master...
I20260709 13:12:45.166522 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:12:45.166783 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:12:45.166956 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602765166935 us; error 0 us; skew 500 ppm
I20260709 13:12:45.166957  3279 heartbeater.cc:507] Master 127.23.140.126:34333 requested a full tablet report, sending...
I20260709 13:12:45.167585 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:12:45.169356  3152 ts_manager.cc:194] Registered new tserver with Master: 05b65d4b42f94b5c9c3cfc7729165b03 (127.23.140.65:39107)
I20260709 13:12:45.170804 24113 webserver.cc:533] Webserver started at http://127.23.140.66:43213/ using document root <none> and password file <none>
I20260709 13:12:45.171505 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:12:45.171716 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:12:45.171902  3152 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59334
I20260709 13:12:45.172031 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:12:45.173707 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/ts-1-root/instance:
uuid: "2bb10e216823411d9acb895921ceb9c1"
format_stamp: "Formatted at 2026-07-09 13:12:45 on dist-test-slave-k73n"
I20260709 13:12:45.178383 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 13:12:45.181635  3291 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:45.182360 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260709 13:12:45.182642 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/ts-1-root
uuid: "2bb10e216823411d9acb895921ceb9c1"
format_stamp: "Formatted at 2026-07-09 13:12:45 on dist-test-slave-k73n"
I20260709 13:12:45.182897 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:12:45.231590 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:12:45.232788 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:12:45.234280 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:12:45.236419 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:12:45.236636 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:45.236872 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:12:45.237037 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:45.276319 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.66:39307
I20260709 13:12:45.276463  3353 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.66:39307 every 8 connection(s)
I20260709 13:12:45.281314 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:12:45.288448  3357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:45.289263  3358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:12:45.298282  3354 heartbeater.cc:344] Connected to a master server at 127.23.140.126:34333
I20260709 13:12:45.299010  3354 heartbeater.cc:461] Registering TS with master...
I20260709 13:12:45.304589  3354 heartbeater.cc:507] Master 127.23.140.126:34333 requested a full tablet report, sending...
I20260709 13:12:45.305965 24113 server_base.cc:1061] running on GCE node
W20260709 13:12:45.307219  3360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:12:45.307284  3152 ts_manager.cc:194] Registered new tserver with Master: 2bb10e216823411d9acb895921ceb9c1 (127.23.140.66:39307)
I20260709 13:12:45.308413 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:12:45.308691 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:12:45.308910 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602765308891 us; error 0 us; skew 500 ppm
I20260709 13:12:45.309356  3152 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59340
I20260709 13:12:45.309741 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:12:45.312304 24113 webserver.cc:533] Webserver started at http://127.23.140.67:42125/ using document root <none> and password file <none>
I20260709 13:12:45.312736 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:12:45.312889 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:12:45.313094 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:12:45.314311 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/ts-2-root/instance:
uuid: "cdd67666f92e478ab7ada7cb61f95dad"
format_stamp: "Formatted at 2026-07-09 13:12:45 on dist-test-slave-k73n"
I20260709 13:12:45.318522 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 13:12:45.321599  3365 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:45.322329 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 13:12:45.322626 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/ts-2-root
uuid: "cdd67666f92e478ab7ada7cb61f95dad"
format_stamp: "Formatted at 2026-07-09 13:12:45 on dist-test-slave-k73n"
I20260709 13:12:45.322889 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602412999163-24113-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:12:45.340665 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:12:45.341908 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:12:45.343291 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:12:45.345479 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:12:45.345709 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:45.345940 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:12:45.346107 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:45.386487 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.67:43645
I20260709 13:12:45.386652  3427 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.67:43645 every 8 connection(s)
I20260709 13:12:45.399941  3428 heartbeater.cc:344] Connected to a master server at 127.23.140.126:34333
I20260709 13:12:45.400393  3428 heartbeater.cc:461] Registering TS with master...
I20260709 13:12:45.401167  3428 heartbeater.cc:507] Master 127.23.140.126:34333 requested a full tablet report, sending...
I20260709 13:12:45.402925  3152 ts_manager.cc:194] Registered new tserver with Master: cdd67666f92e478ab7ada7cb61f95dad (127.23.140.67:43645)
I20260709 13:12:45.403075 24113 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013395473s
I20260709 13:12:45.404253  3152 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59352
I20260709 13:12:45.427471  3152 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59358:
name: "multi_table_leader_balance_table1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:12:45.429617  3152 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:12:45.485016  3393 tablet_service.cc:1511] Processing CreateTablet for tablet 3018ccc7f3b34dc7a851fa77512e01dc (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=1b69e28562574275b722715c58a69725]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 13:12:45.485221  3392 tablet_service.cc:1511] Processing CreateTablet for tablet 39f0fb4f03d5428a86e3556149ff158d (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=1b69e28562574275b722715c58a69725]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 13:12:45.486625  3393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3018ccc7f3b34dc7a851fa77512e01dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:45.487329  3244 tablet_service.cc:1511] Processing CreateTablet for tablet 3018ccc7f3b34dc7a851fa77512e01dc (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=1b69e28562574275b722715c58a69725]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 13:12:45.488217  3392 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39f0fb4f03d5428a86e3556149ff158d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:45.488749  3244 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3018ccc7f3b34dc7a851fa77512e01dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:45.488682  3319 tablet_service.cc:1511] Processing CreateTablet for tablet 3018ccc7f3b34dc7a851fa77512e01dc (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=1b69e28562574275b722715c58a69725]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 13:12:45.490155  3319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3018ccc7f3b34dc7a851fa77512e01dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:45.494236  3318 tablet_service.cc:1511] Processing CreateTablet for tablet 39f0fb4f03d5428a86e3556149ff158d (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=1b69e28562574275b722715c58a69725]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 13:12:45.495586  3318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39f0fb4f03d5428a86e3556149ff158d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:45.487419  3243 tablet_service.cc:1511] Processing CreateTablet for tablet 39f0fb4f03d5428a86e3556149ff158d (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=1b69e28562574275b722715c58a69725]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 13:12:45.502444  3243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39f0fb4f03d5428a86e3556149ff158d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:45.529263  3444 tablet_bootstrap.cc:492] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03: Bootstrap starting.
I20260709 13:12:45.535903  3444 tablet_bootstrap.cc:654] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:45.537345  3443 tablet_bootstrap.cc:492] T 39f0fb4f03d5428a86e3556149ff158d P cdd67666f92e478ab7ada7cb61f95dad: Bootstrap starting.
I20260709 13:12:45.539295  3445 tablet_bootstrap.cc:492] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1: Bootstrap starting.
I20260709 13:12:45.542922  3444 tablet_bootstrap.cc:492] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03: No bootstrap required, opened a new log
I20260709 13:12:45.543385  3444 ts_tablet_manager.cc:1403] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03: Time spent bootstrapping tablet: real 0.015s	user 0.012s	sys 0.000s
I20260709 13:12:45.545472  3443 tablet_bootstrap.cc:654] T 39f0fb4f03d5428a86e3556149ff158d P cdd67666f92e478ab7ada7cb61f95dad: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:45.545663  3444 raft_consensus.cc:359] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } }
I20260709 13:12:45.546351  3444 raft_consensus.cc:385] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:45.546587  3444 raft_consensus.cc:740] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b65d4b42f94b5c9c3cfc7729165b03, State: Initialized, Role: FOLLOWER
I20260709 13:12:45.547062  3445 tablet_bootstrap.cc:654] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:45.547269  3444 consensus_queue.cc:260] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } }
I20260709 13:12:45.554774  3279 heartbeater.cc:499] Master 127.23.140.126:34333 was elected leader, sending a full tablet report...
I20260709 13:12:45.555341  3445 tablet_bootstrap.cc:492] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1: No bootstrap required, opened a new log
I20260709 13:12:45.555377  3444 ts_tablet_manager.cc:1434] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.003s
I20260709 13:12:45.555974  3445 ts_tablet_manager.cc:1403] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1: Time spent bootstrapping tablet: real 0.017s	user 0.011s	sys 0.005s
I20260709 13:12:45.556727  3443 tablet_bootstrap.cc:492] T 39f0fb4f03d5428a86e3556149ff158d P cdd67666f92e478ab7ada7cb61f95dad: No bootstrap required, opened a new log
I20260709 13:12:45.556834  3444 tablet_bootstrap.cc:492] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03: Bootstrap starting.
I20260709 13:12:45.557298  3443 ts_tablet_manager.cc:1403] T 39f0fb4f03d5428a86e3556149ff158d P cdd67666f92e478ab7ada7cb61f95dad: Time spent bootstrapping tablet: real 0.020s	user 0.012s	sys 0.007s
I20260709 13:12:45.559274  3445 raft_consensus.cc:359] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.560097  3445 raft_consensus.cc:385] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:45.560460  3445 raft_consensus.cc:740] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bb10e216823411d9acb895921ceb9c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:45.560343  3443 raft_consensus.cc:359] T 39f0fb4f03d5428a86e3556149ff158d P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.561213  3443 raft_consensus.cc:385] T 39f0fb4f03d5428a86e3556149ff158d P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:45.561568  3443 raft_consensus.cc:740] T 39f0fb4f03d5428a86e3556149ff158d P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd67666f92e478ab7ada7cb61f95dad, State: Initialized, Role: FOLLOWER
I20260709 13:12:45.561636  3445 consensus_queue.cc:260] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.562583  3443 consensus_queue.cc:260] T 39f0fb4f03d5428a86e3556149ff158d P cdd67666f92e478ab7ada7cb61f95dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.564882  3444 tablet_bootstrap.cc:654] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:45.573943  3445 ts_tablet_manager.cc:1434] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1: Time spent starting tablet: real 0.018s	user 0.009s	sys 0.005s
I20260709 13:12:45.575129  3354 heartbeater.cc:499] Master 127.23.140.126:34333 was elected leader, sending a full tablet report...
I20260709 13:12:45.577253  3444 tablet_bootstrap.cc:492] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03: No bootstrap required, opened a new log
I20260709 13:12:45.577778  3444 ts_tablet_manager.cc:1403] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03: Time spent bootstrapping tablet: real 0.021s	user 0.015s	sys 0.004s
I20260709 13:12:45.579295  3445 tablet_bootstrap.cc:492] T 3018ccc7f3b34dc7a851fa77512e01dc P 2bb10e216823411d9acb895921ceb9c1: Bootstrap starting.
I20260709 13:12:45.581092  3444 raft_consensus.cc:359] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.582165  3444 raft_consensus.cc:385] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:45.582553  3444 raft_consensus.cc:740] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b65d4b42f94b5c9c3cfc7729165b03, State: Initialized, Role: FOLLOWER
I20260709 13:12:45.583307  3428 heartbeater.cc:499] Master 127.23.140.126:34333 was elected leader, sending a full tablet report...
I20260709 13:12:45.583840  3443 ts_tablet_manager.cc:1434] T 39f0fb4f03d5428a86e3556149ff158d P cdd67666f92e478ab7ada7cb61f95dad: Time spent starting tablet: real 0.026s	user 0.019s	sys 0.005s
I20260709 13:12:45.583459  3444 consensus_queue.cc:260] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.584821  3443 tablet_bootstrap.cc:492] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad: Bootstrap starting.
I20260709 13:12:45.586376  3444 ts_tablet_manager.cc:1434] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:12:45.592187  3443 tablet_bootstrap.cc:654] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:45.596580  3445 tablet_bootstrap.cc:654] T 3018ccc7f3b34dc7a851fa77512e01dc P 2bb10e216823411d9acb895921ceb9c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:45.602433  3443 tablet_bootstrap.cc:492] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad: No bootstrap required, opened a new log
I20260709 13:12:45.602949  3443 ts_tablet_manager.cc:1403] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad: Time spent bootstrapping tablet: real 0.018s	user 0.008s	sys 0.008s
I20260709 13:12:45.603062  3445 tablet_bootstrap.cc:492] T 3018ccc7f3b34dc7a851fa77512e01dc P 2bb10e216823411d9acb895921ceb9c1: No bootstrap required, opened a new log
I20260709 13:12:45.603749  3445 ts_tablet_manager.cc:1403] T 3018ccc7f3b34dc7a851fa77512e01dc P 2bb10e216823411d9acb895921ceb9c1: Time spent bootstrapping tablet: real 0.025s	user 0.010s	sys 0.005s
I20260709 13:12:45.605578  3443 raft_consensus.cc:359] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } }
I20260709 13:12:45.606148  3443 raft_consensus.cc:385] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:45.606359  3443 raft_consensus.cc:740] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd67666f92e478ab7ada7cb61f95dad, State: Initialized, Role: FOLLOWER
I20260709 13:12:45.606436  3445 raft_consensus.cc:359] T 3018ccc7f3b34dc7a851fa77512e01dc P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } }
I20260709 13:12:45.607235  3445 raft_consensus.cc:385] T 3018ccc7f3b34dc7a851fa77512e01dc P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:45.607059  3443 consensus_queue.cc:260] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } }
I20260709 13:12:45.607542  3445 raft_consensus.cc:740] T 3018ccc7f3b34dc7a851fa77512e01dc P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bb10e216823411d9acb895921ceb9c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:45.608407  3445 consensus_queue.cc:260] T 3018ccc7f3b34dc7a851fa77512e01dc P 2bb10e216823411d9acb895921ceb9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } }
I20260709 13:12:45.609143  3443 ts_tablet_manager.cc:1434] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20260709 13:12:45.610556  3445 ts_tablet_manager.cc:1434] T 3018ccc7f3b34dc7a851fa77512e01dc P 2bb10e216823411d9acb895921ceb9c1: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:12:45.661203  3451 raft_consensus.cc:493] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:45.661616  3451 raft_consensus.cc:515] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } }
I20260709 13:12:45.663537  3451 leader_election.cc:290] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bb10e216823411d9acb895921ceb9c1 (127.23.140.66:39307), 05b65d4b42f94b5c9c3cfc7729165b03 (127.23.140.65:39107)
I20260709 13:12:45.672230  3329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3018ccc7f3b34dc7a851fa77512e01dc" candidate_uuid: "cdd67666f92e478ab7ada7cb61f95dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb10e216823411d9acb895921ceb9c1" is_pre_election: true
I20260709 13:12:45.672576  3254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3018ccc7f3b34dc7a851fa77512e01dc" candidate_uuid: "cdd67666f92e478ab7ada7cb61f95dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" is_pre_election: true
I20260709 13:12:45.672933  3329 raft_consensus.cc:2468] T 3018ccc7f3b34dc7a851fa77512e01dc P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdd67666f92e478ab7ada7cb61f95dad in term 0.
I20260709 13:12:45.673298  3254 raft_consensus.cc:2468] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdd67666f92e478ab7ada7cb61f95dad in term 0.
I20260709 13:12:45.674402  3366 leader_election.cc:304] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bb10e216823411d9acb895921ceb9c1, cdd67666f92e478ab7ada7cb61f95dad; no voters: 
I20260709 13:12:45.675079  3451 raft_consensus.cc:2804] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:45.675401  3451 raft_consensus.cc:493] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:45.675666  3451 raft_consensus.cc:3060] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:45.680845  3451 raft_consensus.cc:515] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } }
I20260709 13:12:45.682432  3451 leader_election.cc:290] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [CANDIDATE]: Term 1 election: Requested vote from peers 2bb10e216823411d9acb895921ceb9c1 (127.23.140.66:39307), 05b65d4b42f94b5c9c3cfc7729165b03 (127.23.140.65:39107)
I20260709 13:12:45.683306  3329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3018ccc7f3b34dc7a851fa77512e01dc" candidate_uuid: "cdd67666f92e478ab7ada7cb61f95dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb10e216823411d9acb895921ceb9c1"
I20260709 13:12:45.683398  3254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3018ccc7f3b34dc7a851fa77512e01dc" candidate_uuid: "cdd67666f92e478ab7ada7cb61f95dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b65d4b42f94b5c9c3cfc7729165b03"
I20260709 13:12:45.683894  3329 raft_consensus.cc:3060] T 3018ccc7f3b34dc7a851fa77512e01dc P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:45.684017  3254 raft_consensus.cc:3060] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:45.689522  3329 raft_consensus.cc:2468] T 3018ccc7f3b34dc7a851fa77512e01dc P 2bb10e216823411d9acb895921ceb9c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdd67666f92e478ab7ada7cb61f95dad in term 1.
I20260709 13:12:45.689641  3254 raft_consensus.cc:2468] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdd67666f92e478ab7ada7cb61f95dad in term 1.
I20260709 13:12:45.690591  3366 leader_election.cc:304] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bb10e216823411d9acb895921ceb9c1, cdd67666f92e478ab7ada7cb61f95dad; no voters: 
I20260709 13:12:45.691260  3451 raft_consensus.cc:2804] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:45.692065  3451 raft_consensus.cc:697] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 1 LEADER]: Becoming Leader. State: Replica: cdd67666f92e478ab7ada7cb61f95dad, State: Running, Role: LEADER
I20260709 13:12:45.692898  3451 consensus_queue.cc:237] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } }
I20260709 13:12:45.700443  3149 catalog_manager.cc:5697] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad reported cstate change: term changed from 0 to 1, leader changed from <none> to cdd67666f92e478ab7ada7cb61f95dad (127.23.140.67). New cstate: current_term: 1 leader_uuid: "cdd67666f92e478ab7ada7cb61f95dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:45.718102  3448 raft_consensus.cc:493] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:45.718636  3448 raft_consensus.cc:515] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.720714  3448 leader_election.cc:290] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bb10e216823411d9acb895921ceb9c1 (127.23.140.66:39307), cdd67666f92e478ab7ada7cb61f95dad (127.23.140.67:43645)
I20260709 13:12:45.730753  3329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39f0fb4f03d5428a86e3556149ff158d" candidate_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb10e216823411d9acb895921ceb9c1" is_pre_election: true
I20260709 13:12:45.730835  3403 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39f0fb4f03d5428a86e3556149ff158d" candidate_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd67666f92e478ab7ada7cb61f95dad" is_pre_election: true
I20260709 13:12:45.731446  3329 raft_consensus.cc:2468] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b65d4b42f94b5c9c3cfc7729165b03 in term 0.
I20260709 13:12:45.731519  3403 raft_consensus.cc:2468] T 39f0fb4f03d5428a86e3556149ff158d P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b65d4b42f94b5c9c3cfc7729165b03 in term 0.
I20260709 13:12:45.732404  3217 leader_election.cc:304] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b65d4b42f94b5c9c3cfc7729165b03, 2bb10e216823411d9acb895921ceb9c1; no voters: 
I20260709 13:12:45.733093  3448 raft_consensus.cc:2804] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:45.733457  3448 raft_consensus.cc:493] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:45.733691  3448 raft_consensus.cc:3060] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:45.738664  3448 raft_consensus.cc:515] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.740347  3448 leader_election.cc:290] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [CANDIDATE]: Term 1 election: Requested vote from peers 2bb10e216823411d9acb895921ceb9c1 (127.23.140.66:39307), cdd67666f92e478ab7ada7cb61f95dad (127.23.140.67:43645)
I20260709 13:12:45.741020  3329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39f0fb4f03d5428a86e3556149ff158d" candidate_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb10e216823411d9acb895921ceb9c1"
I20260709 13:12:45.741675  3329 raft_consensus.cc:3060] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:45.741568  3403 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39f0fb4f03d5428a86e3556149ff158d" candidate_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd67666f92e478ab7ada7cb61f95dad"
I20260709 13:12:45.742319  3403 raft_consensus.cc:3060] T 39f0fb4f03d5428a86e3556149ff158d P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:45.749105  3329 raft_consensus.cc:2468] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b65d4b42f94b5c9c3cfc7729165b03 in term 1.
I20260709 13:12:45.749720  3403 raft_consensus.cc:2468] T 39f0fb4f03d5428a86e3556149ff158d P cdd67666f92e478ab7ada7cb61f95dad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b65d4b42f94b5c9c3cfc7729165b03 in term 1.
I20260709 13:12:45.750401  3217 leader_election.cc:304] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b65d4b42f94b5c9c3cfc7729165b03, 2bb10e216823411d9acb895921ceb9c1; no voters: 
I20260709 13:12:45.751302  3448 raft_consensus.cc:2804] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:45.752331  3448 raft_consensus.cc:697] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 LEADER]: Becoming Leader. State: Replica: 05b65d4b42f94b5c9c3cfc7729165b03, State: Running, Role: LEADER
I20260709 13:12:45.753100  3448 consensus_queue.cc:237] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.759939  3149 catalog_manager.cc:5697] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b65d4b42f94b5c9c3cfc7729165b03 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:45.820715  3149 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59792:
name: "multi_table_leader_balance_table2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:12:45.822716  3149 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:12:45.844739  3244 tablet_service.cc:1511] Processing CreateTablet for tablet fecd607247614897a832471c280a73b1 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=7a6e3ae0156d49f3af6868ad0b1a7548]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 13:12:45.846078  3244 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fecd607247614897a832471c280a73b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:45.846335  3319 tablet_service.cc:1511] Processing CreateTablet for tablet fecd607247614897a832471c280a73b1 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=7a6e3ae0156d49f3af6868ad0b1a7548]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 13:12:45.847555  3319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fecd607247614897a832471c280a73b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:45.847594  3392 tablet_service.cc:1511] Processing CreateTablet for tablet fecd607247614897a832471c280a73b1 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=7a6e3ae0156d49f3af6868ad0b1a7548]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 13:12:45.848906  3392 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fecd607247614897a832471c280a73b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:45.851688  3318 tablet_service.cc:1511] Processing CreateTablet for tablet f715f76e5bd74699ad948688bb7d7378 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=7a6e3ae0156d49f3af6868ad0b1a7548]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 13:12:45.852897  3318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f715f76e5bd74699ad948688bb7d7378. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:45.853014  3393 tablet_service.cc:1511] Processing CreateTablet for tablet f715f76e5bd74699ad948688bb7d7378 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=7a6e3ae0156d49f3af6868ad0b1a7548]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 13:12:45.854265  3393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f715f76e5bd74699ad948688bb7d7378. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:45.858680  3243 tablet_service.cc:1511] Processing CreateTablet for tablet f715f76e5bd74699ad948688bb7d7378 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=7a6e3ae0156d49f3af6868ad0b1a7548]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 13:12:45.859999  3243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f715f76e5bd74699ad948688bb7d7378. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:45.862424  3444 tablet_bootstrap.cc:492] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03: Bootstrap starting.
I20260709 13:12:45.868824  3444 tablet_bootstrap.cc:654] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:45.871724  3443 tablet_bootstrap.cc:492] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad: Bootstrap starting.
I20260709 13:12:45.878858  3443 tablet_bootstrap.cc:654] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:45.882501  3445 tablet_bootstrap.cc:492] T fecd607247614897a832471c280a73b1 P 2bb10e216823411d9acb895921ceb9c1: Bootstrap starting.
I20260709 13:12:45.884586  3444 tablet_bootstrap.cc:492] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03: No bootstrap required, opened a new log
I20260709 13:12:45.885277  3444 ts_tablet_manager.cc:1403] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03: Time spent bootstrapping tablet: real 0.023s	user 0.012s	sys 0.004s
I20260709 13:12:45.888382  3444 raft_consensus.cc:359] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.889151  3444 raft_consensus.cc:385] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:45.889477  3444 raft_consensus.cc:740] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b65d4b42f94b5c9c3cfc7729165b03, State: Initialized, Role: FOLLOWER
I20260709 13:12:45.890345  3445 tablet_bootstrap.cc:654] T fecd607247614897a832471c280a73b1 P 2bb10e216823411d9acb895921ceb9c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:45.890208  3444 consensus_queue.cc:260] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.894749  3444 ts_tablet_manager.cc:1434] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 13:12:45.895862  3444 tablet_bootstrap.cc:492] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03: Bootstrap starting.
I20260709 13:12:45.896706  3443 tablet_bootstrap.cc:492] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad: No bootstrap required, opened a new log
I20260709 13:12:45.896971  3445 tablet_bootstrap.cc:492] T fecd607247614897a832471c280a73b1 P 2bb10e216823411d9acb895921ceb9c1: No bootstrap required, opened a new log
I20260709 13:12:45.897367  3443 ts_tablet_manager.cc:1403] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad: Time spent bootstrapping tablet: real 0.026s	user 0.012s	sys 0.006s
I20260709 13:12:45.897532  3445 ts_tablet_manager.cc:1403] T fecd607247614897a832471c280a73b1 P 2bb10e216823411d9acb895921ceb9c1: Time spent bootstrapping tablet: real 0.015s	user 0.013s	sys 0.000s
I20260709 13:12:45.900751  3443 raft_consensus.cc:359] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.901589  3443 raft_consensus.cc:385] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:45.901932  3443 raft_consensus.cc:740] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd67666f92e478ab7ada7cb61f95dad, State: Initialized, Role: FOLLOWER
I20260709 13:12:45.902242  3445 raft_consensus.cc:359] T fecd607247614897a832471c280a73b1 P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.903044  3445 raft_consensus.cc:385] T fecd607247614897a832471c280a73b1 P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:45.902989  3443 consensus_queue.cc:260] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.903481  3445 raft_consensus.cc:740] T fecd607247614897a832471c280a73b1 P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bb10e216823411d9acb895921ceb9c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:45.905442  3444 tablet_bootstrap.cc:654] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:45.905248  3445 consensus_queue.cc:260] T fecd607247614897a832471c280a73b1 P 2bb10e216823411d9acb895921ceb9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.907061  3443 ts_tablet_manager.cc:1434] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.004s
I20260709 13:12:45.907558  3445 ts_tablet_manager.cc:1434] T fecd607247614897a832471c280a73b1 P 2bb10e216823411d9acb895921ceb9c1: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 13:12:45.908197  3443 tablet_bootstrap.cc:492] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad: Bootstrap starting.
I20260709 13:12:45.909430  3445 tablet_bootstrap.cc:492] T f715f76e5bd74699ad948688bb7d7378 P 2bb10e216823411d9acb895921ceb9c1: Bootstrap starting.
I20260709 13:12:45.915481  3443 tablet_bootstrap.cc:654] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:45.916646  3445 tablet_bootstrap.cc:654] T f715f76e5bd74699ad948688bb7d7378 P 2bb10e216823411d9acb895921ceb9c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:45.928437  3443 tablet_bootstrap.cc:492] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad: No bootstrap required, opened a new log
I20260709 13:12:45.929239  3443 ts_tablet_manager.cc:1403] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad: Time spent bootstrapping tablet: real 0.021s	user 0.015s	sys 0.004s
I20260709 13:12:45.933372  3443 raft_consensus.cc:359] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.935125  3443 raft_consensus.cc:385] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:45.935506  3443 raft_consensus.cc:740] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd67666f92e478ab7ada7cb61f95dad, State: Initialized, Role: FOLLOWER
I20260709 13:12:45.936357  3443 consensus_queue.cc:260] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.937625  3445 tablet_bootstrap.cc:492] T f715f76e5bd74699ad948688bb7d7378 P 2bb10e216823411d9acb895921ceb9c1: No bootstrap required, opened a new log
I20260709 13:12:45.938264  3445 ts_tablet_manager.cc:1403] T f715f76e5bd74699ad948688bb7d7378 P 2bb10e216823411d9acb895921ceb9c1: Time spent bootstrapping tablet: real 0.029s	user 0.014s	sys 0.014s
I20260709 13:12:45.938689  3443 ts_tablet_manager.cc:1434] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.003s
I20260709 13:12:45.941340  3445 raft_consensus.cc:359] T f715f76e5bd74699ad948688bb7d7378 P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.942119  3445 raft_consensus.cc:385] T f715f76e5bd74699ad948688bb7d7378 P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:45.942417  3445 raft_consensus.cc:740] T f715f76e5bd74699ad948688bb7d7378 P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bb10e216823411d9acb895921ceb9c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:45.942736  3444 tablet_bootstrap.cc:492] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03: No bootstrap required, opened a new log
I20260709 13:12:45.943358  3444 ts_tablet_manager.cc:1403] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03: Time spent bootstrapping tablet: real 0.048s	user 0.014s	sys 0.030s
I20260709 13:12:45.943313  3445 consensus_queue.cc:260] T f715f76e5bd74699ad948688bb7d7378 P 2bb10e216823411d9acb895921ceb9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.945704  3445 ts_tablet_manager.cc:1434] T f715f76e5bd74699ad948688bb7d7378 P 2bb10e216823411d9acb895921ceb9c1: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.002s
I20260709 13:12:45.946671  3444 raft_consensus.cc:359] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.947378  3444 raft_consensus.cc:385] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:45.947665  3444 raft_consensus.cc:740] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b65d4b42f94b5c9c3cfc7729165b03, State: Initialized, Role: FOLLOWER
I20260709 13:12:45.948429  3444 consensus_queue.cc:260] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.950614  3444 ts_tablet_manager.cc:1434] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:12:45.963748  3451 raft_consensus.cc:493] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:45.964253  3451 raft_consensus.cc:515] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.965916  3451 leader_election.cc:290] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b65d4b42f94b5c9c3cfc7729165b03 (127.23.140.65:39107), 2bb10e216823411d9acb895921ceb9c1 (127.23.140.66:39307)
I20260709 13:12:45.966679  3254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fecd607247614897a832471c280a73b1" candidate_uuid: "cdd67666f92e478ab7ada7cb61f95dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" is_pre_election: true
I20260709 13:12:45.966892  3329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fecd607247614897a832471c280a73b1" candidate_uuid: "cdd67666f92e478ab7ada7cb61f95dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb10e216823411d9acb895921ceb9c1" is_pre_election: true
I20260709 13:12:45.967368  3254 raft_consensus.cc:2468] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdd67666f92e478ab7ada7cb61f95dad in term 0.
I20260709 13:12:45.967499  3329 raft_consensus.cc:2468] T fecd607247614897a832471c280a73b1 P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdd67666f92e478ab7ada7cb61f95dad in term 0.
I20260709 13:12:45.968474  3368 leader_election.cc:304] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b65d4b42f94b5c9c3cfc7729165b03, cdd67666f92e478ab7ada7cb61f95dad; no voters: 
I20260709 13:12:45.969175  3451 raft_consensus.cc:2804] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:45.969472  3451 raft_consensus.cc:493] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:45.969741  3451 raft_consensus.cc:3060] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:45.974637  3451 raft_consensus.cc:515] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.976405  3451 leader_election.cc:290] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [CANDIDATE]: Term 1 election: Requested vote from peers 05b65d4b42f94b5c9c3cfc7729165b03 (127.23.140.65:39107), 2bb10e216823411d9acb895921ceb9c1 (127.23.140.66:39307)
I20260709 13:12:45.977017  3254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fecd607247614897a832471c280a73b1" candidate_uuid: "cdd67666f92e478ab7ada7cb61f95dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b65d4b42f94b5c9c3cfc7729165b03"
I20260709 13:12:45.977484  3329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fecd607247614897a832471c280a73b1" candidate_uuid: "cdd67666f92e478ab7ada7cb61f95dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb10e216823411d9acb895921ceb9c1"
I20260709 13:12:45.977689  3254 raft_consensus.cc:3060] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:45.978075  3329 raft_consensus.cc:3060] T fecd607247614897a832471c280a73b1 P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:45.984752  3329 raft_consensus.cc:2468] T fecd607247614897a832471c280a73b1 P 2bb10e216823411d9acb895921ceb9c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdd67666f92e478ab7ada7cb61f95dad in term 1.
I20260709 13:12:45.984752  3254 raft_consensus.cc:2468] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdd67666f92e478ab7ada7cb61f95dad in term 1.
I20260709 13:12:45.986568  3368 leader_election.cc:304] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b65d4b42f94b5c9c3cfc7729165b03, cdd67666f92e478ab7ada7cb61f95dad; no voters: 
W20260709 13:12:45.986596  3200 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:12:45.987397  3451 raft_consensus.cc:2804] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:45.987823  3451 raft_consensus.cc:697] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 1 LEADER]: Becoming Leader. State: Replica: cdd67666f92e478ab7ada7cb61f95dad, State: Running, Role: LEADER
I20260709 13:12:45.988514  3451 consensus_queue.cc:237] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:45.994926  3152 catalog_manager.cc:5697] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad reported cstate change: term changed from 0 to 1, leader changed from <none> to cdd67666f92e478ab7ada7cb61f95dad (127.23.140.67). New cstate: current_term: 1 leader_uuid: "cdd67666f92e478ab7ada7cb61f95dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:46.108079  3448 raft_consensus.cc:493] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:46.108619  3448 raft_consensus.cc:515] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:46.110340  3448 leader_election.cc:290] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bb10e216823411d9acb895921ceb9c1 (127.23.140.66:39307), cdd67666f92e478ab7ada7cb61f95dad (127.23.140.67:43645)
I20260709 13:12:46.111129  3329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f715f76e5bd74699ad948688bb7d7378" candidate_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb10e216823411d9acb895921ceb9c1" is_pre_election: true
I20260709 13:12:46.111375  3403 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f715f76e5bd74699ad948688bb7d7378" candidate_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd67666f92e478ab7ada7cb61f95dad" is_pre_election: true
I20260709 13:12:46.111738  3329 raft_consensus.cc:2468] T f715f76e5bd74699ad948688bb7d7378 P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b65d4b42f94b5c9c3cfc7729165b03 in term 0.
I20260709 13:12:46.112039  3403 raft_consensus.cc:2468] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b65d4b42f94b5c9c3cfc7729165b03 in term 0.
I20260709 13:12:46.112789  3217 leader_election.cc:304] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b65d4b42f94b5c9c3cfc7729165b03, 2bb10e216823411d9acb895921ceb9c1; no voters: 
I20260709 13:12:46.113600  3448 raft_consensus.cc:2804] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:46.113906  3448 raft_consensus.cc:493] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:46.114164  3448 raft_consensus.cc:3060] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:46.118991  3448 raft_consensus.cc:515] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:46.120539  3448 leader_election.cc:290] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [CANDIDATE]: Term 1 election: Requested vote from peers 2bb10e216823411d9acb895921ceb9c1 (127.23.140.66:39307), cdd67666f92e478ab7ada7cb61f95dad (127.23.140.67:43645)
I20260709 13:12:46.121249  3329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f715f76e5bd74699ad948688bb7d7378" candidate_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb10e216823411d9acb895921ceb9c1"
I20260709 13:12:46.121501  3403 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f715f76e5bd74699ad948688bb7d7378" candidate_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd67666f92e478ab7ada7cb61f95dad"
I20260709 13:12:46.121784  3329 raft_consensus.cc:3060] T f715f76e5bd74699ad948688bb7d7378 P 2bb10e216823411d9acb895921ceb9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:46.122115  3403 raft_consensus.cc:3060] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:46.126817  3329 raft_consensus.cc:2468] T f715f76e5bd74699ad948688bb7d7378 P 2bb10e216823411d9acb895921ceb9c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b65d4b42f94b5c9c3cfc7729165b03 in term 1.
I20260709 13:12:46.126995  3403 raft_consensus.cc:2468] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b65d4b42f94b5c9c3cfc7729165b03 in term 1.
I20260709 13:12:46.127950  3217 leader_election.cc:304] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b65d4b42f94b5c9c3cfc7729165b03, 2bb10e216823411d9acb895921ceb9c1; no voters: 
I20260709 13:12:46.128628  3448 raft_consensus.cc:2804] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:46.129072  3448 raft_consensus.cc:697] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 LEADER]: Becoming Leader. State: Replica: 05b65d4b42f94b5c9c3cfc7729165b03, State: Running, Role: LEADER
I20260709 13:12:46.129796  3448 consensus_queue.cc:237] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:46.136843  3149 catalog_manager.cc:5697] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b65d4b42f94b5c9c3cfc7729165b03 (127.23.140.65). New cstate: current_term: 1 leader_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:46.144176  3451 consensus_queue.cc:1048] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:46.155577  3477 consensus_queue.cc:1048] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:46.166752  3448 consensus_queue.cc:1048] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:46.182940  3448 consensus_queue.cc:1048] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:46.197576  3403 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3018ccc7f3b34dc7a851fa77512e01dc"
dest_uuid: "cdd67666f92e478ab7ada7cb61f95dad"
mode: GRACEFUL
new_leader_uuid: "05b65d4b42f94b5c9c3cfc7729165b03"
 from {username='slave'} at 127.0.0.1:47276
I20260709 13:12:46.198285  3403 raft_consensus.cc:606] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 1 LEADER]: Received request to transfer leadership to TS 05b65d4b42f94b5c9c3cfc7729165b03
I20260709 13:12:46.211958  3254 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "39f0fb4f03d5428a86e3556149ff158d"
dest_uuid: "05b65d4b42f94b5c9c3cfc7729165b03"
mode: GRACEFUL
new_leader_uuid: "2bb10e216823411d9acb895921ceb9c1"
 from {username='slave'} at 127.0.0.1:46074
I20260709 13:12:46.212435  3254 raft_consensus.cc:606] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 LEADER]: Received request to transfer leadership to TS 2bb10e216823411d9acb895921ceb9c1
I20260709 13:12:46.217636  3403 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fecd607247614897a832471c280a73b1"
dest_uuid: "cdd67666f92e478ab7ada7cb61f95dad"
mode: GRACEFUL
new_leader_uuid: "05b65d4b42f94b5c9c3cfc7729165b03"
 from {username='slave'} at 127.0.0.1:47276
I20260709 13:12:46.218079  3403 raft_consensus.cc:606] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 1 LEADER]: Received request to transfer leadership to TS 05b65d4b42f94b5c9c3cfc7729165b03
I20260709 13:12:46.598156  3451 consensus_queue.cc:1048] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:46.602187  3477 consensus_queue.cc:1048] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:46.605764  3456 raft_consensus.cc:993] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad: : Instructing follower 05b65d4b42f94b5c9c3cfc7729165b03 to start an election
I20260709 13:12:46.606230  3456 raft_consensus.cc:1081] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 1 LEADER]: Signalling peer 05b65d4b42f94b5c9c3cfc7729165b03 to start an election
I20260709 13:12:46.607963  3253 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fecd607247614897a832471c280a73b1"
dest_uuid: "05b65d4b42f94b5c9c3cfc7729165b03"
 from {username='slave'} at 127.0.0.1:46060
I20260709 13:12:46.608537  3253 raft_consensus.cc:493] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:46.608889  3253 raft_consensus.cc:3060] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:46.617458  3253 raft_consensus.cc:515] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:46.618484  3254 raft_consensus.cc:1240] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 2 FOLLOWER]: Rejecting Update request from peer cdd67666f92e478ab7ada7cb61f95dad for earlier term 1. Current term is 2. Ops: []
I20260709 13:12:46.620031  3253 leader_election.cc:290] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [CANDIDATE]: Term 2 election: Requested vote from peers 2bb10e216823411d9acb895921ceb9c1 (127.23.140.66:39307), cdd67666f92e478ab7ada7cb61f95dad (127.23.140.67:43645)
I20260709 13:12:46.621701  3328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fecd607247614897a832471c280a73b1" candidate_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2bb10e216823411d9acb895921ceb9c1"
I20260709 13:12:46.621835  3403 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fecd607247614897a832471c280a73b1" candidate_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdd67666f92e478ab7ada7cb61f95dad"
I20260709 13:12:46.622409  3328 raft_consensus.cc:3060] T fecd607247614897a832471c280a73b1 P 2bb10e216823411d9acb895921ceb9c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:46.622493  3403 raft_consensus.cc:3055] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:46.622957  3403 raft_consensus.cc:740] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cdd67666f92e478ab7ada7cb61f95dad, State: Running, Role: LEADER
I20260709 13:12:46.622685  3477 consensus_queue.cc:1059] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:12:46.628065  3403 consensus_queue.cc:260] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:46.629060  3403 raft_consensus.cc:3060] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:46.630196  3328 raft_consensus.cc:2468] T fecd607247614897a832471c280a73b1 P 2bb10e216823411d9acb895921ceb9c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b65d4b42f94b5c9c3cfc7729165b03 in term 2.
I20260709 13:12:46.631395  3217 leader_election.cc:304] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b65d4b42f94b5c9c3cfc7729165b03, 2bb10e216823411d9acb895921ceb9c1; no voters: 
I20260709 13:12:46.632302  3482 raft_consensus.cc:2804] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:46.632737  3482 raft_consensus.cc:697] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 2 LEADER]: Becoming Leader. State: Replica: 05b65d4b42f94b5c9c3cfc7729165b03, State: Running, Role: LEADER
I20260709 13:12:46.633657  3482 consensus_queue.cc:237] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:46.635114  3403 raft_consensus.cc:2468] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b65d4b42f94b5c9c3cfc7729165b03 in term 2.
W20260709 13:12:46.635600  3477 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 13:12:46.641705  3149 catalog_manager.cc:5697] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 reported cstate change: term changed from 1 to 2, leader changed from cdd67666f92e478ab7ada7cb61f95dad (127.23.140.67) to 05b65d4b42f94b5c9c3cfc7729165b03 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:46.672683  3482 consensus_queue.cc:1048] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:46.684649  3448 consensus_queue.cc:1048] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:46.716393  3451 raft_consensus.cc:993] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad: : Instructing follower 05b65d4b42f94b5c9c3cfc7729165b03 to start an election
I20260709 13:12:46.716856  3477 raft_consensus.cc:1081] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 1 LEADER]: Signalling peer 05b65d4b42f94b5c9c3cfc7729165b03 to start an election
I20260709 13:12:46.718184  3254 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3018ccc7f3b34dc7a851fa77512e01dc"
dest_uuid: "05b65d4b42f94b5c9c3cfc7729165b03"
 from {username='slave'} at 127.0.0.1:46060
I20260709 13:12:46.718597  3254 raft_consensus.cc:493] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:46.718891  3254 raft_consensus.cc:3060] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:46.724226  3254 raft_consensus.cc:515] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } }
I20260709 13:12:46.725881  3254 leader_election.cc:290] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [CANDIDATE]: Term 2 election: Requested vote from peers cdd67666f92e478ab7ada7cb61f95dad (127.23.140.67:43645), 2bb10e216823411d9acb895921ceb9c1 (127.23.140.66:39307)
I20260709 13:12:46.726531  3403 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3018ccc7f3b34dc7a851fa77512e01dc" candidate_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdd67666f92e478ab7ada7cb61f95dad"
I20260709 13:12:46.726873  3328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3018ccc7f3b34dc7a851fa77512e01dc" candidate_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2bb10e216823411d9acb895921ceb9c1"
I20260709 13:12:46.727224  3403 raft_consensus.cc:3055] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:46.727453  3328 raft_consensus.cc:3060] T 3018ccc7f3b34dc7a851fa77512e01dc P 2bb10e216823411d9acb895921ceb9c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:46.727597  3403 raft_consensus.cc:740] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cdd67666f92e478ab7ada7cb61f95dad, State: Running, Role: LEADER
I20260709 13:12:46.728444  3403 consensus_queue.cc:260] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } }
I20260709 13:12:46.729700  3403 raft_consensus.cc:3060] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:46.734443  3328 raft_consensus.cc:2468] T 3018ccc7f3b34dc7a851fa77512e01dc P 2bb10e216823411d9acb895921ceb9c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b65d4b42f94b5c9c3cfc7729165b03 in term 2.
I20260709 13:12:46.735710  3217 leader_election.cc:304] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b65d4b42f94b5c9c3cfc7729165b03, 2bb10e216823411d9acb895921ceb9c1; no voters: 
I20260709 13:12:46.736398  3448 raft_consensus.cc:2804] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:46.736379  3403 raft_consensus.cc:2468] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b65d4b42f94b5c9c3cfc7729165b03 in term 2.
I20260709 13:12:46.737181  3448 raft_consensus.cc:697] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [term 2 LEADER]: Becoming Leader. State: Replica: 05b65d4b42f94b5c9c3cfc7729165b03, State: Running, Role: LEADER
I20260709 13:12:46.737994  3448 consensus_queue.cc:237] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } }
I20260709 13:12:46.743701  3149 catalog_manager.cc:5697] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 reported cstate change: term changed from 1 to 2, leader changed from cdd67666f92e478ab7ada7cb61f95dad (127.23.140.67) to 05b65d4b42f94b5c9c3cfc7729165b03 (127.23.140.65). New cstate: current_term: 2 leader_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:46.764684  3482 raft_consensus.cc:993] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03: : Instructing follower 2bb10e216823411d9acb895921ceb9c1 to start an election
I20260709 13:12:46.765205  3448 raft_consensus.cc:1081] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 LEADER]: Signalling peer 2bb10e216823411d9acb895921ceb9c1 to start an election
I20260709 13:12:46.766597  3328 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "39f0fb4f03d5428a86e3556149ff158d"
dest_uuid: "2bb10e216823411d9acb895921ceb9c1"
 from {username='slave'} at 127.0.0.1:57800
I20260709 13:12:46.767050  3328 raft_consensus.cc:493] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:46.767408  3328 raft_consensus.cc:3060] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:46.772730  3328 raft_consensus.cc:515] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:46.774703  3328 leader_election.cc:290] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [CANDIDATE]: Term 2 election: Requested vote from peers 05b65d4b42f94b5c9c3cfc7729165b03 (127.23.140.65:39107), cdd67666f92e478ab7ada7cb61f95dad (127.23.140.67:43645)
I20260709 13:12:46.786878  3403 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39f0fb4f03d5428a86e3556149ff158d" candidate_uuid: "2bb10e216823411d9acb895921ceb9c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdd67666f92e478ab7ada7cb61f95dad"
I20260709 13:12:46.787035  3254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39f0fb4f03d5428a86e3556149ff158d" candidate_uuid: "2bb10e216823411d9acb895921ceb9c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05b65d4b42f94b5c9c3cfc7729165b03"
I20260709 13:12:46.787520  3403 raft_consensus.cc:3060] T 39f0fb4f03d5428a86e3556149ff158d P cdd67666f92e478ab7ada7cb61f95dad [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:46.787631  3254 raft_consensus.cc:3055] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:46.788022  3254 raft_consensus.cc:740] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 05b65d4b42f94b5c9c3cfc7729165b03, State: Running, Role: LEADER
I20260709 13:12:46.788655  3254 consensus_queue.cc:260] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:46.789589  3254 raft_consensus.cc:3060] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:46.792491  3403 raft_consensus.cc:2468] T 39f0fb4f03d5428a86e3556149ff158d P cdd67666f92e478ab7ada7cb61f95dad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bb10e216823411d9acb895921ceb9c1 in term 2.
I20260709 13:12:46.793690  3295 leader_election.cc:304] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bb10e216823411d9acb895921ceb9c1, cdd67666f92e478ab7ada7cb61f95dad; no voters: 
I20260709 13:12:46.794566  3450 raft_consensus.cc:2804] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:46.794581  3254 raft_consensus.cc:2468] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bb10e216823411d9acb895921ceb9c1 in term 2.
I20260709 13:12:46.795792  3450 raft_consensus.cc:697] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [term 2 LEADER]: Becoming Leader. State: Replica: 2bb10e216823411d9acb895921ceb9c1, State: Running, Role: LEADER
I20260709 13:12:46.796623  3450 consensus_queue.cc:237] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:46.803261  3149 catalog_manager.cc:5697] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 reported cstate change: term changed from 1 to 2, leader changed from 05b65d4b42f94b5c9c3cfc7729165b03 (127.23.140.65) to 2bb10e216823411d9acb895921ceb9c1 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2bb10e216823411d9acb895921ceb9c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:46.924778 24113 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table1
I20260709 13:12:46.926235 24113 auto_leader_rebalancer.cc:452] table: multi_table_leader_balance_table1, leader rebalance finish, leader transfer count: 0
I20260709 13:12:46.926554 24113 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table2
I20260709 13:12:46.937808  3254 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f715f76e5bd74699ad948688bb7d7378"
dest_uuid: "05b65d4b42f94b5c9c3cfc7729165b03"
mode: GRACEFUL
new_leader_uuid: "cdd67666f92e478ab7ada7cb61f95dad"
 from {username='slave'} at 127.0.0.1:46088
I20260709 13:12:46.938347  3254 raft_consensus.cc:606] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 LEADER]: Received request to transfer leadership to TS cdd67666f92e478ab7ada7cb61f95dad
I20260709 13:12:46.939368 24113 auto_leader_rebalancer.cc:452] table: multi_table_leader_balance_table2, leader rebalance finish, leader transfer count: 1
I20260709 13:12:47.154739  3328 raft_consensus.cc:1275] T fecd607247614897a832471c280a73b1 P 2bb10e216823411d9acb895921ceb9c1 [term 2 FOLLOWER]: Refusing update from remote peer 05b65d4b42f94b5c9c3cfc7729165b03: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:47.155992  3448 consensus_queue.cc:1048] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:47.163479  3403 raft_consensus.cc:1275] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 2 FOLLOWER]: Refusing update from remote peer 05b65d4b42f94b5c9c3cfc7729165b03: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:47.165033  3482 consensus_queue.cc:1048] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:47.172753  3403 raft_consensus.cc:1275] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 2 FOLLOWER]: Refusing update from remote peer 05b65d4b42f94b5c9c3cfc7729165b03: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:47.174007  3482 consensus_queue.cc:1048] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:47.181445  3448 raft_consensus.cc:993] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03: : Instructing follower cdd67666f92e478ab7ada7cb61f95dad to start an election
I20260709 13:12:47.184875  3328 raft_consensus.cc:1275] T 3018ccc7f3b34dc7a851fa77512e01dc P 2bb10e216823411d9acb895921ceb9c1 [term 2 FOLLOWER]: Refusing update from remote peer 05b65d4b42f94b5c9c3cfc7729165b03: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:47.184913  3448 raft_consensus.cc:1081] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 LEADER]: Signalling peer cdd67666f92e478ab7ada7cb61f95dad to start an election
I20260709 13:12:47.187510  3502 consensus_queue.cc:1048] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:47.187820  3402 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f715f76e5bd74699ad948688bb7d7378"
dest_uuid: "cdd67666f92e478ab7ada7cb61f95dad"
 from {username='slave'} at 127.0.0.1:47270
I20260709 13:12:47.188367  3402 raft_consensus.cc:493] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:47.188742  3402 raft_consensus.cc:3060] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:47.197278  3402 raft_consensus.cc:515] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:47.199272  3402 leader_election.cc:290] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad [CANDIDATE]: Term 2 election: Requested vote from peers 05b65d4b42f94b5c9c3cfc7729165b03 (127.23.140.65:39107), 2bb10e216823411d9acb895921ceb9c1 (127.23.140.66:39307)
I20260709 13:12:47.200066  3254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f715f76e5bd74699ad948688bb7d7378" candidate_uuid: "cdd67666f92e478ab7ada7cb61f95dad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05b65d4b42f94b5c9c3cfc7729165b03"
I20260709 13:12:47.200328  3328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f715f76e5bd74699ad948688bb7d7378" candidate_uuid: "cdd67666f92e478ab7ada7cb61f95dad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2bb10e216823411d9acb895921ceb9c1"
I20260709 13:12:47.200742  3254 raft_consensus.cc:3055] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:47.200886  3328 raft_consensus.cc:3060] T f715f76e5bd74699ad948688bb7d7378 P 2bb10e216823411d9acb895921ceb9c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:47.201171  3254 raft_consensus.cc:740] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 05b65d4b42f94b5c9c3cfc7729165b03, State: Running, Role: LEADER
I20260709 13:12:47.201975  3254 consensus_queue.cc:260] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:47.202947  3254 raft_consensus.cc:3060] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:47.206882  3328 raft_consensus.cc:2468] T f715f76e5bd74699ad948688bb7d7378 P 2bb10e216823411d9acb895921ceb9c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdd67666f92e478ab7ada7cb61f95dad in term 2.
I20260709 13:12:47.207948  3366 leader_election.cc:304] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bb10e216823411d9acb895921ceb9c1, cdd67666f92e478ab7ada7cb61f95dad; no voters: 
I20260709 13:12:47.208617  3477 raft_consensus.cc:2804] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:47.209007  3477 raft_consensus.cc:697] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad [term 2 LEADER]: Becoming Leader. State: Replica: cdd67666f92e478ab7ada7cb61f95dad, State: Running, Role: LEADER
I20260709 13:12:47.209245  3254 raft_consensus.cc:2468] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdd67666f92e478ab7ada7cb61f95dad in term 2.
I20260709 13:12:47.209779  3477 consensus_queue.cc:237] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } }
I20260709 13:12:47.215577  3150 catalog_manager.cc:5697] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad reported cstate change: term changed from 1 to 2, leader changed from 05b65d4b42f94b5c9c3cfc7729165b03 (127.23.140.65) to cdd67666f92e478ab7ada7cb61f95dad (127.23.140.67). New cstate: current_term: 2 leader_uuid: "cdd67666f92e478ab7ada7cb61f95dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bb10e216823411d9acb895921ceb9c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:47.220257  3254 raft_consensus.cc:1275] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 2 FOLLOWER]: Refusing update from remote peer 2bb10e216823411d9acb895921ceb9c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:47.221596  3450 consensus_queue.cc:1048] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b65d4b42f94b5c9c3cfc7729165b03" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 39107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:47.230427  3402 raft_consensus.cc:1275] T 39f0fb4f03d5428a86e3556149ff158d P cdd67666f92e478ab7ada7cb61f95dad [term 2 FOLLOWER]: Refusing update from remote peer 2bb10e216823411d9acb895921ceb9c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:47.232082  3498 consensus_queue.cc:1048] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd67666f92e478ab7ada7cb61f95dad" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:47.275398 24113 tablet_server.cc:179] TabletServer@127.23.140.65:0 shutting down...
I20260709 13:12:47.295341 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:12:47.295945 24113 tablet_replica.cc:333] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03: stopping tablet replica
I20260709 13:12:47.296492 24113 raft_consensus.cc:2243] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:47.296895 24113 raft_consensus.cc:2272] T f715f76e5bd74699ad948688bb7d7378 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:47.299249 24113 tablet_replica.cc:333] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03: stopping tablet replica
I20260709 13:12:47.299820 24113 raft_consensus.cc:2243] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:47.300609 24113 raft_consensus.cc:2272] T fecd607247614897a832471c280a73b1 P 05b65d4b42f94b5c9c3cfc7729165b03 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:47.302950 24113 tablet_replica.cc:333] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03: stopping tablet replica
I20260709 13:12:47.303475 24113 raft_consensus.cc:2243] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:47.303886 24113 raft_consensus.cc:2272] T 39f0fb4f03d5428a86e3556149ff158d P 05b65d4b42f94b5c9c3cfc7729165b03 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:47.305963 24113 tablet_replica.cc:333] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03: stopping tablet replica
I20260709 13:12:47.306458 24113 raft_consensus.cc:2243] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:47.307482 24113 raft_consensus.cc:2272] T 3018ccc7f3b34dc7a851fa77512e01dc P 05b65d4b42f94b5c9c3cfc7729165b03 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:47.330310 24113 tablet_server.cc:196] TabletServer@127.23.140.65:0 shutdown complete.
I20260709 13:12:47.341853 24113 tablet_server.cc:179] TabletServer@127.23.140.66:0 shutting down...
I20260709 13:12:47.359731 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:12:47.360275 24113 tablet_replica.cc:333] T f715f76e5bd74699ad948688bb7d7378 P 2bb10e216823411d9acb895921ceb9c1: stopping tablet replica
I20260709 13:12:47.360764 24113 raft_consensus.cc:2243] T f715f76e5bd74699ad948688bb7d7378 P 2bb10e216823411d9acb895921ceb9c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:47.361196 24113 raft_consensus.cc:2272] T f715f76e5bd74699ad948688bb7d7378 P 2bb10e216823411d9acb895921ceb9c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:47.363291 24113 tablet_replica.cc:333] T fecd607247614897a832471c280a73b1 P 2bb10e216823411d9acb895921ceb9c1: stopping tablet replica
I20260709 13:12:47.363749 24113 raft_consensus.cc:2243] T fecd607247614897a832471c280a73b1 P 2bb10e216823411d9acb895921ceb9c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:47.364096 24113 raft_consensus.cc:2272] T fecd607247614897a832471c280a73b1 P 2bb10e216823411d9acb895921ceb9c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:47.366211 24113 tablet_replica.cc:333] T 3018ccc7f3b34dc7a851fa77512e01dc P 2bb10e216823411d9acb895921ceb9c1: stopping tablet replica
I20260709 13:12:47.366659 24113 raft_consensus.cc:2243] T 3018ccc7f3b34dc7a851fa77512e01dc P 2bb10e216823411d9acb895921ceb9c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:47.366988 24113 raft_consensus.cc:2272] T 3018ccc7f3b34dc7a851fa77512e01dc P 2bb10e216823411d9acb895921ceb9c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:47.368911 24113 tablet_replica.cc:333] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1: stopping tablet replica
I20260709 13:12:47.369395 24113 raft_consensus.cc:2243] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:47.370123 24113 raft_consensus.cc:2272] T 39f0fb4f03d5428a86e3556149ff158d P 2bb10e216823411d9acb895921ceb9c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:47.391718 24113 tablet_server.cc:196] TabletServer@127.23.140.66:0 shutdown complete.
I20260709 13:12:47.403398 24113 tablet_server.cc:179] TabletServer@127.23.140.67:0 shutting down...
I20260709 13:12:47.421357 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:12:47.421977 24113 tablet_replica.cc:333] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad: stopping tablet replica
I20260709 13:12:47.422521 24113 raft_consensus.cc:2243] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:47.423130 24113 pending_rounds.cc:70] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad: Trying to abort 1 pending ops.
I20260709 13:12:47.423292 24113 pending_rounds.cc:77] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7305636934498996224 op_type: NO_OP noop_request { }
I20260709 13:12:47.423537 24113 raft_consensus.cc:2889] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:12:47.423763 24113 raft_consensus.cc:2272] T f715f76e5bd74699ad948688bb7d7378 P cdd67666f92e478ab7ada7cb61f95dad [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:47.426074 24113 tablet_replica.cc:333] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad: stopping tablet replica
I20260709 13:12:47.426532 24113 raft_consensus.cc:2243] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:47.426878 24113 raft_consensus.cc:2272] T fecd607247614897a832471c280a73b1 P cdd67666f92e478ab7ada7cb61f95dad [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:47.429318 24113 tablet_replica.cc:333] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad: stopping tablet replica
I20260709 13:12:47.429770 24113 raft_consensus.cc:2243] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:47.430119 24113 raft_consensus.cc:2272] T 3018ccc7f3b34dc7a851fa77512e01dc P cdd67666f92e478ab7ada7cb61f95dad [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:47.432057 24113 tablet_replica.cc:333] T 39f0fb4f03d5428a86e3556149ff158d P cdd67666f92e478ab7ada7cb61f95dad: stopping tablet replica
I20260709 13:12:47.432474 24113 raft_consensus.cc:2243] T 39f0fb4f03d5428a86e3556149ff158d P cdd67666f92e478ab7ada7cb61f95dad [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:47.432812 24113 raft_consensus.cc:2272] T 39f0fb4f03d5428a86e3556149ff158d P cdd67666f92e478ab7ada7cb61f95dad [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:47.455031 24113 tablet_server.cc:196] TabletServer@127.23.140.67:0 shutdown complete.
I20260709 13:12:47.466744 24113 master.cc:562] Master@127.23.140.126:34333 shutting down...
I20260709 13:12:47.481652 24113 raft_consensus.cc:2243] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:47.482204 24113 raft_consensus.cc:2272] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:47.482532 24113 tablet_replica.cc:333] T 00000000000000000000000000000000 P dbe21f5caabf41c2abe6770f1e2780af: stopping tablet replica
I20260709 13:12:47.500723 24113 master.cc:584] Master@127.23.140.126:34333 shutdown complete.
[       OK ] LeaderRebalancerTest.MultiTableLeaderBalance (2678 ms)
[----------] 6 tests from LeaderRebalancerTest (354357 ms total)

[----------] 2 tests from FilterSoftDeletedTableTest
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0
I20260709 13:12:47.527734 24113 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.140.126:45465
I20260709 13:12:47.529249 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:12:47.534912  3508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:47.535996  3507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:47.536345  3510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:12:47.536949 24113 server_base.cc:1061] running on GCE node
I20260709 13:12:47.537797 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:12:47.537966 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:12:47.538141 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602767538127 us; error 0 us; skew 500 ppm
I20260709 13:12:47.538604 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:12:47.540769 24113 webserver.cc:533] Webserver started at http://127.23.140.126:43907/ using document root <none> and password file <none>
I20260709 13:12:47.541428 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:12:47.541599 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:12:47.541800 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:12:47.542764 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/master-0-root/instance:
uuid: "38fc5367513e426fa22096b7157fa610"
format_stamp: "Formatted at 2026-07-09 13:12:47 on dist-test-slave-k73n"
I20260709 13:12:47.546933 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 13:12:47.550069  3515 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:47.550823 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 13:12:47.551120 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/master-0-root
uuid: "38fc5367513e426fa22096b7157fa610"
format_stamp: "Formatted at 2026-07-09 13:12:47 on dist-test-slave-k73n"
I20260709 13:12:47.551396 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:12:47.573477 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:12:47.574455 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:12:47.608327 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.126:45465
I20260709 13:12:47.608433  3566 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.126:45465 every 8 connection(s)
I20260709 13:12:47.611992  3567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:47.622210  3567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610: Bootstrap starting.
I20260709 13:12:47.626488  3567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:47.630363  3567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610: No bootstrap required, opened a new log
I20260709 13:12:47.632308  3567 raft_consensus.cc:359] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fc5367513e426fa22096b7157fa610" member_type: VOTER }
I20260709 13:12:47.632652  3567 raft_consensus.cc:385] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:47.632828  3567 raft_consensus.cc:740] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38fc5367513e426fa22096b7157fa610, State: Initialized, Role: FOLLOWER
I20260709 13:12:47.633376  3567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fc5367513e426fa22096b7157fa610" member_type: VOTER }
I20260709 13:12:47.633760  3567 raft_consensus.cc:399] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:12:47.633950  3567 raft_consensus.cc:493] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:12:47.634163  3567 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:47.638474  3567 raft_consensus.cc:515] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fc5367513e426fa22096b7157fa610" member_type: VOTER }
I20260709 13:12:47.638903  3567 leader_election.cc:304] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38fc5367513e426fa22096b7157fa610; no voters: 
I20260709 13:12:47.639954  3567 leader_election.cc:290] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:12:47.640283  3570 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:47.641608  3570 raft_consensus.cc:697] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [term 1 LEADER]: Becoming Leader. State: Replica: 38fc5367513e426fa22096b7157fa610, State: Running, Role: LEADER
I20260709 13:12:47.642262  3570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fc5367513e426fa22096b7157fa610" member_type: VOTER }
I20260709 13:12:47.642870  3567 sys_catalog.cc:565] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:12:47.645156  3572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38fc5367513e426fa22096b7157fa610. Latest consensus state: current_term: 1 leader_uuid: "38fc5367513e426fa22096b7157fa610" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fc5367513e426fa22096b7157fa610" member_type: VOTER } }
I20260709 13:12:47.645102  3571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38fc5367513e426fa22096b7157fa610" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fc5367513e426fa22096b7157fa610" member_type: VOTER } }
I20260709 13:12:47.645964  3571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [sys.catalog]: This master's current role is: LEADER
I20260709 13:12:47.645874  3572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [sys.catalog]: This master's current role is: LEADER
I20260709 13:12:47.649494  3575 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:12:47.654451  3575 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:12:47.658958 24113 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:12:47.663278  3575 catalog_manager.cc:1383] Generated new cluster ID: 71d14c0d385940bba80cbe4aec9f2c44
I20260709 13:12:47.663560  3575 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:12:47.696411  3575 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:12:47.697742  3575 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:12:47.712630  3575 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610: Generated new TSK 0
I20260709 13:12:47.713285  3575 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:12:47.725026 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:12:47.730654  3588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:47.731932  3589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:47.733266  3591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:12:47.733286 24113 server_base.cc:1061] running on GCE node
I20260709 13:12:47.734417 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:12:47.734639 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:12:47.734807 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602767734785 us; error 0 us; skew 500 ppm
I20260709 13:12:47.735303 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:12:47.737509 24113 webserver.cc:533] Webserver started at http://127.23.140.65:38297/ using document root <none> and password file <none>
I20260709 13:12:47.737978 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:12:47.738191 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:12:47.738448 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:12:47.739463 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/ts-0-root/instance:
uuid: "31750ab24a654ede8e5d60d6ab45f76f"
format_stamp: "Formatted at 2026-07-09 13:12:47 on dist-test-slave-k73n"
I20260709 13:12:47.743827 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 13:12:47.747052  3596 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:47.747761 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 13:12:47.748039 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/ts-0-root
uuid: "31750ab24a654ede8e5d60d6ab45f76f"
format_stamp: "Formatted at 2026-07-09 13:12:47 on dist-test-slave-k73n"
I20260709 13:12:47.748322 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:12:47.768956 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:12:47.770133 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:12:47.771492 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:12:47.773661 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:12:47.773864 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:47.774132 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:12:47.774328 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:47.812182 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.65:40049
I20260709 13:12:47.812274  3658 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.65:40049 every 8 connection(s)
I20260709 13:12:47.817147 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:12:47.823892  3663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:47.824321  3664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:47.827648  3666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:12:47.829924  3659 heartbeater.cc:344] Connected to a master server at 127.23.140.126:45465
I20260709 13:12:47.830330  3659 heartbeater.cc:461] Registering TS with master...
I20260709 13:12:47.830927 24113 server_base.cc:1061] running on GCE node
I20260709 13:12:47.831238  3659 heartbeater.cc:507] Master 127.23.140.126:45465 requested a full tablet report, sending...
I20260709 13:12:47.832676 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:12:47.832934 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:12:47.833144 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602767833106 us; error 0 us; skew 500 ppm
I20260709 13:12:47.833609  3532 ts_manager.cc:194] Registered new tserver with Master: 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049)
I20260709 13:12:47.833956 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:12:47.836109  3532 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49366
I20260709 13:12:47.836606 24113 webserver.cc:533] Webserver started at http://127.23.140.66:41731/ using document root <none> and password file <none>
I20260709 13:12:47.837302 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:12:47.837523 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:12:47.837850 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:12:47.839150 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/ts-1-root/instance:
uuid: "2462c58306174ec1a384ea49fddf64c1"
format_stamp: "Formatted at 2026-07-09 13:12:47 on dist-test-slave-k73n"
I20260709 13:12:47.843279 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 13:12:47.846514  3671 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:47.847211 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260709 13:12:47.847451 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/ts-1-root
uuid: "2462c58306174ec1a384ea49fddf64c1"
format_stamp: "Formatted at 2026-07-09 13:12:47 on dist-test-slave-k73n"
I20260709 13:12:47.847687 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:12:47.862844 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:12:47.863786 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:12:47.865046 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:12:47.867194 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:12:47.867383 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:47.867632 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:12:47.867784 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:47.903898 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.66:35473
I20260709 13:12:47.904006  3733 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.66:35473 every 8 connection(s)
I20260709 13:12:47.908504 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:12:47.915408  3737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:47.917063  3738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:47.919173  3740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:12:47.919265 24113 server_base.cc:1061] running on GCE node
I20260709 13:12:47.920428 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 13:12:47.920590  3734 heartbeater.cc:344] Connected to a master server at 127.23.140.126:45465
W20260709 13:12:47.920624 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:12:47.921104 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602767921087 us; error 0 us; skew 500 ppm
I20260709 13:12:47.921216  3734 heartbeater.cc:461] Registering TS with master...
I20260709 13:12:47.921697 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:12:47.922094  3734 heartbeater.cc:507] Master 127.23.140.126:45465 requested a full tablet report, sending...
I20260709 13:12:47.924098 24113 webserver.cc:533] Webserver started at http://127.23.140.67:46139/ using document root <none> and password file <none>
I20260709 13:12:47.924369  3532 ts_manager.cc:194] Registered new tserver with Master: 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:47.924775 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:12:47.925077 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:12:47.925418 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:12:47.926198  3532 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49382
I20260709 13:12:47.926846 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/ts-2-root/instance:
uuid: "cb3f52aae763473cb09593496219dea8"
format_stamp: "Formatted at 2026-07-09 13:12:47 on dist-test-slave-k73n"
I20260709 13:12:47.931102 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.001s	sys 0.004s
I20260709 13:12:47.934113  3745 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:47.934779 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260709 13:12:47.935066 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/ts-2-root
uuid: "cb3f52aae763473cb09593496219dea8"
format_stamp: "Formatted at 2026-07-09 13:12:47 on dist-test-slave-k73n"
I20260709 13:12:47.935323 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602412999163-24113-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:12:47.950023 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:12:47.951004 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:12:47.952301 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:12:47.954505 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:12:47.954728 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:47.954955 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:12:47.955123 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:47.991098 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.67:35263
I20260709 13:12:47.991189  3807 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.67:35263 every 8 connection(s)
I20260709 13:12:48.004249  3808 heartbeater.cc:344] Connected to a master server at 127.23.140.126:45465
I20260709 13:12:48.004709  3808 heartbeater.cc:461] Registering TS with master...
I20260709 13:12:48.005455  3808 heartbeater.cc:507] Master 127.23.140.126:45465 requested a full tablet report, sending...
I20260709 13:12:48.007292  3532 ts_manager.cc:194] Registered new tserver with Master: cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:48.008172 24113 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014153522s
I20260709 13:12:48.008589  3532 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49394
I20260709 13:12:48.031414  3532 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49396:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:12:48.033936  3532 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:12:48.097862  3623 tablet_service.cc:1511] Processing CreateTablet for tablet ab14a07bccb94e4c82b34714c94ccb48 (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:12:48.098232  3622 tablet_service.cc:1511] Processing CreateTablet for tablet a5aed345cd424405a20d159a9f9762a0 (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:12:48.099301  3623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab14a07bccb94e4c82b34714c94ccb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.101028  3621 tablet_service.cc:1511] Processing CreateTablet for tablet 98f4adc13b1f41e0a6df7506b3f2b9a5 (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:12:48.102398  3621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98f4adc13b1f41e0a6df7506b3f2b9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.106142  3624 tablet_service.cc:1511] Processing CreateTablet for tablet c96b46ac691e478f946baba4fc981c1b (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:12:48.107481  3624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c96b46ac691e478f946baba4fc981c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.109802  3620 tablet_service.cc:1511] Processing CreateTablet for tablet e37f28b3b48449d3adecff96febd5598 (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:12:48.111133  3620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e37f28b3b48449d3adecff96febd5598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.114099  3619 tablet_service.cc:1511] Processing CreateTablet for tablet 1ccc13de4df944b1a6695b0f05c83ada (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:12:48.115419  3619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ccc13de4df944b1a6695b0f05c83ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.119009  3622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5aed345cd424405a20d159a9f9762a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.125545  3617 tablet_service.cc:1511] Processing CreateTablet for tablet 536c6c3f241342ec804d9400cbb23da9 (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:12:48.126978  3617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 536c6c3f241342ec804d9400cbb23da9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.127703  3698 tablet_service.cc:1511] Processing CreateTablet for tablet ab14a07bccb94e4c82b34714c94ccb48 (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:12:48.129045  3698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab14a07bccb94e4c82b34714c94ccb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.119031  3618 tablet_service.cc:1511] Processing CreateTablet for tablet 20eb2e33130740e38d072182154c715f (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:12:48.132663  3618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20eb2e33130740e38d072182154c715f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.134173  3616 tablet_service.cc:1511] Processing CreateTablet for tablet 17679967d4734e678c707f380844b93c (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:12:48.135479  3616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17679967d4734e678c707f380844b93c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.125617  3699 tablet_service.cc:1511] Processing CreateTablet for tablet c96b46ac691e478f946baba4fc981c1b (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:12:48.137213  3699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c96b46ac691e478f946baba4fc981c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.158416  3697 tablet_service.cc:1511] Processing CreateTablet for tablet a5aed345cd424405a20d159a9f9762a0 (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:12:48.160552  3693 tablet_service.cc:1511] Processing CreateTablet for tablet 20eb2e33130740e38d072182154c715f (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:12:48.160929  3692 tablet_service.cc:1511] Processing CreateTablet for tablet 536c6c3f241342ec804d9400cbb23da9 (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:12:48.162020  3696 tablet_service.cc:1511] Processing CreateTablet for tablet 98f4adc13b1f41e0a6df7506b3f2b9a5 (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:12:48.164036  3694 tablet_service.cc:1511] Processing CreateTablet for tablet 1ccc13de4df944b1a6695b0f05c83ada (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:12:48.165601  3695 tablet_service.cc:1511] Processing CreateTablet for tablet e37f28b3b48449d3adecff96febd5598 (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:12:48.160604  3697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5aed345cd424405a20d159a9f9762a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.167438  3693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20eb2e33130740e38d072182154c715f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.168171  3695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e37f28b3b48449d3adecff96febd5598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.169441  3694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ccc13de4df944b1a6695b0f05c83ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.173439  3696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98f4adc13b1f41e0a6df7506b3f2b9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.175355  3691 tablet_service.cc:1511] Processing CreateTablet for tablet 17679967d4734e678c707f380844b93c (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:12:48.176659  3691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17679967d4734e678c707f380844b93c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.177688  3692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 536c6c3f241342ec804d9400cbb23da9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.206357  3773 tablet_service.cc:1511] Processing CreateTablet for tablet c96b46ac691e478f946baba4fc981c1b (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:12:48.206624  3772 tablet_service.cc:1511] Processing CreateTablet for tablet ab14a07bccb94e4c82b34714c94ccb48 (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:12:48.207844  3773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c96b46ac691e478f946baba4fc981c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.208732  3771 tablet_service.cc:1511] Processing CreateTablet for tablet a5aed345cd424405a20d159a9f9762a0 (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:12:48.209694  3772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab14a07bccb94e4c82b34714c94ccb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.210548  3771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5aed345cd424405a20d159a9f9762a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.226408  3770 tablet_service.cc:1511] Processing CreateTablet for tablet 98f4adc13b1f41e0a6df7506b3f2b9a5 (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:12:48.227865  3770 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98f4adc13b1f41e0a6df7506b3f2b9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.231227  3769 tablet_service.cc:1511] Processing CreateTablet for tablet e37f28b3b48449d3adecff96febd5598 (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:12:48.234516  3768 tablet_service.cc:1511] Processing CreateTablet for tablet 1ccc13de4df944b1a6695b0f05c83ada (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:12:48.238505  3766 tablet_service.cc:1511] Processing CreateTablet for tablet 536c6c3f241342ec804d9400cbb23da9 (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:12:48.241618  3765 tablet_service.cc:1511] Processing CreateTablet for tablet 17679967d4734e678c707f380844b93c (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:12:48.233045  3769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e37f28b3b48449d3adecff96febd5598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.244799  3767 tablet_service.cc:1511] Processing CreateTablet for tablet 20eb2e33130740e38d072182154c715f (DEFAULT_TABLE table=test-workload [id=42c4ce043e4c4e2da5c2c9e36b56934a]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:12:48.250425  3822 tablet_bootstrap.cc:492] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f: Bootstrap starting.
I20260709 13:12:48.259002  3822 tablet_bootstrap.cc:654] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.259609  3765 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17679967d4734e678c707f380844b93c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.261538  3767 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20eb2e33130740e38d072182154c715f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.265535  3766 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 536c6c3f241342ec804d9400cbb23da9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.281287  3768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ccc13de4df944b1a6695b0f05c83ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:48.311272  3824 tablet_bootstrap.cc:492] T c96b46ac691e478f946baba4fc981c1b P cb3f52aae763473cb09593496219dea8: Bootstrap starting.
I20260709 13:12:48.311481  3823 tablet_bootstrap.cc:492] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1: Bootstrap starting.
I20260709 13:12:48.317878  3824 tablet_bootstrap.cc:654] T c96b46ac691e478f946baba4fc981c1b P cb3f52aae763473cb09593496219dea8: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.321267  3823 tablet_bootstrap.cc:654] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.321894  3822 tablet_bootstrap.cc:492] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f: No bootstrap required, opened a new log
I20260709 13:12:48.322588  3822 ts_tablet_manager.cc:1403] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent bootstrapping tablet: real 0.073s	user 0.022s	sys 0.033s
I20260709 13:12:48.326411  3822 raft_consensus.cc:359] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.327139  3822 raft_consensus.cc:385] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.327427  3822 raft_consensus.cc:740] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.334194  3823 tablet_bootstrap.cc:492] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1: No bootstrap required, opened a new log
I20260709 13:12:48.334705  3823 ts_tablet_manager.cc:1403] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1: Time spent bootstrapping tablet: real 0.024s	user 0.009s	sys 0.001s
I20260709 13:12:48.334853  3824 tablet_bootstrap.cc:492] T c96b46ac691e478f946baba4fc981c1b P cb3f52aae763473cb09593496219dea8: No bootstrap required, opened a new log
I20260709 13:12:48.331285  3822 consensus_queue.cc:260] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.335402  3824 ts_tablet_manager.cc:1403] T c96b46ac691e478f946baba4fc981c1b P cb3f52aae763473cb09593496219dea8: Time spent bootstrapping tablet: real 0.024s	user 0.007s	sys 0.008s
I20260709 13:12:48.337185  3659 heartbeater.cc:499] Master 127.23.140.126:45465 was elected leader, sending a full tablet report...
I20260709 13:12:48.337668  3822 ts_tablet_manager.cc:1434] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.003s
I20260709 13:12:48.338743  3822 tablet_bootstrap.cc:492] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f: Bootstrap starting.
I20260709 13:12:48.342198  3824 raft_consensus.cc:359] T c96b46ac691e478f946baba4fc981c1b P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.342993  3824 raft_consensus.cc:385] T c96b46ac691e478f946baba4fc981c1b P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.343334  3824 raft_consensus.cc:740] T c96b46ac691e478f946baba4fc981c1b P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.343746  3822 tablet_bootstrap.cc:654] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.344128  3824 consensus_queue.cc:260] T c96b46ac691e478f946baba4fc981c1b P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.346129  3823 raft_consensus.cc:359] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.346948  3823 raft_consensus.cc:385] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.347258  3823 raft_consensus.cc:740] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.347751  3824 ts_tablet_manager.cc:1434] T c96b46ac691e478f946baba4fc981c1b P cb3f52aae763473cb09593496219dea8: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.002s
I20260709 13:12:48.347627  3808 heartbeater.cc:499] Master 127.23.140.126:45465 was elected leader, sending a full tablet report...
I20260709 13:12:48.348100  3823 consensus_queue.cc:260] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.348958  3824 tablet_bootstrap.cc:492] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8: Bootstrap starting.
I20260709 13:12:48.352850  3822 tablet_bootstrap.cc:492] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f: No bootstrap required, opened a new log
I20260709 13:12:48.353555  3822 ts_tablet_manager.cc:1403] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f: Time spent bootstrapping tablet: real 0.015s	user 0.012s	sys 0.000s
I20260709 13:12:48.355466  3823 ts_tablet_manager.cc:1434] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.003s
I20260709 13:12:48.355923  3824 tablet_bootstrap.cc:654] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.356297  3734 heartbeater.cc:499] Master 127.23.140.126:45465 was elected leader, sending a full tablet report...
I20260709 13:12:48.356546  3823 tablet_bootstrap.cc:492] T ab14a07bccb94e4c82b34714c94ccb48 P 2462c58306174ec1a384ea49fddf64c1: Bootstrap starting.
I20260709 13:12:48.356693  3822 raft_consensus.cc:359] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.357568  3822 raft_consensus.cc:385] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.357915  3822 raft_consensus.cc:740] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.358713  3822 consensus_queue.cc:260] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.361192  3822 ts_tablet_manager.cc:1434] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:12:48.362150  3822 tablet_bootstrap.cc:492] T 1ccc13de4df944b1a6695b0f05c83ada P 31750ab24a654ede8e5d60d6ab45f76f: Bootstrap starting.
I20260709 13:12:48.364296  3823 tablet_bootstrap.cc:654] T ab14a07bccb94e4c82b34714c94ccb48 P 2462c58306174ec1a384ea49fddf64c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.368134  3822 tablet_bootstrap.cc:654] T 1ccc13de4df944b1a6695b0f05c83ada P 31750ab24a654ede8e5d60d6ab45f76f: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.370361  3824 tablet_bootstrap.cc:492] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8: No bootstrap required, opened a new log
I20260709 13:12:48.370867  3824 ts_tablet_manager.cc:1403] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8: Time spent bootstrapping tablet: real 0.022s	user 0.007s	sys 0.013s
I20260709 13:12:48.373402  3824 raft_consensus.cc:359] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.373988  3824 raft_consensus.cc:385] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.374192  3824 raft_consensus.cc:740] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.374955  3824 consensus_queue.cc:260] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.377771  3822 tablet_bootstrap.cc:492] T 1ccc13de4df944b1a6695b0f05c83ada P 31750ab24a654ede8e5d60d6ab45f76f: No bootstrap required, opened a new log
I20260709 13:12:48.378284  3822 ts_tablet_manager.cc:1403] T 1ccc13de4df944b1a6695b0f05c83ada P 31750ab24a654ede8e5d60d6ab45f76f: Time spent bootstrapping tablet: real 0.016s	user 0.006s	sys 0.007s
I20260709 13:12:48.376999  3824 ts_tablet_manager.cc:1434] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.002s
I20260709 13:12:48.380853  3822 raft_consensus.cc:359] T 1ccc13de4df944b1a6695b0f05c83ada P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.381714  3822 raft_consensus.cc:385] T 1ccc13de4df944b1a6695b0f05c83ada P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.381712  3824 tablet_bootstrap.cc:492] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8: Bootstrap starting.
I20260709 13:12:48.382187  3822 raft_consensus.cc:740] T 1ccc13de4df944b1a6695b0f05c83ada P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.383122  3823 tablet_bootstrap.cc:492] T ab14a07bccb94e4c82b34714c94ccb48 P 2462c58306174ec1a384ea49fddf64c1: No bootstrap required, opened a new log
I20260709 13:12:48.383031  3822 consensus_queue.cc:260] T 1ccc13de4df944b1a6695b0f05c83ada P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.383759  3823 ts_tablet_manager.cc:1403] T ab14a07bccb94e4c82b34714c94ccb48 P 2462c58306174ec1a384ea49fddf64c1: Time spent bootstrapping tablet: real 0.027s	user 0.011s	sys 0.005s
I20260709 13:12:48.385524  3822 ts_tablet_manager.cc:1434] T 1ccc13de4df944b1a6695b0f05c83ada P 31750ab24a654ede8e5d60d6ab45f76f: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:12:48.386554  3822 tablet_bootstrap.cc:492] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f: Bootstrap starting.
I20260709 13:12:48.386938  3823 raft_consensus.cc:359] T ab14a07bccb94e4c82b34714c94ccb48 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.387753  3823 raft_consensus.cc:385] T ab14a07bccb94e4c82b34714c94ccb48 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.388085  3823 raft_consensus.cc:740] T ab14a07bccb94e4c82b34714c94ccb48 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.389003  3823 consensus_queue.cc:260] T ab14a07bccb94e4c82b34714c94ccb48 P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.391070  3824 tablet_bootstrap.cc:654] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.391271  3823 ts_tablet_manager.cc:1434] T ab14a07bccb94e4c82b34714c94ccb48 P 2462c58306174ec1a384ea49fddf64c1: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:12:48.392409  3823 tablet_bootstrap.cc:492] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1: Bootstrap starting.
I20260709 13:12:48.392970  3822 tablet_bootstrap.cc:654] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.399500  3823 tablet_bootstrap.cc:654] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.402400  3824 tablet_bootstrap.cc:492] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8: No bootstrap required, opened a new log
I20260709 13:12:48.402896  3824 ts_tablet_manager.cc:1403] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8: Time spent bootstrapping tablet: real 0.021s	user 0.014s	sys 0.003s
I20260709 13:12:48.405603  3824 raft_consensus.cc:359] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.406313  3824 raft_consensus.cc:385] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.406596  3824 raft_consensus.cc:740] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.408550  3824 consensus_queue.cc:260] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.410691  3824 ts_tablet_manager.cc:1434] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.004s
I20260709 13:12:48.411165  3822 tablet_bootstrap.cc:492] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f: No bootstrap required, opened a new log
I20260709 13:12:48.411662  3824 tablet_bootstrap.cc:492] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8: Bootstrap starting.
I20260709 13:12:48.411914  3822 ts_tablet_manager.cc:1403] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent bootstrapping tablet: real 0.026s	user 0.016s	sys 0.008s
I20260709 13:12:48.415922  3822 raft_consensus.cc:359] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.416776  3822 raft_consensus.cc:385] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.417058  3822 raft_consensus.cc:740] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.418311  3824 tablet_bootstrap.cc:654] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.417955  3822 consensus_queue.cc:260] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.420895  3822 ts_tablet_manager.cc:1434] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.003s
I20260709 13:12:48.421928  3822 tablet_bootstrap.cc:492] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f: Bootstrap starting.
I20260709 13:12:48.427306  3823 tablet_bootstrap.cc:492] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1: No bootstrap required, opened a new log
I20260709 13:12:48.427337  3822 tablet_bootstrap.cc:654] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.428157  3823 ts_tablet_manager.cc:1403] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1: Time spent bootstrapping tablet: real 0.036s	user 0.017s	sys 0.008s
I20260709 13:12:48.428933  3824 tablet_bootstrap.cc:492] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8: No bootstrap required, opened a new log
I20260709 13:12:48.429492  3824 ts_tablet_manager.cc:1403] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8: Time spent bootstrapping tablet: real 0.018s	user 0.005s	sys 0.011s
I20260709 13:12:48.432595  3824 raft_consensus.cc:359] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.433337  3824 raft_consensus.cc:385] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.433642  3824 raft_consensus.cc:740] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.434372  3824 consensus_queue.cc:260] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.435106  3823 raft_consensus.cc:359] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.436139  3823 raft_consensus.cc:385] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.436327  3824 ts_tablet_manager.cc:1434] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.003s
I20260709 13:12:48.436486  3823 raft_consensus.cc:740] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.437621  3824 tablet_bootstrap.cc:492] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8: Bootstrap starting.
I20260709 13:12:48.437690  3823 consensus_queue.cc:260] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.439344  3822 tablet_bootstrap.cc:492] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f: No bootstrap required, opened a new log
I20260709 13:12:48.439891  3822 ts_tablet_manager.cc:1403] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent bootstrapping tablet: real 0.018s	user 0.011s	sys 0.004s
I20260709 13:12:48.440230  3823 ts_tablet_manager.cc:1434] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 13:12:48.441330  3823 tablet_bootstrap.cc:492] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1: Bootstrap starting.
I20260709 13:12:48.442883  3822 raft_consensus.cc:359] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.443661  3822 raft_consensus.cc:385] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.443956  3822 raft_consensus.cc:740] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.444896  3822 consensus_queue.cc:260] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.447476  3822 ts_tablet_manager.cc:1434] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:12:48.448446  3822 tablet_bootstrap.cc:492] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f: Bootstrap starting.
I20260709 13:12:48.448475  3823 tablet_bootstrap.cc:654] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.452543  3828 raft_consensus.cc:493] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:48.453234  3828 raft_consensus.cc:515] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.453989  3824 tablet_bootstrap.cc:654] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.459576  3828 leader_election.cc:290] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:48.460057  3823 tablet_bootstrap.cc:492] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1: No bootstrap required, opened a new log
I20260709 13:12:48.460851  3823 ts_tablet_manager.cc:1403] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1: Time spent bootstrapping tablet: real 0.020s	user 0.011s	sys 0.007s
I20260709 13:12:48.464690  3823 raft_consensus.cc:359] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.465610  3823 raft_consensus.cc:385] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.465932  3823 raft_consensus.cc:740] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.466827  3823 consensus_queue.cc:260] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.470841  3824 tablet_bootstrap.cc:492] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8: No bootstrap required, opened a new log
I20260709 13:12:48.471380  3824 ts_tablet_manager.cc:1403] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8: Time spent bootstrapping tablet: real 0.034s	user 0.020s	sys 0.001s
I20260709 13:12:48.473616  3823 ts_tablet_manager.cc:1434] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.004s
I20260709 13:12:48.474748  3823 tablet_bootstrap.cc:492] T c96b46ac691e478f946baba4fc981c1b P 2462c58306174ec1a384ea49fddf64c1: Bootstrap starting.
I20260709 13:12:48.474459  3824 raft_consensus.cc:359] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.475247  3824 raft_consensus.cc:385] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.475561  3824 raft_consensus.cc:740] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.468297  3822 tablet_bootstrap.cc:654] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.476428  3824 consensus_queue.cc:260] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.482678  3824 ts_tablet_manager.cc:1434] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.002s
I20260709 13:12:48.482808  3823 tablet_bootstrap.cc:654] T c96b46ac691e478f946baba4fc981c1b P 2462c58306174ec1a384ea49fddf64c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.483729  3824 tablet_bootstrap.cc:492] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8: Bootstrap starting.
I20260709 13:12:48.490471  3822 tablet_bootstrap.cc:492] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f: No bootstrap required, opened a new log
I20260709 13:12:48.491223  3822 ts_tablet_manager.cc:1403] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f: Time spent bootstrapping tablet: real 0.043s	user 0.006s	sys 0.006s
I20260709 13:12:48.495272  3822 raft_consensus.cc:359] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.499837  3783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c96b46ac691e478f946baba4fc981c1b" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:48.500559  3824 tablet_bootstrap.cc:654] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.500833  3783 raft_consensus.cc:2468] T c96b46ac691e478f946baba4fc981c1b P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 0.
I20260709 13:12:48.502609  3597 leader_election.cc:304] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:48.504633  3828 raft_consensus.cc:2804] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:48.505200  3828 raft_consensus.cc:493] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:48.505638  3828 raft_consensus.cc:3060] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.511188  3709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c96b46ac691e478f946baba4fc981c1b" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
I20260709 13:12:48.496244  3822 raft_consensus.cc:385] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.512195  3822 raft_consensus.cc:740] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Initialized, Role: FOLLOWER
W20260709 13:12:48.512750  3597 leader_election.cc:343] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:48.513674  3823 tablet_bootstrap.cc:492] T c96b46ac691e478f946baba4fc981c1b P 2462c58306174ec1a384ea49fddf64c1: No bootstrap required, opened a new log
I20260709 13:12:48.513224  3822 consensus_queue.cc:260] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.514305  3823 ts_tablet_manager.cc:1403] T c96b46ac691e478f946baba4fc981c1b P 2462c58306174ec1a384ea49fddf64c1: Time spent bootstrapping tablet: real 0.040s	user 0.029s	sys 0.009s
I20260709 13:12:48.517445  3823 raft_consensus.cc:359] T c96b46ac691e478f946baba4fc981c1b P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.517813  3824 tablet_bootstrap.cc:492] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8: No bootstrap required, opened a new log
I20260709 13:12:48.518277  3823 raft_consensus.cc:385] T c96b46ac691e478f946baba4fc981c1b P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.518142  3828 raft_consensus.cc:515] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.518611  3824 ts_tablet_manager.cc:1403] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8: Time spent bootstrapping tablet: real 0.035s	user 0.015s	sys 0.010s
I20260709 13:12:48.520867  3822 ts_tablet_manager.cc:1434] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f: Time spent starting tablet: real 0.029s	user 0.008s	sys 0.007s
I20260709 13:12:48.521107  3709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c96b46ac691e478f946baba4fc981c1b" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:48.521865  3822 tablet_bootstrap.cc:492] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f: Bootstrap starting.
I20260709 13:12:48.522068  3824 raft_consensus.cc:359] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.522753  3824 raft_consensus.cc:385] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.523023  3824 raft_consensus.cc:740] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.523319  3783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c96b46ac691e478f946baba4fc981c1b" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:48.523910  3783 raft_consensus.cc:3060] T c96b46ac691e478f946baba4fc981c1b P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.518725  3823 raft_consensus.cc:740] T c96b46ac691e478f946baba4fc981c1b P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.523855  3824 consensus_queue.cc:260] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.529107  3822 tablet_bootstrap.cc:654] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.531383  3824 ts_tablet_manager.cc:1434] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.002s
I20260709 13:12:48.532521  3824 tablet_bootstrap.cc:492] T 536c6c3f241342ec804d9400cbb23da9 P cb3f52aae763473cb09593496219dea8: Bootstrap starting.
I20260709 13:12:48.532794  3828 leader_election.cc:290] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 election: Requested vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:48.532361  3783 raft_consensus.cc:2468] T c96b46ac691e478f946baba4fc981c1b P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 1.
I20260709 13:12:48.525712  3823 consensus_queue.cc:260] T c96b46ac691e478f946baba4fc981c1b P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.534889  3709 raft_consensus.cc:3060] T c96b46ac691e478f946baba4fc981c1b P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.535284  3597 leader_election.cc:304] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:48.536509  3828 raft_consensus.cc:2804] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:48.542395  3709 raft_consensus.cc:2468] T c96b46ac691e478f946baba4fc981c1b P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 1.
I20260709 13:12:48.544078  3824 tablet_bootstrap.cc:654] T 536c6c3f241342ec804d9400cbb23da9 P cb3f52aae763473cb09593496219dea8: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.545661  3823 ts_tablet_manager.cc:1434] T c96b46ac691e478f946baba4fc981c1b P 2462c58306174ec1a384ea49fddf64c1: Time spent starting tablet: real 0.031s	user 0.007s	sys 0.000s
I20260709 13:12:48.547681  3823 tablet_bootstrap.cc:492] T a5aed345cd424405a20d159a9f9762a0 P 2462c58306174ec1a384ea49fddf64c1: Bootstrap starting.
I20260709 13:12:48.547910  3828 raft_consensus.cc:697] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Becoming Leader. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:48.549157  3828 consensus_queue.cc:237] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.554656  3823 tablet_bootstrap.cc:654] T a5aed345cd424405a20d159a9f9762a0 P 2462c58306174ec1a384ea49fddf64c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.558291  3822 tablet_bootstrap.cc:492] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f: No bootstrap required, opened a new log
I20260709 13:12:48.558944  3822 ts_tablet_manager.cc:1403] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent bootstrapping tablet: real 0.037s	user 0.013s	sys 0.019s
I20260709 13:12:48.562774  3822 raft_consensus.cc:359] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.563808  3822 raft_consensus.cc:385] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.564190  3822 raft_consensus.cc:740] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.567279  3822 consensus_queue.cc:260] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.569047  3824 tablet_bootstrap.cc:492] T 536c6c3f241342ec804d9400cbb23da9 P cb3f52aae763473cb09593496219dea8: No bootstrap required, opened a new log
I20260709 13:12:48.569283  3822 ts_tablet_manager.cc:1434] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.002s
I20260709 13:12:48.569640  3824 ts_tablet_manager.cc:1403] T 536c6c3f241342ec804d9400cbb23da9 P cb3f52aae763473cb09593496219dea8: Time spent bootstrapping tablet: real 0.037s	user 0.013s	sys 0.012s
I20260709 13:12:48.570399  3822 tablet_bootstrap.cc:492] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f: Bootstrap starting.
I20260709 13:12:48.571193  3823 tablet_bootstrap.cc:492] T a5aed345cd424405a20d159a9f9762a0 P 2462c58306174ec1a384ea49fddf64c1: No bootstrap required, opened a new log
I20260709 13:12:48.571012  3529 catalog_manager.cc:5697] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f reported cstate change: term changed from 0 to 1, leader changed from <none> to 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65). New cstate: current_term: 1 leader_uuid: "31750ab24a654ede8e5d60d6ab45f76f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:48.572150  3823 ts_tablet_manager.cc:1403] T a5aed345cd424405a20d159a9f9762a0 P 2462c58306174ec1a384ea49fddf64c1: Time spent bootstrapping tablet: real 0.025s	user 0.009s	sys 0.011s
I20260709 13:12:48.572883  3824 raft_consensus.cc:359] T 536c6c3f241342ec804d9400cbb23da9 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.573688  3824 raft_consensus.cc:385] T 536c6c3f241342ec804d9400cbb23da9 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.573998  3824 raft_consensus.cc:740] T 536c6c3f241342ec804d9400cbb23da9 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.574756  3824 consensus_queue.cc:260] T 536c6c3f241342ec804d9400cbb23da9 P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.576151  3823 raft_consensus.cc:359] T a5aed345cd424405a20d159a9f9762a0 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.579546  3823 raft_consensus.cc:385] T a5aed345cd424405a20d159a9f9762a0 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.579879  3823 raft_consensus.cc:740] T a5aed345cd424405a20d159a9f9762a0 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.580134  3822 tablet_bootstrap.cc:654] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.580865  3823 consensus_queue.cc:260] T a5aed345cd424405a20d159a9f9762a0 P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.581300  3828 raft_consensus.cc:493] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:48.577189  3824 ts_tablet_manager.cc:1434] T 536c6c3f241342ec804d9400cbb23da9 P cb3f52aae763473cb09593496219dea8: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.002s
I20260709 13:12:48.581888  3828 raft_consensus.cc:515] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.583063  3824 tablet_bootstrap.cc:492] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8: Bootstrap starting.
I20260709 13:12:48.583966  3828 leader_election.cc:290] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:48.585891  3823 ts_tablet_manager.cc:1434] T a5aed345cd424405a20d159a9f9762a0 P 2462c58306174ec1a384ea49fddf64c1: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 13:12:48.586979  3823 tablet_bootstrap.cc:492] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1: Bootstrap starting.
I20260709 13:12:48.587409  3709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "536c6c3f241342ec804d9400cbb23da9" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
I20260709 13:12:48.588084  3709 raft_consensus.cc:2468] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 0.
I20260709 13:12:48.589555  3783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "536c6c3f241342ec804d9400cbb23da9" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:48.590209  3783 raft_consensus.cc:2468] T 536c6c3f241342ec804d9400cbb23da9 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 0.
I20260709 13:12:48.591686  3824 tablet_bootstrap.cc:654] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.593340  3823 tablet_bootstrap.cc:654] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.594074  3597 leader_election.cc:304] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:48.595531  3828 raft_consensus.cc:493] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:48.598171  3849 raft_consensus.cc:2804] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:48.603343  3849 raft_consensus.cc:493] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:48.603778  3849 raft_consensus.cc:3060] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.596139  3828 raft_consensus.cc:515] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.613914  3829 raft_consensus.cc:493] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:48.614586  3829 raft_consensus.cc:515] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.615105  3709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5aed345cd424405a20d159a9f9762a0" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
I20260709 13:12:48.616042  3709 raft_consensus.cc:2468] T a5aed345cd424405a20d159a9f9762a0 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 0.
I20260709 13:12:48.617654  3597 leader_election.cc:304] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, 31750ab24a654ede8e5d60d6ab45f76f; no voters: 
I20260709 13:12:48.618952  3847 raft_consensus.cc:2804] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:48.619302  3847 raft_consensus.cc:493] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:48.622728  3859 raft_consensus.cc:493] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:48.619632  3847 raft_consensus.cc:3060] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.620256  3829 leader_election.cc:290] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:48.628043  3859 raft_consensus.cc:515] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.629848  3829 raft_consensus.cc:493] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:48.632268  3823 tablet_bootstrap.cc:492] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1: No bootstrap required, opened a new log
I20260709 13:12:48.632094  3829 raft_consensus.cc:515] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.632838  3823 ts_tablet_manager.cc:1403] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1: Time spent bootstrapping tablet: real 0.046s	user 0.013s	sys 0.002s
I20260709 13:12:48.633899  3783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5aed345cd424405a20d159a9f9762a0" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:48.634560  3783 raft_consensus.cc:2468] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 0.
I20260709 13:12:48.634804  3829 leader_election.cc:290] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:48.637666  3822 tablet_bootstrap.cc:492] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f: No bootstrap required, opened a new log
I20260709 13:12:48.638290  3822 ts_tablet_manager.cc:1403] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent bootstrapping tablet: real 0.068s	user 0.010s	sys 0.037s
I20260709 13:12:48.640758  3849 raft_consensus.cc:515] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.640924  3822 raft_consensus.cc:359] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.641588  3822 raft_consensus.cc:385] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.641850  3822 raft_consensus.cc:740] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.643003  3849 leader_election.cc:290] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 election: Requested vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:48.643775  3849 raft_consensus.cc:493] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:48.631407  3828 leader_election.cc:290] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:48.646955  3822 consensus_queue.cc:260] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.648183  3823 raft_consensus.cc:359] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.648672  3709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f4adc13b1f41e0a6df7506b3f2b9a5" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
I20260709 13:12:48.648955  3823 raft_consensus.cc:385] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.644861  3847 raft_consensus.cc:515] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.658581  3824 tablet_bootstrap.cc:492] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8: No bootstrap required, opened a new log
I20260709 13:12:48.659219  3824 ts_tablet_manager.cc:1403] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8: Time spent bootstrapping tablet: real 0.076s	user 0.014s	sys 0.004s
I20260709 13:12:48.649405  3709 raft_consensus.cc:2468] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 0.
I20260709 13:12:48.661187  3823 raft_consensus.cc:740] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.649820  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab14a07bccb94e4c82b34714c94ccb48" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
I20260709 13:12:48.658978  3847 leader_election.cc:290] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 election: Requested vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:48.662358  3708 raft_consensus.cc:2468] T ab14a07bccb94e4c82b34714c94ccb48 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 0.
I20260709 13:12:48.662500  3823 consensus_queue.cc:260] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.665297  3823 ts_tablet_manager.cc:1434] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1: Time spent starting tablet: real 0.032s	user 0.008s	sys 0.000s
I20260709 13:12:48.666304  3823 tablet_bootstrap.cc:492] T 17679967d4734e678c707f380844b93c P 2462c58306174ec1a384ea49fddf64c1: Bootstrap starting.
I20260709 13:12:48.661540  3746 leader_election.cc:304] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:48.644279  3849 raft_consensus.cc:515] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.669523  3746 leader_election.cc:304] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:48.670574  3829 raft_consensus.cc:2804] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:48.670955  3829 raft_consensus.cc:493] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:48.671104  3849 leader_election.cc:290] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:48.671288  3829 raft_consensus.cc:3060] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.673342  3859 leader_election.cc:290] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:48.673915  3859 raft_consensus.cc:2804] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:48.674222  3859 raft_consensus.cc:493] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:48.674526  3859 raft_consensus.cc:3060] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.674955  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17679967d4734e678c707f380844b93c" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
W20260709 13:12:48.677155  3746 leader_election.cc:343] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:48.679224  3829 raft_consensus.cc:515] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.682785  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f4adc13b1f41e0a6df7506b3f2b9a5" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:48.682713  3824 raft_consensus.cc:359] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.683565  3708 raft_consensus.cc:3060] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.687372  3823 tablet_bootstrap.cc:654] T 17679967d4734e678c707f380844b93c P 2462c58306174ec1a384ea49fddf64c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.683593  3824 raft_consensus.cc:385] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.689558  3824 raft_consensus.cc:740] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.690274  3824 consensus_queue.cc:260] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.691448  3859 raft_consensus.cc:515] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.692389  3824 ts_tablet_manager.cc:1434] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8: Time spent starting tablet: real 0.033s	user 0.006s	sys 0.000s
I20260709 13:12:48.693403  3824 tablet_bootstrap.cc:492] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8: Bootstrap starting.
I20260709 13:12:48.693832  3859 leader_election.cc:290] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:48.695152  3709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab14a07bccb94e4c82b34714c94ccb48" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:48.695611  3831 raft_consensus.cc:493] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:48.696007  3709 raft_consensus.cc:3060] T ab14a07bccb94e4c82b34714c94ccb48 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:12:48.699268  3580 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:12:48.703112  3634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f4adc13b1f41e0a6df7506b3f2b9a5" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f" is_pre_election: true
I20260709 13:12:48.703909  3634 raft_consensus.cc:2468] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 0.
I20260709 13:12:48.705286  3633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab14a07bccb94e4c82b34714c94ccb48" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f" is_pre_election: true
I20260709 13:12:48.706020  3633 raft_consensus.cc:2468] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 0.
I20260709 13:12:48.696240  3831 raft_consensus.cc:515] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.707188  3632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17679967d4734e678c707f380844b93c" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f" is_pre_election: true
I20260709 13:12:48.708406  3709 raft_consensus.cc:2468] T ab14a07bccb94e4c82b34714c94ccb48 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 1.
I20260709 13:12:48.708601  3631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f4adc13b1f41e0a6df7506b3f2b9a5" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:48.709362  3631 raft_consensus.cc:3060] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.712039  3630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab14a07bccb94e4c82b34714c94ccb48" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:48.712761  3630 raft_consensus.cc:3060] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.716938  3831 leader_election.cc:290] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
W20260709 13:12:48.720628  3749 leader_election.cc:343] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:48.721298  3749 leader_election.cc:304] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb3f52aae763473cb09593496219dea8; no voters: 2462c58306174ec1a384ea49fddf64c1, 31750ab24a654ede8e5d60d6ab45f76f
I20260709 13:12:48.722240  3859 raft_consensus.cc:2749] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:12:48.722913  3829 leader_election.cc:290] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:48.723881  3630 raft_consensus.cc:2468] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 1.
I20260709 13:12:48.731029  3783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "536c6c3f241342ec804d9400cbb23da9" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:48.731890  3783 raft_consensus.cc:3060] T 536c6c3f241342ec804d9400cbb23da9 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.739190  3782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5aed345cd424405a20d159a9f9762a0" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:48.740003  3782 raft_consensus.cc:3060] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.741297  3822 ts_tablet_manager.cc:1434] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent starting tablet: real 0.103s	user 0.003s	sys 0.005s
I20260709 13:12:48.743067  3783 raft_consensus.cc:2468] T 536c6c3f241342ec804d9400cbb23da9 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 1.
I20260709 13:12:48.743321  3822 tablet_bootstrap.cc:492] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f: Bootstrap starting.
I20260709 13:12:48.746945  3707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "536c6c3f241342ec804d9400cbb23da9" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:48.747781  3707 raft_consensus.cc:3060] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.750639  3706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5aed345cd424405a20d159a9f9762a0" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:48.750953  3782 raft_consensus.cc:2468] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 1.
I20260709 13:12:48.751567  3706 raft_consensus.cc:3060] T a5aed345cd424405a20d159a9f9762a0 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.753460  3746 leader_election.cc:304] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:48.761941  3706 raft_consensus.cc:2468] T a5aed345cd424405a20d159a9f9762a0 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 1.
I20260709 13:12:48.763471  3862 raft_consensus.cc:493] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:48.763722  3822 tablet_bootstrap.cc:654] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.764025  3862 raft_consensus.cc:515] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.754458  3829 raft_consensus.cc:2804] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:48.767446  3631 raft_consensus.cc:2468] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 1.
I20260709 13:12:48.766461  3862 leader_election.cc:290] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:48.768297  3859 raft_consensus.cc:493] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:48.712734  3708 raft_consensus.cc:2468] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 1.
I20260709 13:12:48.768976  3859 raft_consensus.cc:515] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.769253  3749 leader_election.cc:304] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:48.770560  3861 raft_consensus.cc:2804] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:48.771124  3859 leader_election.cc:290] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:48.773180  3631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ccc13de4df944b1a6695b0f05c83ada" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f" is_pre_election: true
I20260709 13:12:48.774068  3631 raft_consensus.cc:2468] T 1ccc13de4df944b1a6695b0f05c83ada P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 0.
I20260709 13:12:48.774863  3709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f4adc13b1f41e0a6df7506b3f2b9a5" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
I20260709 13:12:48.774972  3597 leader_election.cc:304] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:48.775487  3749 leader_election.cc:304] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:48.775543  3862 raft_consensus.cc:493] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:48.780409  3782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f4adc13b1f41e0a6df7506b3f2b9a5" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:48.774322  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ccc13de4df944b1a6695b0f05c83ada" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
I20260709 13:12:48.780790  3849 raft_consensus.cc:2804] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:48.790109  3859 raft_consensus.cc:2804] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:48.791674  3859 raft_consensus.cc:493] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:48.791985  3859 raft_consensus.cc:3060] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.792723  3823 tablet_bootstrap.cc:492] T 17679967d4734e678c707f380844b93c P 2462c58306174ec1a384ea49fddf64c1: No bootstrap required, opened a new log
I20260709 13:12:48.793234  3823 ts_tablet_manager.cc:1403] T 17679967d4734e678c707f380844b93c P 2462c58306174ec1a384ea49fddf64c1: Time spent bootstrapping tablet: real 0.127s	user 0.040s	sys 0.047s
I20260709 13:12:48.790263  3849 raft_consensus.cc:697] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Becoming Leader. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:48.795099  3849 consensus_queue.cc:237] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.795964  3823 raft_consensus.cc:359] T 17679967d4734e678c707f380844b93c P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.796643  3823 raft_consensus.cc:385] T 17679967d4734e678c707f380844b93c P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.796882  3823 raft_consensus.cc:740] T 17679967d4734e678c707f380844b93c P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.780498  3862 raft_consensus.cc:515] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.788338  3709 raft_consensus.cc:2393] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31750ab24a654ede8e5d60d6ab45f76f in current term 1: Already voted for candidate cb3f52aae763473cb09593496219dea8 in this term.
I20260709 13:12:48.797881  3823 consensus_queue.cc:260] T 17679967d4734e678c707f380844b93c P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.799726  3862 leader_election.cc:290] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:48.800254  3783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ccc13de4df944b1a6695b0f05c83ada" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:48.798453  3829 raft_consensus.cc:697] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Becoming Leader. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:48.780930  3597 leader_election.cc:304] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:48.804409  3862 raft_consensus.cc:2804] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:48.805224  3862 raft_consensus.cc:697] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Becoming Leader. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:48.805531  3706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e37f28b3b48449d3adecff96febd5598" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
I20260709 13:12:48.789067  3708 raft_consensus.cc:2468] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 0.
I20260709 13:12:48.806222  3862 consensus_queue.cc:237] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.806933  3781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e37f28b3b48449d3adecff96febd5598" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:48.807664  3781 raft_consensus.cc:2468] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 0.
I20260709 13:12:48.809448  3824 tablet_bootstrap.cc:654] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.809942  3859 raft_consensus.cc:515] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.811271  3783 raft_consensus.cc:2393] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2462c58306174ec1a384ea49fddf64c1 in current term 1: Already voted for candidate cb3f52aae763473cb09593496219dea8 in this term.
I20260709 13:12:48.815387  3822 tablet_bootstrap.cc:492] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f: No bootstrap required, opened a new log
I20260709 13:12:48.816075  3822 ts_tablet_manager.cc:1403] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f: Time spent bootstrapping tablet: real 0.073s	user 0.016s	sys 0.000s
I20260709 13:12:48.812757  3631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ccc13de4df944b1a6695b0f05c83ada" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:48.820722  3861 raft_consensus.cc:697] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Becoming Leader. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:48.822072  3707 raft_consensus.cc:2468] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 1.
I20260709 13:12:48.822042  3861 consensus_queue.cc:237] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.826356  3859 leader_election.cc:290] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:48.827423  3707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ccc13de4df944b1a6695b0f05c83ada" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:48.828195  3707 raft_consensus.cc:3060] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.831445  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20eb2e33130740e38d072182154c715f" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
I20260709 13:12:48.832078  3708 raft_consensus.cc:2468] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 0.
W20260709 13:12:48.833966  3597 leader_election.cc:343] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:48.803771  3829 consensus_queue.cc:237] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.835448  3597 leader_election.cc:304] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, 31750ab24a654ede8e5d60d6ab45f76f; no voters: 
I20260709 13:12:48.836740  3847 raft_consensus.cc:2804] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:48.837759  3847 raft_consensus.cc:493] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:48.838124  3847 raft_consensus.cc:3060] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.825196  3631 raft_consensus.cc:3060] T 1ccc13de4df944b1a6695b0f05c83ada P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.836632  3823 ts_tablet_manager.cc:1434] T 17679967d4734e678c707f380844b93c P 2462c58306174ec1a384ea49fddf64c1: Time spent starting tablet: real 0.043s	user 0.003s	sys 0.004s
I20260709 13:12:48.836130  3783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20eb2e33130740e38d072182154c715f" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:48.840092  3823 tablet_bootstrap.cc:492] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1: Bootstrap starting.
I20260709 13:12:48.844049  3822 raft_consensus.cc:359] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.844828  3822 raft_consensus.cc:385] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.845109  3822 raft_consensus.cc:740] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.845889  3822 consensus_queue.cc:260] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.847682  3822 ts_tablet_manager.cc:1434] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f: Time spent starting tablet: real 0.031s	user 0.004s	sys 0.003s
I20260709 13:12:48.850859  3597 leader_election.cc:304] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 2462c58306174ec1a384ea49fddf64c1
I20260709 13:12:48.854003  3630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ccc13de4df944b1a6695b0f05c83ada" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f" is_pre_election: true
I20260709 13:12:48.855772  3675 leader_election.cc:304] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1; no voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8
I20260709 13:12:48.855872  3707 raft_consensus.cc:2468] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 1.
I20260709 13:12:48.856675  3847 raft_consensus.cc:515] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.856966  3831 raft_consensus.cc:2749] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:12:48.857355  3746 leader_election.cc:304] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:48.860814  3847 leader_election.cc:290] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 election: Requested vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:48.861560  3861 raft_consensus.cc:2804] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:48.862401  3861 raft_consensus.cc:697] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Becoming Leader. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:48.862244  3528 catalog_manager.cc:5697] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f reported cstate change: term changed from 0 to 1, leader changed from <none> to 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65). New cstate: current_term: 1 leader_uuid: "31750ab24a654ede8e5d60d6ab45f76f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:48.863462  3631 raft_consensus.cc:2468] T 1ccc13de4df944b1a6695b0f05c83ada P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 1.
I20260709 13:12:48.863437  3861 consensus_queue.cc:237] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
W20260709 13:12:48.852113  3597 leader_election.cc:343] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:48.864431  3823 tablet_bootstrap.cc:654] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:48.866281  3597 leader_election.cc:304] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f; no voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8
I20260709 13:12:48.868582  3707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20eb2e33130740e38d072182154c715f" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:48.869453  3824 tablet_bootstrap.cc:492] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8: No bootstrap required, opened a new log
I20260709 13:12:48.858922  3849 raft_consensus.cc:2804] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:48.870319  3824 ts_tablet_manager.cc:1403] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8: Time spent bootstrapping tablet: real 0.177s	user 0.012s	sys 0.011s
I20260709 13:12:48.868892  3782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20eb2e33130740e38d072182154c715f" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:48.870160  3707 raft_consensus.cc:3060] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.869072  3847 raft_consensus.cc:2749] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:12:48.873826  3824 raft_consensus.cc:359] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.874497  3824 raft_consensus.cc:385] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.874800  3824 raft_consensus.cc:740] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.870661  3849 raft_consensus.cc:493] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:48.875377  3849 raft_consensus.cc:3060] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.875761  3824 consensus_queue.cc:260] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.876658  3782 raft_consensus.cc:3060] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:48.879770  3707 raft_consensus.cc:2468] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 1.
I20260709 13:12:48.881152  3597 leader_election.cc:304] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, 31750ab24a654ede8e5d60d6ab45f76f; no voters: 
I20260709 13:12:48.882329  3847 raft_consensus.cc:2804] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:48.882987  3847 raft_consensus.cc:697] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Becoming Leader. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:48.884038  3847 consensus_queue.cc:237] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.886611  3782 raft_consensus.cc:2468] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 1.
I20260709 13:12:48.894626  3824 ts_tablet_manager.cc:1434] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.000s
I20260709 13:12:48.899375  3823 tablet_bootstrap.cc:492] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1: No bootstrap required, opened a new log
I20260709 13:12:48.899837  3823 ts_tablet_manager.cc:1403] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1: Time spent bootstrapping tablet: real 0.060s	user 0.008s	sys 0.002s
I20260709 13:12:48.896760  3849 raft_consensus.cc:515] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.906527  3707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e37f28b3b48449d3adecff96febd5598" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:48.907078  3782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e37f28b3b48449d3adecff96febd5598" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:48.907717  3782 raft_consensus.cc:3060] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:12:48.908272  3597 leader_election.cc:343] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:48.909261  3528 catalog_manager.cc:5697] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb3f52aae763473cb09593496219dea8 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "cb3f52aae763473cb09593496219dea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:48.911041  3528 catalog_manager.cc:5697] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb3f52aae763473cb09593496219dea8 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "cb3f52aae763473cb09593496219dea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:48.915234  3782 raft_consensus.cc:2468] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 1.
I20260709 13:12:48.916486  3849 leader_election.cc:290] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 election: Requested vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:48.922757  3823 raft_consensus.cc:359] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.923327  3823 raft_consensus.cc:385] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:48.923547  3823 raft_consensus.cc:740] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:48.924198  3823 consensus_queue.cc:260] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.916499  3597 leader_election.cc:304] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 2462c58306174ec1a384ea49fddf64c1
I20260709 13:12:48.925508  3849 raft_consensus.cc:2804] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:48.925999  3849 raft_consensus.cc:697] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Becoming Leader. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:48.926669  3823 ts_tablet_manager.cc:1434] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.000s
I20260709 13:12:48.926661  3849 consensus_queue.cc:237] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:48.927177  3529 catalog_manager.cc:5697] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f reported cstate change: term changed from 0 to 1, leader changed from <none> to 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65). New cstate: current_term: 1 leader_uuid: "31750ab24a654ede8e5d60d6ab45f76f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:48.929265  3529 catalog_manager.cc:5697] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f reported cstate change: term changed from 0 to 1, leader changed from <none> to 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65). New cstate: current_term: 1 leader_uuid: "31750ab24a654ede8e5d60d6ab45f76f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:48.939294  3530 catalog_manager.cc:5697] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb3f52aae763473cb09593496219dea8 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "cb3f52aae763473cb09593496219dea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:48.945658  3528 catalog_manager.cc:5697] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f reported cstate change: term changed from 0 to 1, leader changed from <none> to 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65). New cstate: current_term: 1 leader_uuid: "31750ab24a654ede8e5d60d6ab45f76f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:48.953658  3849 consensus_queue.cc:1048] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:48.959538  3849 consensus_queue.cc:1048] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:49.062094  3828 raft_consensus.cc:493] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:49.062574  3828 raft_consensus.cc:515] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.064224  3828 leader_election.cc:290] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:49.064987  3707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17679967d4734e678c707f380844b93c" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
I20260709 13:12:49.065227  3782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17679967d4734e678c707f380844b93c" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:49.065641  3707 raft_consensus.cc:2468] T 17679967d4734e678c707f380844b93c P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 0.
I20260709 13:12:49.065819  3782 raft_consensus.cc:2468] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 0.
I20260709 13:12:49.066605  3597 leader_election.cc:304] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, 31750ab24a654ede8e5d60d6ab45f76f; no voters: 
I20260709 13:12:49.067271  3828 raft_consensus.cc:2804] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:49.067632  3828 raft_consensus.cc:493] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:49.067920  3828 raft_consensus.cc:3060] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:49.073068  3828 raft_consensus.cc:515] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.074712  3828 leader_election.cc:290] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 election: Requested vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:49.075397  3707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17679967d4734e678c707f380844b93c" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:49.075676  3782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17679967d4734e678c707f380844b93c" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:49.076072  3707 raft_consensus.cc:3060] T 17679967d4734e678c707f380844b93c P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:49.076267  3782 raft_consensus.cc:3060] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:49.082868  3782 raft_consensus.cc:2468] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 1.
I20260709 13:12:49.082868  3707 raft_consensus.cc:2468] T 17679967d4734e678c707f380844b93c P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 1.
I20260709 13:12:49.084043  3597 leader_election.cc:304] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, 31750ab24a654ede8e5d60d6ab45f76f; no voters: 
I20260709 13:12:49.084662  3828 raft_consensus.cc:2804] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:49.085111  3828 raft_consensus.cc:697] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Becoming Leader. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:49.085916  3828 consensus_queue.cc:237] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.093021  3528 catalog_manager.cc:5697] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f reported cstate change: term changed from 0 to 1, leader changed from <none> to 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65). New cstate: current_term: 1 leader_uuid: "31750ab24a654ede8e5d60d6ab45f76f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:49.247709  3782 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ab14a07bccb94e4c82b34714c94ccb48"
dest_uuid: "cb3f52aae763473cb09593496219dea8"
mode: GRACEFUL
new_leader_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
 from {username='slave'} at 127.0.0.1:47852
I20260709 13:12:49.248230  3782 raft_consensus.cc:606] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Received request to transfer leadership to TS 31750ab24a654ede8e5d60d6ab45f76f
I20260709 13:12:49.255777  3831 raft_consensus.cc:493] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:49.256011  3782 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "98f4adc13b1f41e0a6df7506b3f2b9a5"
dest_uuid: "cb3f52aae763473cb09593496219dea8"
mode: GRACEFUL
new_leader_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
 from {username='slave'} at 127.0.0.1:47852
I20260709 13:12:49.256310  3831 raft_consensus.cc:515] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.256632  3782 raft_consensus.cc:606] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Received request to transfer leadership to TS 31750ab24a654ede8e5d60d6ab45f76f
I20260709 13:12:49.258201  3831 leader_election.cc:290] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:49.258970  3631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e37f28b3b48449d3adecff96febd5598" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f" is_pre_election: true
I20260709 13:12:49.259549  3782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e37f28b3b48449d3adecff96febd5598" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:49.260335  3782 raft_consensus.cc:2393] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2462c58306174ec1a384ea49fddf64c1 in current term 1: Already voted for candidate 31750ab24a654ede8e5d60d6ab45f76f in this term.
I20260709 13:12:49.261425  3672 leader_election.cc:304] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1; no voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8
I20260709 13:12:49.262143  3831 raft_consensus.cc:3060] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:49.268522  3831 raft_consensus.cc:2749] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:12:49.272635  3631 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e37f28b3b48449d3adecff96febd5598"
dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
mode: GRACEFUL
new_leader_uuid: "2462c58306174ec1a384ea49fddf64c1"
 from {username='slave'} at 127.0.0.1:53662
I20260709 13:12:49.273259  3631 raft_consensus.cc:606] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Received request to transfer leadership to TS 2462c58306174ec1a384ea49fddf64c1
I20260709 13:12:49.275931  3828 consensus_queue.cc:1048] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:49.278859  3782 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1ccc13de4df944b1a6695b0f05c83ada"
dest_uuid: "cb3f52aae763473cb09593496219dea8"
mode: GRACEFUL
new_leader_uuid: "2462c58306174ec1a384ea49fddf64c1"
 from {username='slave'} at 127.0.0.1:47852
I20260709 13:12:49.279412  3782 raft_consensus.cc:606] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Received request to transfer leadership to TS 2462c58306174ec1a384ea49fddf64c1
I20260709 13:12:49.280349  3828 consensus_queue.cc:1048] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:49.290949  3631 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "20eb2e33130740e38d072182154c715f"
dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
mode: GRACEFUL
new_leader_uuid: "2462c58306174ec1a384ea49fddf64c1"
 from {username='slave'} at 127.0.0.1:53662
I20260709 13:12:49.291575  3631 raft_consensus.cc:606] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Received request to transfer leadership to TS 2462c58306174ec1a384ea49fddf64c1
I20260709 13:12:49.293537  3847 consensus_queue.cc:1048] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:49.298516  3861 consensus_queue.cc:1048] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:49.302966  3630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "536c6c3f241342ec804d9400cbb23da9"
dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
mode: GRACEFUL
new_leader_uuid: "2462c58306174ec1a384ea49fddf64c1"
 from {username='slave'} at 127.0.0.1:53662
I20260709 13:12:49.303766  3630 raft_consensus.cc:606] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Received request to transfer leadership to TS 2462c58306174ec1a384ea49fddf64c1
I20260709 13:12:49.307494  3847 consensus_queue.cc:1048] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:12:49.310804  3829 raft_consensus.cc:993] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8: : Instructing follower 31750ab24a654ede8e5d60d6ab45f76f to start an election
I20260709 13:12:49.311321  3829 raft_consensus.cc:1081] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Signalling peer 31750ab24a654ede8e5d60d6ab45f76f to start an election
I20260709 13:12:49.314086  3859 consensus_queue.cc:1048] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:12:49.315579  3631 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ab14a07bccb94e4c82b34714c94ccb48"
dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
 from {username='slave'} at 127.0.0.1:53642
I20260709 13:12:49.316172  3631 raft_consensus.cc:493] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:49.316534  3631 raft_consensus.cc:3060] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.324539  3630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "17679967d4734e678c707f380844b93c"
dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
mode: GRACEFUL
new_leader_uuid: "cb3f52aae763473cb09593496219dea8"
 from {username='slave'} at 127.0.0.1:53662
I20260709 13:12:49.325381  3630 raft_consensus.cc:606] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Received request to transfer leadership to TS cb3f52aae763473cb09593496219dea8
I20260709 13:12:49.328665  3631 raft_consensus.cc:515] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.331905  3631 leader_election.cc:290] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 2 election: Requested vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:49.335013  3706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab14a07bccb94e4c82b34714c94ccb48" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:49.340274  3859 consensus_queue.cc:1048] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:49.343293  3782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab14a07bccb94e4c82b34714c94ccb48" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:49.343990  3782 raft_consensus.cc:3055] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:49.344331  3782 raft_consensus.cc:740] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:49.344638  3828 raft_consensus.cc:993] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f: : Instructing follower 2462c58306174ec1a384ea49fddf64c1 to start an election
I20260709 13:12:49.345145  3828 raft_consensus.cc:1081] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Signalling peer 2462c58306174ec1a384ea49fddf64c1 to start an election
I20260709 13:12:49.345170  3782 consensus_queue.cc:260] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.346638  3782 raft_consensus.cc:3060] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.347220  3707 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "536c6c3f241342ec804d9400cbb23da9"
dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
 from {username='slave'} at 127.0.0.1:47952
I20260709 13:12:49.347739  3707 raft_consensus.cc:493] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:49.348052  3707 raft_consensus.cc:3060] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.355862  3782 raft_consensus.cc:2468] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 2.
I20260709 13:12:49.357861  3597 leader_election.cc:304] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 2462c58306174ec1a384ea49fddf64c1
I20260709 13:12:49.355708  3707 raft_consensus.cc:515] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.359772  3828 consensus_queue.cc:1048] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:12:49.361708  3707 leader_election.cc:290] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 2 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:49.364727  3631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "536c6c3f241342ec804d9400cbb23da9" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:49.365312  3631 raft_consensus.cc:3055] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:49.365625  3631 raft_consensus.cc:740] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:49.366313  3631 consensus_queue.cc:260] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.367435  3631 raft_consensus.cc:3060] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.369884  3859 consensus_queue.cc:1048] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:12:49.358676  3847 raft_consensus.cc:2804] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:49.373232  3847 raft_consensus.cc:697] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 LEADER]: Becoming Leader. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:49.373747  3631 raft_consensus.cc:2468] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 2.
I20260709 13:12:49.374058  3847 consensus_queue.cc:237] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.376549  3782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "536c6c3f241342ec804d9400cbb23da9" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:49.377194  3782 raft_consensus.cc:3060] T 536c6c3f241342ec804d9400cbb23da9 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.377731  3859 consensus_queue.cc:1048] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:49.378042  3675 leader_election.cc:304] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, 31750ab24a654ede8e5d60d6ab45f76f; no voters: 
I20260709 13:12:49.384083  3782 raft_consensus.cc:2468] T 536c6c3f241342ec804d9400cbb23da9 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 2.
I20260709 13:12:49.385953  3831 raft_consensus.cc:2804] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:49.388072  3849 raft_consensus.cc:993] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f: : Instructing follower 2462c58306174ec1a384ea49fddf64c1 to start an election
I20260709 13:12:49.388559  3849 raft_consensus.cc:1081] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Signalling peer 2462c58306174ec1a384ea49fddf64c1 to start an election
I20260709 13:12:49.394194  3530 catalog_manager.cc:5697] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f reported cstate change: term changed from 1 to 2, leader changed from cb3f52aae763473cb09593496219dea8 (127.23.140.67) to 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65). New cstate: current_term: 2 leader_uuid: "31750ab24a654ede8e5d60d6ab45f76f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:49.396554  3847 consensus_queue.cc:1048] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:49.397323  3708 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e37f28b3b48449d3adecff96febd5598"
dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
 from {username='slave'} at 127.0.0.1:47952
I20260709 13:12:49.397912  3708 raft_consensus.cc:493] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:49.398301  3708 raft_consensus.cc:3060] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.407660  3831 raft_consensus.cc:697] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Becoming Leader. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Running, Role: LEADER
I20260709 13:12:49.408509  3859 raft_consensus.cc:993] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8: : Instructing follower 31750ab24a654ede8e5d60d6ab45f76f to start an election
I20260709 13:12:49.409101  3859 raft_consensus.cc:1081] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Signalling peer 31750ab24a654ede8e5d60d6ab45f76f to start an election
I20260709 13:12:49.408607  3831 consensus_queue.cc:237] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.414110  3631 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "98f4adc13b1f41e0a6df7506b3f2b9a5"
dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
 from {username='slave'} at 127.0.0.1:53642
I20260709 13:12:49.414667  3631 raft_consensus.cc:493] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:49.415179  3631 raft_consensus.cc:3060] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.416224  3708 raft_consensus.cc:515] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.419348  3859 consensus_queue.cc:1048] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:49.421200  3708 leader_election.cc:290] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 2 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:49.424908  3632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e37f28b3b48449d3adecff96febd5598" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:49.425618  3632 raft_consensus.cc:3055] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:49.425979  3632 raft_consensus.cc:740] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:49.427240  3632 consensus_queue.cc:260] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.427874  3869 raft_consensus.cc:993] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8: : Instructing follower 2462c58306174ec1a384ea49fddf64c1 to start an election
I20260709 13:12:49.428562  3869 raft_consensus.cc:1081] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Signalling peer 2462c58306174ec1a384ea49fddf64c1 to start an election
I20260709 13:12:49.429024  3632 raft_consensus.cc:3060] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.428638  3529 catalog_manager.cc:5697] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 reported cstate change: term changed from 1 to 2, leader changed from 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65) to 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2462c58306174ec1a384ea49fddf64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:49.432058  3709 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1ccc13de4df944b1a6695b0f05c83ada"
dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
 from {username='slave'} at 127.0.0.1:47968
I20260709 13:12:49.432717  3709 raft_consensus.cc:493] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:49.433073  3709 raft_consensus.cc:3060] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.435339  3631 raft_consensus.cc:515] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.437170  3632 raft_consensus.cc:2468] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 2.
I20260709 13:12:49.440243  3631 leader_election.cc:290] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 2 election: Requested vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:49.440459  3709 raft_consensus.cc:515] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.441862  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f4adc13b1f41e0a6df7506b3f2b9a5" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:49.442538  3708 raft_consensus.cc:3060] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.444109  3781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ccc13de4df944b1a6695b0f05c83ada" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:49.444777  3781 raft_consensus.cc:3055] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:49.445115  3781 raft_consensus.cc:740] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:49.445972  3781 consensus_queue.cc:260] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.447206  3781 raft_consensus.cc:3060] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.447916  3675 leader_election.cc:304] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, 31750ab24a654ede8e5d60d6ab45f76f; no voters: 
I20260709 13:12:49.449214  3898 raft_consensus.cc:2804] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:49.449718  3898 raft_consensus.cc:697] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Becoming Leader. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Running, Role: LEADER
I20260709 13:12:49.450201  3708 raft_consensus.cc:2468] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 2.
I20260709 13:12:49.450628  3898 consensus_queue.cc:237] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.451129  3782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e37f28b3b48449d3adecff96febd5598" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:49.451447  3597 leader_election.cc:304] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, 31750ab24a654ede8e5d60d6ab45f76f; no voters: 
I20260709 13:12:49.451936  3782 raft_consensus.cc:3060] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.452356  3862 raft_consensus.cc:2804] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:49.452872  3862 raft_consensus.cc:697] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 LEADER]: Becoming Leader. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:49.441862  3783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f4adc13b1f41e0a6df7506b3f2b9a5" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:49.453701  3783 raft_consensus.cc:3055] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:49.454121  3783 raft_consensus.cc:740] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:49.453738  3862 consensus_queue.cc:237] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.454922  3783 consensus_queue.cc:260] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.456223  3783 raft_consensus.cc:3060] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.457628  3709 leader_election.cc:290] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 2 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:49.459880  3782 raft_consensus.cc:2468] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 2.
I20260709 13:12:49.461162  3781 raft_consensus.cc:2468] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 2.
I20260709 13:12:49.462304  3672 leader_election.cc:304] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:49.463137  3898 raft_consensus.cc:2804] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:49.463596  3898 raft_consensus.cc:697] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Becoming Leader. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Running, Role: LEADER
I20260709 13:12:49.464447  3783 raft_consensus.cc:2468] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 2.
I20260709 13:12:49.464330  3898 consensus_queue.cc:237] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.467305  3632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ccc13de4df944b1a6695b0f05c83ada" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:49.467005  3528 catalog_manager.cc:5697] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 reported cstate change: term changed from 1 to 2, leader changed from 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65) to 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2462c58306174ec1a384ea49fddf64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:49.467906  3632 raft_consensus.cc:3060] T 1ccc13de4df944b1a6695b0f05c83ada P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.472960  3632 raft_consensus.cc:2468] T 1ccc13de4df944b1a6695b0f05c83ada P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 2.
I20260709 13:12:49.480499  3530 catalog_manager.cc:5697] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 reported cstate change: term changed from 1 to 2, leader changed from cb3f52aae763473cb09593496219dea8 (127.23.140.67) to 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2462c58306174ec1a384ea49fddf64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:49.482152  3528 catalog_manager.cc:5697] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f reported cstate change: term changed from 1 to 2, leader changed from cb3f52aae763473cb09593496219dea8 (127.23.140.67) to 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65). New cstate: current_term: 2 leader_uuid: "31750ab24a654ede8e5d60d6ab45f76f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:49.490931  3847 consensus_queue.cc:1048] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:49.500779  3862 consensus_queue.cc:1048] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:49.508203  3847 raft_consensus.cc:993] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f: : Instructing follower 2462c58306174ec1a384ea49fddf64c1 to start an election
I20260709 13:12:49.508625  3862 raft_consensus.cc:1081] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Signalling peer 2462c58306174ec1a384ea49fddf64c1 to start an election
I20260709 13:12:49.510149  3708 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "20eb2e33130740e38d072182154c715f"
dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
 from {username='slave'} at 127.0.0.1:47952
I20260709 13:12:49.510667  3708 raft_consensus.cc:493] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:49.510916  3708 raft_consensus.cc:3060] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.515657  3708 raft_consensus.cc:515] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.517926  3708 leader_election.cc:290] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 2 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:49.518085  3632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20eb2e33130740e38d072182154c715f" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:49.518729  3632 raft_consensus.cc:3055] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:49.518767  3783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20eb2e33130740e38d072182154c715f" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:49.519135  3632 raft_consensus.cc:740] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:49.519380  3783 raft_consensus.cc:3060] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.519979  3632 consensus_queue.cc:260] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.525926  3632 raft_consensus.cc:3060] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.527246  3783 raft_consensus.cc:2468] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 2.
I20260709 13:12:49.528517  3672 leader_election.cc:304] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:49.529222  3898 raft_consensus.cc:2804] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:49.529711  3898 raft_consensus.cc:697] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Becoming Leader. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Running, Role: LEADER
I20260709 13:12:49.530529  3898 consensus_queue.cc:237] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.532737  3632 raft_consensus.cc:2468] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 2.
I20260709 13:12:49.537456  3528 catalog_manager.cc:5697] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 reported cstate change: term changed from 1 to 2, leader changed from 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65) to 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2462c58306174ec1a384ea49fddf64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:49.592186  3862 consensus_queue.cc:1048] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:49.602761  3849 consensus_queue.cc:1048] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:49.611069  3847 raft_consensus.cc:993] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f: : Instructing follower cb3f52aae763473cb09593496219dea8 to start an election
I20260709 13:12:49.611528  3847 raft_consensus.cc:1081] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Signalling peer cb3f52aae763473cb09593496219dea8 to start an election
I20260709 13:12:49.612886  3783 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "17679967d4734e678c707f380844b93c"
dest_uuid: "cb3f52aae763473cb09593496219dea8"
 from {username='slave'} at 127.0.0.1:47836
I20260709 13:12:49.613360  3783 raft_consensus.cc:493] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:49.613646  3783 raft_consensus.cc:3060] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.618582  3783 raft_consensus.cc:515] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.620434  3783 leader_election.cc:290] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 2 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:49.621033  3632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17679967d4734e678c707f380844b93c" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:49.621706  3632 raft_consensus.cc:3055] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:49.621944  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17679967d4734e678c707f380844b93c" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:49.622030  3632 raft_consensus.cc:740] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:49.622658  3708 raft_consensus.cc:3060] T 17679967d4734e678c707f380844b93c P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.623101  3632 consensus_queue.cc:260] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.624408  3632 raft_consensus.cc:3060] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.629587  3708 raft_consensus.cc:2468] T 17679967d4734e678c707f380844b93c P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 2.
I20260709 13:12:49.630824  3746 leader_election.cc:304] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:49.631357  3632 raft_consensus.cc:2468] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 2.
I20260709 13:12:49.631625  3859 raft_consensus.cc:2804] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:49.632153  3859 raft_consensus.cc:697] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [term 2 LEADER]: Becoming Leader. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:49.632938  3859 consensus_queue.cc:237] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:49.638648  3530 catalog_manager.cc:5697] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 reported cstate change: term changed from 1 to 2, leader changed from 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65) to cb3f52aae763473cb09593496219dea8 (127.23.140.67). New cstate: current_term: 2 leader_uuid: "cb3f52aae763473cb09593496219dea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:49.850348  3708 raft_consensus.cc:1275] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Refusing update from remote peer 31750ab24a654ede8e5d60d6ab45f76f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:49.851714  3847 consensus_queue.cc:1048] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:49.860539  3781 raft_consensus.cc:1275] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Refusing update from remote peer 2462c58306174ec1a384ea49fddf64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:49.861728  3783 raft_consensus.cc:1275] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Refusing update from remote peer 31750ab24a654ede8e5d60d6ab45f76f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:49.862201  3898 consensus_queue.cc:1048] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:49.864362  3849 consensus_queue.cc:1048] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:49.871528  3632 raft_consensus.cc:1275] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Refusing update from remote peer 2462c58306174ec1a384ea49fddf64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:49.873746  3831 consensus_queue.cc:1048] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:49.881908  3632 raft_consensus.cc:1275] T 1ccc13de4df944b1a6695b0f05c83ada P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Refusing update from remote peer 2462c58306174ec1a384ea49fddf64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:49.883082  3831 consensus_queue.cc:1048] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:49.890595  3783 raft_consensus.cc:1275] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Refusing update from remote peer 2462c58306174ec1a384ea49fddf64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:49.892138  3898 consensus_queue.cc:1048] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:49.894549  3783 raft_consensus.cc:1275] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Refusing update from remote peer 31750ab24a654ede8e5d60d6ab45f76f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:49.896015  3849 consensus_queue.cc:1048] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:49.904352  3708 raft_consensus.cc:3060] T ab14a07bccb94e4c82b34714c94ccb48 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:49.914338  3708 raft_consensus.cc:1275] T ab14a07bccb94e4c82b34714c94ccb48 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Refusing update from remote peer 31750ab24a654ede8e5d60d6ab45f76f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:49.915671  3906 consensus_queue.cc:1048] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:49.975133  3632 raft_consensus.cc:1275] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Refusing update from remote peer 2462c58306174ec1a384ea49fddf64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:49.976292  3898 consensus_queue.cc:1048] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:49.983824  3783 raft_consensus.cc:1275] T 536c6c3f241342ec804d9400cbb23da9 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Refusing update from remote peer 2462c58306174ec1a384ea49fddf64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:49.985823  3899 consensus_queue.cc:1048] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:50.036029  3632 raft_consensus.cc:1275] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Refusing update from remote peer 2462c58306174ec1a384ea49fddf64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:50.037240  3899 consensus_queue.cc:1048] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:50.044305  3783 raft_consensus.cc:1275] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Refusing update from remote peer 2462c58306174ec1a384ea49fddf64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:50.045801  3831 consensus_queue.cc:1048] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:50.158562  3632 raft_consensus.cc:1275] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Refusing update from remote peer cb3f52aae763473cb09593496219dea8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:50.159826  3829 consensus_queue.cc:1048] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:50.167303  3708 raft_consensus.cc:1275] T 17679967d4734e678c707f380844b93c P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Refusing update from remote peer cb3f52aae763473cb09593496219dea8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:50.169066  3907 consensus_queue.cc:1048] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:52.351076  3529 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49398:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:12:52.354070  3529 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:12:52.402815  3616 tablet_service.cc:1511] Processing CreateTablet for tablet 57720afe068341ab8da51d15b129d2d6 (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:12:52.404129  3616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57720afe068341ab8da51d15b129d2d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.403982  3695 tablet_service.cc:1511] Processing CreateTablet for tablet 57720afe068341ab8da51d15b129d2d6 (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:12:52.405303  3695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57720afe068341ab8da51d15b129d2d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.405118  3767 tablet_service.cc:1511] Processing CreateTablet for tablet 57720afe068341ab8da51d15b129d2d6 (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:12:52.406340  3767 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57720afe068341ab8da51d15b129d2d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.411741  3766 tablet_service.cc:1511] Processing CreateTablet for tablet 605b25b7889545a4b965c755cbead46a (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:12:52.412038  3768 tablet_service.cc:1511] Processing CreateTablet for tablet dbcad4026f8440949bb13e46b88fc340 (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:12:52.412454  3691 tablet_service.cc:1511] Processing CreateTablet for tablet 605b25b7889545a4b965c755cbead46a (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:12:52.416862  3691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 605b25b7889545a4b965c755cbead46a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.416766  3621 tablet_service.cc:1511] Processing CreateTablet for tablet 605b25b7889545a4b965c755cbead46a (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:12:52.417178  3617 tablet_service.cc:1511] Processing CreateTablet for tablet dbcad4026f8440949bb13e46b88fc340 (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:12:52.418111  3621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 605b25b7889545a4b965c755cbead46a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.418694  3617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbcad4026f8440949bb13e46b88fc340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.413053  3766 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 605b25b7889545a4b965c755cbead46a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.421444  3768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbcad4026f8440949bb13e46b88fc340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.414408  3693 tablet_service.cc:1511] Processing CreateTablet for tablet dbcad4026f8440949bb13e46b88fc340 (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:12:52.424324  3693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbcad4026f8440949bb13e46b88fc340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.432618  3620 tablet_service.cc:1511] Processing CreateTablet for tablet 2fe236a52db543c18530f238587d8346 (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:12:52.434240  3620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fe236a52db543c18530f238587d8346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.444244  3692 tablet_service.cc:1511] Processing CreateTablet for tablet 2fe236a52db543c18530f238587d8346 (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:12:52.445643  3692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fe236a52db543c18530f238587d8346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.445515  3770 tablet_service.cc:1511] Processing CreateTablet for tablet 2fe236a52db543c18530f238587d8346 (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:12:52.446798  3770 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fe236a52db543c18530f238587d8346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.455636  3697 tablet_service.cc:1511] Processing CreateTablet for tablet 49efa8eb906145db89256ddc020530a3 (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:12:52.457054  3697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49efa8eb906145db89256ddc020530a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.458011  3622 tablet_service.cc:1511] Processing CreateTablet for tablet d34d16630b304796b7cb8c4de4f0c3a8 (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:12:52.459247  3622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d34d16630b304796b7cb8c4de4f0c3a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.463111  3765 tablet_service.cc:1511] Processing CreateTablet for tablet 49efa8eb906145db89256ddc020530a3 (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:12:52.464398  3765 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49efa8eb906145db89256ddc020530a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.471683  3621 tablet_service.cc:1511] Processing CreateTablet for tablet 49efa8eb906145db89256ddc020530a3 (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:12:52.479135  3699 tablet_service.cc:1511] Processing CreateTablet for tablet d34d16630b304796b7cb8c4de4f0c3a8 (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:12:52.480482  3699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d34d16630b304796b7cb8c4de4f0c3a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.481316  3618 tablet_service.cc:1511] Processing CreateTablet for tablet 9fdc51f4bd024752b1b8a75cd782a03e (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:12:52.485723  3772 tablet_service.cc:1511] Processing CreateTablet for tablet d34d16630b304796b7cb8c4de4f0c3a8 (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:12:52.487046  3772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d34d16630b304796b7cb8c4de4f0c3a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.522315  3773 tablet_service.cc:1511] Processing CreateTablet for tablet 9fdc51f4bd024752b1b8a75cd782a03e (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:12:52.523204  3623 tablet_service.cc:1511] Processing CreateTablet for tablet b2c195102cee49d39fe701c20ac7dacf (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:12:52.523648  3773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fdc51f4bd024752b1b8a75cd782a03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.517051  3621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49efa8eb906145db89256ddc020530a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.525761  3623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c195102cee49d39fe701c20ac7dacf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.526358  3618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fdc51f4bd024752b1b8a75cd782a03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.542227  3619 tablet_service.cc:1511] Processing CreateTablet for tablet b0d21b4836594593ac3f578d56f91e4a (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:12:52.543540  3619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d21b4836594593ac3f578d56f91e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.544147  3694 tablet_service.cc:1511] Processing CreateTablet for tablet b0d21b4836594593ac3f578d56f91e4a (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:12:52.545439  3694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d21b4836594593ac3f578d56f91e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.550518  3770 tablet_service.cc:1511] Processing CreateTablet for tablet b0d21b4836594593ac3f578d56f91e4a (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:12:52.551849  3770 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d21b4836594593ac3f578d56f91e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.552874  3693 tablet_service.cc:1511] Processing CreateTablet for tablet b2c195102cee49d39fe701c20ac7dacf (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:12:52.554780  3693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c195102cee49d39fe701c20ac7dacf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.578042  3922 tablet_bootstrap.cc:492] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8: Bootstrap starting.
I20260709 13:12:52.581815  3920 tablet_bootstrap.cc:492] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f: Bootstrap starting.
I20260709 13:12:52.588343  3922 tablet_bootstrap.cc:654] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.592159  3923 tablet_bootstrap.cc:492] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1: Bootstrap starting.
I20260709 13:12:52.595160  3920 tablet_bootstrap.cc:654] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.537101  3699 tablet_service.cc:1511] Processing CreateTablet for tablet 9fdc51f4bd024752b1b8a75cd782a03e (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:12:52.598356  3699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fdc51f4bd024752b1b8a75cd782a03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.599586  3765 tablet_service.cc:1511] Processing CreateTablet for tablet b2c195102cee49d39fe701c20ac7dacf (DEFAULT_TABLE table=soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:12:52.603466  3765 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c195102cee49d39fe701c20ac7dacf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:12:52.614697  3923 tablet_bootstrap.cc:654] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.631569  3923 tablet_bootstrap.cc:492] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1: No bootstrap required, opened a new log
I20260709 13:12:52.632069  3923 ts_tablet_manager.cc:1403] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1: Time spent bootstrapping tablet: real 0.040s	user 0.009s	sys 0.005s
I20260709 13:12:52.635164  3923 raft_consensus.cc:359] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.635874  3923 raft_consensus.cc:385] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.636169  3923 raft_consensus.cc:740] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.637051  3923 consensus_queue.cc:260] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.639912  3923 ts_tablet_manager.cc:1434] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:12:52.641307  3923 tablet_bootstrap.cc:492] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1: Bootstrap starting.
I20260709 13:12:52.647606  3923 tablet_bootstrap.cc:654] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.654567  3920 tablet_bootstrap.cc:492] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f: No bootstrap required, opened a new log
I20260709 13:12:52.655160  3920 ts_tablet_manager.cc:1403] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent bootstrapping tablet: real 0.074s	user 0.031s	sys 0.026s
I20260709 13:12:52.657505  3922 tablet_bootstrap.cc:492] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8: No bootstrap required, opened a new log
I20260709 13:12:52.658071  3922 ts_tablet_manager.cc:1403] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8: Time spent bootstrapping tablet: real 0.080s	user 0.022s	sys 0.018s
I20260709 13:12:52.659531  3920 raft_consensus.cc:359] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.660084  3923 tablet_bootstrap.cc:492] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1: No bootstrap required, opened a new log
I20260709 13:12:52.660341  3920 raft_consensus.cc:385] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.660574  3923 ts_tablet_manager.cc:1403] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1: Time spent bootstrapping tablet: real 0.020s	user 0.003s	sys 0.014s
I20260709 13:12:52.660691  3920 raft_consensus.cc:740] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.661108  3922 raft_consensus.cc:359] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.661854  3922 raft_consensus.cc:385] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.661613  3920 consensus_queue.cc:260] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.662177  3922 raft_consensus.cc:740] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.663127  3922 consensus_queue.cc:260] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.664259  3920 ts_tablet_manager.cc:1434] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.000s
I20260709 13:12:52.665337  3920 tablet_bootstrap.cc:492] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f: Bootstrap starting.
I20260709 13:12:52.665612  3922 ts_tablet_manager.cc:1434] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:12:52.666667  3922 tablet_bootstrap.cc:492] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8: Bootstrap starting.
I20260709 13:12:52.663671  3923 raft_consensus.cc:359] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.670233  3923 raft_consensus.cc:385] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.670586  3923 raft_consensus.cc:740] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.672572  3920 tablet_bootstrap.cc:654] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.674719  3923 consensus_queue.cc:260] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.673727  3922 tablet_bootstrap.cc:654] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.677567  3923 ts_tablet_manager.cc:1434] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1: Time spent starting tablet: real 0.017s	user 0.003s	sys 0.006s
I20260709 13:12:52.678575  3923 tablet_bootstrap.cc:492] T dbcad4026f8440949bb13e46b88fc340 P 2462c58306174ec1a384ea49fddf64c1: Bootstrap starting.
I20260709 13:12:52.685734  3923 tablet_bootstrap.cc:654] T dbcad4026f8440949bb13e46b88fc340 P 2462c58306174ec1a384ea49fddf64c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.688298  3922 tablet_bootstrap.cc:492] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8: No bootstrap required, opened a new log
I20260709 13:12:52.688953  3922 ts_tablet_manager.cc:1403] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8: Time spent bootstrapping tablet: real 0.023s	user 0.016s	sys 0.000s
I20260709 13:12:52.692771  3922 raft_consensus.cc:359] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.694988  3922 raft_consensus.cc:385] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.695284  3922 raft_consensus.cc:740] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.696060  3920 tablet_bootstrap.cc:492] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f: No bootstrap required, opened a new log
I20260709 13:12:52.695988  3922 consensus_queue.cc:260] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.696621  3920 ts_tablet_manager.cc:1403] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f: Time spent bootstrapping tablet: real 0.032s	user 0.017s	sys 0.008s
I20260709 13:12:52.698472  3922 ts_tablet_manager.cc:1434] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:12:52.699658  3908 raft_consensus.cc:493] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:52.699975  3920 raft_consensus.cc:359] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.700169  3908 raft_consensus.cc:515] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.700603  3922 tablet_bootstrap.cc:492] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8: Bootstrap starting.
I20260709 13:12:52.700747  3920 raft_consensus.cc:385] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.701174  3920 raft_consensus.cc:740] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.701490  3923 tablet_bootstrap.cc:492] T dbcad4026f8440949bb13e46b88fc340 P 2462c58306174ec1a384ea49fddf64c1: No bootstrap required, opened a new log
I20260709 13:12:52.702127  3923 ts_tablet_manager.cc:1403] T dbcad4026f8440949bb13e46b88fc340 P 2462c58306174ec1a384ea49fddf64c1: Time spent bootstrapping tablet: real 0.024s	user 0.012s	sys 0.008s
I20260709 13:12:52.701922  3920 consensus_queue.cc:260] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.703406  3632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "605b25b7889545a4b965c755cbead46a" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f" is_pre_election: true
I20260709 13:12:52.704017  3632 raft_consensus.cc:2468] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 0.
I20260709 13:12:52.704429  3920 ts_tablet_manager.cc:1434] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:12:52.705472  3920 tablet_bootstrap.cc:492] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f: Bootstrap starting.
I20260709 13:12:52.705412  3749 leader_election.cc:304] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:52.707796  3922 tablet_bootstrap.cc:654] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.709100  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "605b25b7889545a4b965c755cbead46a" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
W20260709 13:12:52.710722  3746 leader_election.cc:343] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:52.711352  3908 leader_election.cc:290] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:52.711923  3908 raft_consensus.cc:2804] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:52.712321  3908 raft_consensus.cc:493] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:52.712726  3908 raft_consensus.cc:3060] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:52.712648  3923 raft_consensus.cc:359] T dbcad4026f8440949bb13e46b88fc340 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.713438  3920 tablet_bootstrap.cc:654] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.713789  3923 raft_consensus.cc:385] T dbcad4026f8440949bb13e46b88fc340 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.714165  3923 raft_consensus.cc:740] T dbcad4026f8440949bb13e46b88fc340 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.715008  3923 consensus_queue.cc:260] T dbcad4026f8440949bb13e46b88fc340 P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.717561  3923 ts_tablet_manager.cc:1434] T dbcad4026f8440949bb13e46b88fc340 P 2462c58306174ec1a384ea49fddf64c1: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.000s
I20260709 13:12:52.718488  3923 tablet_bootstrap.cc:492] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1: Bootstrap starting.
I20260709 13:12:52.719468  3922 tablet_bootstrap.cc:492] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8: No bootstrap required, opened a new log
I20260709 13:12:52.720096  3922 ts_tablet_manager.cc:1403] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8: Time spent bootstrapping tablet: real 0.020s	user 0.009s	sys 0.008s
I20260709 13:12:52.724785  3923 tablet_bootstrap.cc:654] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.726099  3921 raft_consensus.cc:493] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:52.726657  3921 raft_consensus.cc:515] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.727540  3922 raft_consensus.cc:359] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.728282  3922 raft_consensus.cc:385] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.728344  3920 tablet_bootstrap.cc:492] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f: No bootstrap required, opened a new log
I20260709 13:12:52.728600  3922 raft_consensus.cc:740] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.728874  3920 ts_tablet_manager.cc:1403] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent bootstrapping tablet: real 0.024s	user 0.007s	sys 0.011s
I20260709 13:12:52.729501  3922 consensus_queue.cc:260] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.730804  3781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fe236a52db543c18530f238587d8346" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:52.731667  3781 raft_consensus.cc:2468] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 0.
I20260709 13:12:52.731954  3922 ts_tablet_manager.cc:1434] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:12:52.731956  3920 raft_consensus.cc:359] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.732857  3920 raft_consensus.cc:385] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.733153  3922 tablet_bootstrap.cc:492] T dbcad4026f8440949bb13e46b88fc340 P cb3f52aae763473cb09593496219dea8: Bootstrap starting.
I20260709 13:12:52.733228  3920 raft_consensus.cc:740] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.734243  3920 consensus_queue.cc:260] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
W20260709 13:12:52.735597  3580 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:12:52.736465  3920 ts_tablet_manager.cc:1434] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.002s
I20260709 13:12:52.737680  3920 tablet_bootstrap.cc:492] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f: Bootstrap starting.
I20260709 13:12:52.737443  3632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fe236a52db543c18530f238587d8346" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f" is_pre_election: true
I20260709 13:12:52.740370  3922 tablet_bootstrap.cc:654] T dbcad4026f8440949bb13e46b88fc340 P cb3f52aae763473cb09593496219dea8: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.740237  3672 leader_election.cc:304] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:52.741590  3831 raft_consensus.cc:2804] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:52.742057  3831 raft_consensus.cc:493] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:52.742479  3831 raft_consensus.cc:3060] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:52.745107  3920 tablet_bootstrap.cc:654] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.747212  3906 raft_consensus.cc:493] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:52.747714  3906 raft_consensus.cc:515] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
W20260709 13:12:52.749336  3675 leader_election.cc:343] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:52.749919  3906 leader_election.cc:290] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:52.751772  3781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57720afe068341ab8da51d15b129d2d6" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:52.751912  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57720afe068341ab8da51d15b129d2d6" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
I20260709 13:12:52.752444  3922 tablet_bootstrap.cc:492] T dbcad4026f8440949bb13e46b88fc340 P cb3f52aae763473cb09593496219dea8: No bootstrap required, opened a new log
I20260709 13:12:52.753013  3922 ts_tablet_manager.cc:1403] T dbcad4026f8440949bb13e46b88fc340 P cb3f52aae763473cb09593496219dea8: Time spent bootstrapping tablet: real 0.020s	user 0.010s	sys 0.007s
W20260709 13:12:52.753095  3597 leader_election.cc:343] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:12:52.753903  3597 leader_election.cc:343] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:52.732915  3921 leader_election.cc:290] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:52.754446  3597 leader_election.cc:304] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f; no voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8
I20260709 13:12:52.755581  3906 raft_consensus.cc:2749] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:12:52.756222  3922 raft_consensus.cc:359] T dbcad4026f8440949bb13e46b88fc340 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.757182  3922 raft_consensus.cc:385] T dbcad4026f8440949bb13e46b88fc340 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.757557  3922 raft_consensus.cc:740] T dbcad4026f8440949bb13e46b88fc340 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.758544  3922 consensus_queue.cc:260] T dbcad4026f8440949bb13e46b88fc340 P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.763921  3920 tablet_bootstrap.cc:492] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f: No bootstrap required, opened a new log
I20260709 13:12:52.764482  3920 ts_tablet_manager.cc:1403] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent bootstrapping tablet: real 0.027s	user 0.010s	sys 0.008s
I20260709 13:12:52.765074  3922 ts_tablet_manager.cc:1434] T dbcad4026f8440949bb13e46b88fc340 P cb3f52aae763473cb09593496219dea8: Time spent starting tablet: real 0.012s	user 0.010s	sys 0.000s
I20260709 13:12:52.766180  3922 tablet_bootstrap.cc:492] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8: Bootstrap starting.
I20260709 13:12:52.767479  3920 raft_consensus.cc:359] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.768227  3920 raft_consensus.cc:385] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.771162  3920 raft_consensus.cc:740] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.772012  3920 consensus_queue.cc:260] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.773061  3922 tablet_bootstrap.cc:654] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.774659  3920 ts_tablet_manager.cc:1434] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.004s
I20260709 13:12:52.776961  3920 tablet_bootstrap.cc:492] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f: Bootstrap starting.
I20260709 13:12:52.770443  3831 raft_consensus.cc:515] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.784978  3920 tablet_bootstrap.cc:654] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.787868  3922 tablet_bootstrap.cc:492] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8: No bootstrap required, opened a new log
I20260709 13:12:52.788467  3922 ts_tablet_manager.cc:1403] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8: Time spent bootstrapping tablet: real 0.023s	user 0.014s	sys 0.004s
I20260709 13:12:52.792292  3922 raft_consensus.cc:359] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.793357  3922 raft_consensus.cc:385] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.793738  3922 raft_consensus.cc:740] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.794674  3922 consensus_queue.cc:260] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.797621  3922 ts_tablet_manager.cc:1434] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.007s
W20260709 13:12:52.797402  3749 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:40049, user_credentials={real_user=slave}} blocked reactor thread for 92071.8us
I20260709 13:12:52.800832  3922 tablet_bootstrap.cc:492] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8: Bootstrap starting.
I20260709 13:12:52.802326  3632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fe236a52db543c18530f238587d8346" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
W20260709 13:12:52.806216  3675 leader_election.cc:343] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:52.808996  3781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fe236a52db543c18530f238587d8346" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:52.809775  3781 raft_consensus.cc:3060] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:52.809429  3922 tablet_bootstrap.cc:654] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.812306  3920 tablet_bootstrap.cc:492] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f: No bootstrap required, opened a new log
I20260709 13:12:52.812847  3920 ts_tablet_manager.cc:1403] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f: Time spent bootstrapping tablet: real 0.036s	user 0.017s	sys 0.014s
I20260709 13:12:52.821632  3931 raft_consensus.cc:493] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:52.817025  3920 raft_consensus.cc:359] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.824674  3908 raft_consensus.cc:515] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.823783  3922 tablet_bootstrap.cc:492] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8: No bootstrap required, opened a new log
I20260709 13:12:52.825515  3931 raft_consensus.cc:515] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.826177  3922 ts_tablet_manager.cc:1403] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8: Time spent bootstrapping tablet: real 0.027s	user 0.014s	sys 0.007s
I20260709 13:12:52.828989  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d34d16630b304796b7cb8c4de4f0c3a8" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
I20260709 13:12:52.828934  3908 leader_election.cc:290] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:52.829408  3922 raft_consensus.cc:359] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.829870  3708 raft_consensus.cc:2468] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 0.
I20260709 13:12:52.825481  3920 raft_consensus.cc:385] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.830348  3920 raft_consensus.cc:740] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.831228  3746 leader_election.cc:304] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:52.831120  3920 consensus_queue.cc:260] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.832080  3908 raft_consensus.cc:2804] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:52.832647  3908 raft_consensus.cc:493] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:52.833158  3908 raft_consensus.cc:3060] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:52.833627  3922 raft_consensus.cc:385] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.833979  3922 raft_consensus.cc:740] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.834646  3922 consensus_queue.cc:260] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.836766  3922 ts_tablet_manager.cc:1434] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
I20260709 13:12:52.836979  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "605b25b7889545a4b965c755cbead46a" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:52.837818  3922 tablet_bootstrap.cc:492] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8: Bootstrap starting.
W20260709 13:12:52.838718  3746 leader_election.cc:343] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:52.840417  3631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d34d16630b304796b7cb8c4de4f0c3a8" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f" is_pre_election: true
I20260709 13:12:52.841099  3631 raft_consensus.cc:2468] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 0.
I20260709 13:12:52.833441  3920 ts_tablet_manager.cc:1434] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f: Time spent starting tablet: real 0.020s	user 0.010s	sys 0.000s
I20260709 13:12:52.844089  3632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "605b25b7889545a4b965c755cbead46a" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:52.844456  3920 tablet_bootstrap.cc:492] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f: Bootstrap starting.
I20260709 13:12:52.844823  3632 raft_consensus.cc:3060] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:52.847229  3908 raft_consensus.cc:515] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.850373  3908 leader_election.cc:290] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:52.851323  3920 tablet_bootstrap.cc:654] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.852675  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d34d16630b304796b7cb8c4de4f0c3a8" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:52.853581  3708 raft_consensus.cc:3060] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:52.853749  3631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d34d16630b304796b7cb8c4de4f0c3a8" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:52.854419  3631 raft_consensus.cc:3060] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:52.858844  3922 tablet_bootstrap.cc:654] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.871624  3708 raft_consensus.cc:2468] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 1.
I20260709 13:12:52.878484  3746 leader_election.cc:304] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:52.879384  3908 raft_consensus.cc:2804] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:52.879715  3920 tablet_bootstrap.cc:492] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f: No bootstrap required, opened a new log
I20260709 13:12:52.880205  3632 raft_consensus.cc:2468] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 1.
I20260709 13:12:52.880365  3920 ts_tablet_manager.cc:1403] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent bootstrapping tablet: real 0.036s	user 0.011s	sys 0.005s
I20260709 13:12:52.888548  3749 leader_election.cc:304] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 2462c58306174ec1a384ea49fddf64c1
I20260709 13:12:52.891172  3937 raft_consensus.cc:2804] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:52.880407  3908 raft_consensus.cc:697] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Becoming Leader. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:52.881650  3831 leader_election.cc:290] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:52.892578  3923 tablet_bootstrap.cc:492] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1: No bootstrap required, opened a new log
I20260709 13:12:52.892283  3908 consensus_queue.cc:237] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.893118  3923 ts_tablet_manager.cc:1403] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1: Time spent bootstrapping tablet: real 0.175s	user 0.053s	sys 0.086s
I20260709 13:12:52.893524  3937 raft_consensus.cc:697] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Becoming Leader. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:52.882756  3922 tablet_bootstrap.cc:492] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8: No bootstrap required, opened a new log
I20260709 13:12:52.885754  3942 raft_consensus.cc:493] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:52.894505  3942 raft_consensus.cc:515] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.884269  3631 raft_consensus.cc:2468] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 1.
I20260709 13:12:52.890837  3920 raft_consensus.cc:359] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.887068  3781 raft_consensus.cc:2468] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 1.
I20260709 13:12:52.895490  3920 raft_consensus.cc:385] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.887387  3931 leader_election.cc:290] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:52.894438  3937 consensus_queue.cc:237] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.896169  3923 raft_consensus.cc:359] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.896721  3942 leader_election.cc:290] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:52.896977  3923 raft_consensus.cc:385] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.897400  3923 raft_consensus.cc:740] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.897557  3922 ts_tablet_manager.cc:1403] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8: Time spent bootstrapping tablet: real 0.060s	user 0.019s	sys 0.004s
I20260709 13:12:52.899300  3782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c195102cee49d39fe701c20ac7dacf" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:52.900523  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c195102cee49d39fe701c20ac7dacf" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
I20260709 13:12:52.900835  3922 raft_consensus.cc:359] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.898165  3923 consensus_queue.cc:260] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
W20260709 13:12:52.901683  3597 leader_election.cc:343] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:52.901634  3922 raft_consensus.cc:385] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.903301  3923 ts_tablet_manager.cc:1434] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
W20260709 13:12:52.903306  3597 leader_election.cc:343] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:52.903802  3922 raft_consensus.cc:740] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.903957  3597 leader_election.cc:304] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f; no voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8
I20260709 13:12:52.904652  3923 tablet_bootstrap.cc:492] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1: Bootstrap starting.
I20260709 13:12:52.904723  3942 raft_consensus.cc:2749] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:12:52.895862  3920 raft_consensus.cc:740] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.905514  3922 consensus_queue.cc:260] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.906502  3920 consensus_queue.cc:260] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.908581  3922 ts_tablet_manager.cc:1434] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.001s
I20260709 13:12:52.908928  3920 ts_tablet_manager.cc:1434] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent starting tablet: real 0.021s	user 0.001s	sys 0.005s
I20260709 13:12:52.909601  3922 tablet_bootstrap.cc:492] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8: Bootstrap starting.
I20260709 13:12:52.910094  3920 tablet_bootstrap.cc:492] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f: Bootstrap starting.
I20260709 13:12:52.912226  3923 tablet_bootstrap.cc:654] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.913029  3944 raft_consensus.cc:493] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:52.913553  3944 raft_consensus.cc:515] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.915913  3944 leader_election.cc:290] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:52.917397  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57720afe068341ab8da51d15b129d2d6" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
W20260709 13:12:52.918987  3746 leader_election.cc:343] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:52.915124  3931 raft_consensus.cc:493] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:52.920631  3922 tablet_bootstrap.cc:654] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.920425  3931 raft_consensus.cc:515] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.921558  3920 tablet_bootstrap.cc:654] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.923524  3630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49efa8eb906145db89256ddc020530a3" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f" is_pre_election: true
I20260709 13:12:52.924116  3923 tablet_bootstrap.cc:492] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1: No bootstrap required, opened a new log
I20260709 13:12:52.924723  3923 ts_tablet_manager.cc:1403] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1: Time spent bootstrapping tablet: real 0.020s	user 0.011s	sys 0.007s
W20260709 13:12:52.925566  3749 leader_election.cc:343] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:52.928388  3632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57720afe068341ab8da51d15b129d2d6" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f" is_pre_election: true
I20260709 13:12:52.929239  3672 leader_election.cc:304] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8; no voters: 31750ab24a654ede8e5d60d6ab45f76f
I20260709 13:12:52.932994  3921 raft_consensus.cc:2804] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:52.934453  3921 raft_consensus.cc:697] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Running, Role: LEADER
I20260709 13:12:52.935313  3921 consensus_queue.cc:237] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.939040  3922 tablet_bootstrap.cc:492] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8: No bootstrap required, opened a new log
I20260709 13:12:52.939530  3922 ts_tablet_manager.cc:1403] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8: Time spent bootstrapping tablet: real 0.030s	user 0.003s	sys 0.008s
I20260709 13:12:52.941594  3942 raft_consensus.cc:493] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:52.942103  3942 raft_consensus.cc:515] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.943488  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49efa8eb906145db89256ddc020530a3" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
I20260709 13:12:52.944341  3942 leader_election.cc:290] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:52.942224  3923 raft_consensus.cc:359] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.945063  3923 raft_consensus.cc:385] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.945399  3923 raft_consensus.cc:740] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.944649  3708 raft_consensus.cc:2468] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 0.
I20260709 13:12:52.947063  3922 raft_consensus.cc:359] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.947296  3709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbcad4026f8440949bb13e46b88fc340" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
I20260709 13:12:52.947750  3922 raft_consensus.cc:385] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.947992  3709 raft_consensus.cc:2468] T dbcad4026f8440949bb13e46b88fc340 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 0.
I20260709 13:12:52.946834  3781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbcad4026f8440949bb13e46b88fc340" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:52.948091  3922 raft_consensus.cc:740] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.950335  3781 raft_consensus.cc:2468] T dbcad4026f8440949bb13e46b88fc340 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 0.
I20260709 13:12:52.951251  3922 consensus_queue.cc:260] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.951885  3923 consensus_queue.cc:260] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.953265  3597 leader_election.cc:304] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:52.955062  3942 raft_consensus.cc:2804] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:52.955420  3942 raft_consensus.cc:493] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:52.955686  3922 ts_tablet_manager.cc:1434] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.001s
I20260709 13:12:52.948189  3746 leader_election.cc:304] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8; no voters: 31750ab24a654ede8e5d60d6ab45f76f
I20260709 13:12:52.955744  3942 raft_consensus.cc:3060] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:52.956676  3922 tablet_bootstrap.cc:492] T b2c195102cee49d39fe701c20ac7dacf P cb3f52aae763473cb09593496219dea8: Bootstrap starting.
I20260709 13:12:52.957420  3944 raft_consensus.cc:2804] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:52.957814  3944 raft_consensus.cc:493] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:52.958140  3944 raft_consensus.cc:3060] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:52.963928  3942 raft_consensus.cc:515] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.955739  3923 ts_tablet_manager.cc:1434] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1: Time spent starting tablet: real 0.031s	user 0.002s	sys 0.006s
I20260709 13:12:52.929219  3632 raft_consensus.cc:2468] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 0.
I20260709 13:12:52.970039  3781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbcad4026f8440949bb13e46b88fc340" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:52.970690  3781 raft_consensus.cc:3060] T dbcad4026f8440949bb13e46b88fc340 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:52.970346  3749 leader_election.cc:304] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 2462c58306174ec1a384ea49fddf64c1
I20260709 13:12:52.971299  3922 tablet_bootstrap.cc:654] T b2c195102cee49d39fe701c20ac7dacf P cb3f52aae763473cb09593496219dea8: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:52.971884  3937 raft_consensus.cc:2804] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:52.968072  3709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbcad4026f8440949bb13e46b88fc340" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:52.972510  3709 raft_consensus.cc:3060] T dbcad4026f8440949bb13e46b88fc340 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:52.974403  3920 tablet_bootstrap.cc:492] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f: No bootstrap required, opened a new log
I20260709 13:12:52.974932  3920 ts_tablet_manager.cc:1403] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f: Time spent bootstrapping tablet: real 0.065s	user 0.015s	sys 0.018s
I20260709 13:12:52.977491  3937 raft_consensus.cc:493] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:52.965898  3923 tablet_bootstrap.cc:492] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1: Bootstrap starting.
I20260709 13:12:52.977929  3937 raft_consensus.cc:3060] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:52.978034  3920 raft_consensus.cc:359] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.978645  3781 raft_consensus.cc:2468] T dbcad4026f8440949bb13e46b88fc340 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 1.
I20260709 13:12:52.978900  3920 raft_consensus.cc:385] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.971608  3944 raft_consensus.cc:515] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.980129  3597 leader_election.cc:304] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:52.980813  3944 leader_election.cc:290] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:52.980988  3906 raft_consensus.cc:2804] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:52.980355  3922 tablet_bootstrap.cc:492] T b2c195102cee49d39fe701c20ac7dacf P cb3f52aae763473cb09593496219dea8: No bootstrap required, opened a new log
I20260709 13:12:52.981405  3920 raft_consensus.cc:740] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.981886  3922 ts_tablet_manager.cc:1403] T b2c195102cee49d39fe701c20ac7dacf P cb3f52aae763473cb09593496219dea8: Time spent bootstrapping tablet: real 0.025s	user 0.011s	sys 0.000s
I20260709 13:12:52.982673  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49efa8eb906145db89256ddc020530a3" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:52.982400  3920 consensus_queue.cc:260] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.983268  3708 raft_consensus.cc:3060] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:52.989239  3906 raft_consensus.cc:697] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Becoming Leader. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:52.984706  3922 raft_consensus.cc:359] T b2c195102cee49d39fe701c20ac7dacf P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.990033  3922 raft_consensus.cc:385] T b2c195102cee49d39fe701c20ac7dacf P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:52.990321  3922 raft_consensus.cc:740] T b2c195102cee49d39fe701c20ac7dacf P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Initialized, Role: FOLLOWER
I20260709 13:12:52.990406  3906 consensus_queue.cc:237] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.991093  3922 consensus_queue.cc:260] T b2c195102cee49d39fe701c20ac7dacf P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:52.992236  3920 ts_tablet_manager.cc:1434] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.008s
I20260709 13:12:52.993569  3920 tablet_bootstrap.cc:492] T b0d21b4836594593ac3f578d56f91e4a P 31750ab24a654ede8e5d60d6ab45f76f: Bootstrap starting.
I20260709 13:12:52.993639  3631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49efa8eb906145db89256ddc020530a3" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:52.994532  3942 leader_election.cc:290] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 election: Requested vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
W20260709 13:12:52.995275  3749 leader_election.cc:343] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:52.995834  3922 ts_tablet_manager.cc:1434] T b2c195102cee49d39fe701c20ac7dacf P cb3f52aae763473cb09593496219dea8: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 13:12:52.996855  3931 leader_election.cc:290] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:52.998625  3923 tablet_bootstrap.cc:654] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:53.003757  3920 tablet_bootstrap.cc:654] T b0d21b4836594593ac3f578d56f91e4a P 31750ab24a654ede8e5d60d6ab45f76f: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:53.008419  3906 raft_consensus.cc:493] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:53.008958  3906 raft_consensus.cc:515] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.012264  3707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fdc51f4bd024752b1b8a75cd782a03e" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
W20260709 13:12:53.014107  3597 leader_election.cc:343] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:53.014353  3709 raft_consensus.cc:2468] T dbcad4026f8440949bb13e46b88fc340 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 1.
I20260709 13:12:53.016990  3781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fdc51f4bd024752b1b8a75cd782a03e" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:53.017894  3781 raft_consensus.cc:2468] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 0.
I20260709 13:12:53.019613  3597 leader_election.cc:304] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 2462c58306174ec1a384ea49fddf64c1
I20260709 13:12:53.020748  3942 raft_consensus.cc:2804] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:53.021241  3942 raft_consensus.cc:493] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:53.021665  3942 raft_consensus.cc:3060] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:53.023586  3708 raft_consensus.cc:2468] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 1.
I20260709 13:12:53.025411  3746 leader_election.cc:304] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8; no voters: 31750ab24a654ede8e5d60d6ab45f76f
I20260709 13:12:53.030210  3931 raft_consensus.cc:2804] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:53.031006  3931 raft_consensus.cc:697] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Becoming Leader. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:53.032287  3931 consensus_queue.cc:237] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.037581  3937 raft_consensus.cc:515] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.015688  3906 leader_election.cc:290] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:53.039789  3937 leader_election.cc:290] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:53.033861  3532 catalog_manager.cc:5697] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "2462c58306174ec1a384ea49fddf64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:53.040820  3631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57720afe068341ab8da51d15b129d2d6" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:53.041507  3631 raft_consensus.cc:3060] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:53.042332  3920 tablet_bootstrap.cc:492] T b0d21b4836594593ac3f578d56f91e4a P 31750ab24a654ede8e5d60d6ab45f76f: No bootstrap required, opened a new log
I20260709 13:12:53.042838  3920 ts_tablet_manager.cc:1403] T b0d21b4836594593ac3f578d56f91e4a P 31750ab24a654ede8e5d60d6ab45f76f: Time spent bootstrapping tablet: real 0.049s	user 0.004s	sys 0.007s
I20260709 13:12:53.045233  3942 raft_consensus.cc:515] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.045617  3920 raft_consensus.cc:359] T b0d21b4836594593ac3f578d56f91e4a P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.046473  3920 raft_consensus.cc:385] T b0d21b4836594593ac3f578d56f91e4a P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:53.046744  3920 raft_consensus.cc:740] T b0d21b4836594593ac3f578d56f91e4a P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Initialized, Role: FOLLOWER
I20260709 13:12:53.047394  3920 consensus_queue.cc:260] T b0d21b4836594593ac3f578d56f91e4a P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.045581  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57720afe068341ab8da51d15b129d2d6" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:53.049643  3920 ts_tablet_manager.cc:1434] T b0d21b4836594593ac3f578d56f91e4a P 31750ab24a654ede8e5d60d6ab45f76f: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:12:53.050696  3920 tablet_bootstrap.cc:492] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f: Bootstrap starting.
W20260709 13:12:53.050652  3746 leader_election.cc:343] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:53.053566  3942 leader_election.cc:290] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 election: Requested vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:53.054642  3709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fdc51f4bd024752b1b8a75cd782a03e" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
W20260709 13:12:53.057178  3597 leader_election.cc:343] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:53.058048  3781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fdc51f4bd024752b1b8a75cd782a03e" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:53.058660  3781 raft_consensus.cc:3060] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:53.065357  3923 tablet_bootstrap.cc:492] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1: No bootstrap required, opened a new log
I20260709 13:12:53.065886  3923 ts_tablet_manager.cc:1403] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1: Time spent bootstrapping tablet: real 0.100s	user 0.021s	sys 0.022s
I20260709 13:12:53.067394  3528 catalog_manager.cc:5697] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f reported cstate change: term changed from 0 to 1, leader changed from <none> to 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65). New cstate: current_term: 1 leader_uuid: "31750ab24a654ede8e5d60d6ab45f76f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:53.068737  3923 raft_consensus.cc:359] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.073073  3923 raft_consensus.cc:385] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:53.073508  3923 raft_consensus.cc:740] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:53.071102  3920 tablet_bootstrap.cc:654] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:53.073736  3631 raft_consensus.cc:2468] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 1.
I20260709 13:12:53.078758  3749 leader_election.cc:304] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 2462c58306174ec1a384ea49fddf64c1
I20260709 13:12:53.079671  3937 raft_consensus.cc:2804] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:53.080312  3937 raft_consensus.cc:697] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Becoming Leader. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:53.080763  3923 consensus_queue.cc:260] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.081293  3937 consensus_queue.cc:237] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.083159  3923 ts_tablet_manager.cc:1434] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 13:12:53.083781  3908 raft_consensus.cc:493] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:53.084172  3923 tablet_bootstrap.cc:492] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1: Bootstrap starting.
I20260709 13:12:53.084250  3908 raft_consensus.cc:515] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.086485  3908 leader_election.cc:290] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:53.093031  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0d21b4836594593ac3f578d56f91e4a" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
W20260709 13:12:53.094498  3746 leader_election.cc:343] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:53.095804  3631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0d21b4836594593ac3f578d56f91e4a" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f" is_pre_election: true
I20260709 13:12:53.096481  3923 tablet_bootstrap.cc:654] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:53.096537  3631 raft_consensus.cc:2468] T b0d21b4836594593ac3f578d56f91e4a P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 0.
I20260709 13:12:53.098086  3749 leader_election.cc:304] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 2462c58306174ec1a384ea49fddf64c1
I20260709 13:12:53.099509  3908 raft_consensus.cc:2804] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:53.100006  3908 raft_consensus.cc:493] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:53.100416  3908 raft_consensus.cc:3060] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:53.077291  3781 raft_consensus.cc:2468] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 1.
I20260709 13:12:53.113304  3597 leader_election.cc:304] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 2462c58306174ec1a384ea49fddf64c1
I20260709 13:12:53.114952  3908 raft_consensus.cc:515] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.115021  3942 raft_consensus.cc:2804] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:53.116855  3942 raft_consensus.cc:697] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Becoming Leader. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:53.117725  3942 consensus_queue.cc:237] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.118963  3631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0d21b4836594593ac3f578d56f91e4a" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:53.119621  3631 raft_consensus.cc:3060] T b0d21b4836594593ac3f578d56f91e4a P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:53.115623  3923 tablet_bootstrap.cc:492] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1: No bootstrap required, opened a new log
I20260709 13:12:53.127099  3923 ts_tablet_manager.cc:1403] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1: Time spent bootstrapping tablet: real 0.043s	user 0.014s	sys 0.000s
I20260709 13:12:53.129026  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0d21b4836594593ac3f578d56f91e4a" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:53.130165  3923 raft_consensus.cc:359] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
W20260709 13:12:53.130733  3746 leader_election.cc:343] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:12:53.134547  3908 leader_election.cc:290] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:53.130853  3923 raft_consensus.cc:385] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:53.135939  3923 raft_consensus.cc:740] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:53.137548  3923 consensus_queue.cc:260] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.140677  3920 tablet_bootstrap.cc:492] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f: No bootstrap required, opened a new log
I20260709 13:12:53.141363  3920 ts_tablet_manager.cc:1403] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent bootstrapping tablet: real 0.091s	user 0.015s	sys 0.013s
I20260709 13:12:53.143884  3921 raft_consensus.cc:493] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:53.144413  3921 raft_consensus.cc:515] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.146646  3921 leader_election.cc:290] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:53.144937  3920 raft_consensus.cc:359] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.148818  3920 raft_consensus.cc:385] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:53.148969  3631 raft_consensus.cc:2468] T b0d21b4836594593ac3f578d56f91e4a P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 1.
I20260709 13:12:53.149113  3920 raft_consensus.cc:740] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Initialized, Role: FOLLOWER
I20260709 13:12:53.150456  3920 consensus_queue.cc:260] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.153215  3920 ts_tablet_manager.cc:1434] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 13:12:53.155123  3923 ts_tablet_manager.cc:1434] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1: Time spent starting tablet: real 0.028s	user 0.006s	sys 0.001s
I20260709 13:12:53.156584  3923 tablet_bootstrap.cc:492] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1: Bootstrap starting.
I20260709 13:12:53.149232  3632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57720afe068341ab8da51d15b129d2d6" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f" is_pre_election: true
I20260709 13:12:53.159514  3632 raft_consensus.cc:2393] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2462c58306174ec1a384ea49fddf64c1 in current term 1: Already voted for candidate cb3f52aae763473cb09593496219dea8 in this term.
I20260709 13:12:53.158049  3749 leader_election.cc:304] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 2462c58306174ec1a384ea49fddf64c1
I20260709 13:12:53.161577  3937 raft_consensus.cc:2804] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:53.162002  3781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57720afe068341ab8da51d15b129d2d6" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:53.162081  3937 raft_consensus.cc:697] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Becoming Leader. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:53.163229  3937 consensus_queue.cc:237] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.163744  3672 leader_election.cc:304] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1; no voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8
I20260709 13:12:53.164139  3923 tablet_bootstrap.cc:654] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:53.165386  3921 raft_consensus.cc:3060] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:53.175696  3921 raft_consensus.cc:2749] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:12:53.175850  3923 tablet_bootstrap.cc:492] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1: No bootstrap required, opened a new log
I20260709 13:12:53.176621  3923 ts_tablet_manager.cc:1403] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1: Time spent bootstrapping tablet: real 0.020s	user 0.008s	sys 0.012s
I20260709 13:12:53.178591  3942 raft_consensus.cc:493] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:53.179152  3942 raft_consensus.cc:515] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.179404  3923 raft_consensus.cc:359] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.180140  3923 raft_consensus.cc:385] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:53.180384  3923 raft_consensus.cc:740] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:53.181035  3942 leader_election.cc:290] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:53.181015  3923 consensus_queue.cc:260] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.182078  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fe236a52db543c18530f238587d8346" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
I20260709 13:12:53.183167  3923 ts_tablet_manager.cc:1434] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.003s
I20260709 13:12:53.183795  3781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fe236a52db543c18530f238587d8346" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:53.184134  3923 tablet_bootstrap.cc:492] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1: Bootstrap starting.
I20260709 13:12:53.184578  3781 raft_consensus.cc:2393] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31750ab24a654ede8e5d60d6ab45f76f in current term 1: Already voted for candidate 2462c58306174ec1a384ea49fddf64c1 in this term.
I20260709 13:12:53.186853  3530 catalog_manager.cc:5697] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb3f52aae763473cb09593496219dea8 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "cb3f52aae763473cb09593496219dea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:53.188162  3530 catalog_manager.cc:5697] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb3f52aae763473cb09593496219dea8 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "cb3f52aae763473cb09593496219dea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:53.190012  3923 tablet_bootstrap.cc:654] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:12:53.190315  3597 leader_election.cc:304] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f; no voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8
I20260709 13:12:53.191562  3942 raft_consensus.cc:3060] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:53.202716  3942 raft_consensus.cc:2749] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:12:53.203675  3923 tablet_bootstrap.cc:492] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1: No bootstrap required, opened a new log
I20260709 13:12:53.204212  3923 ts_tablet_manager.cc:1403] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1: Time spent bootstrapping tablet: real 0.020s	user 0.010s	sys 0.006s
I20260709 13:12:53.206511  3923 raft_consensus.cc:359] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.207015  3923 raft_consensus.cc:385] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:12:53.207268  3923 raft_consensus.cc:740] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Initialized, Role: FOLLOWER
I20260709 13:12:53.207859  3923 consensus_queue.cc:260] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.209793  3923 ts_tablet_manager.cc:1434] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1: Time spent starting tablet: real 0.005s	user 0.007s	sys 0.000s
I20260709 13:12:53.209271  3532 catalog_manager.cc:5697] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f reported cstate change: term changed from 0 to 1, leader changed from <none> to 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65). New cstate: current_term: 1 leader_uuid: "31750ab24a654ede8e5d60d6ab45f76f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:53.225021  3528 catalog_manager.cc:5697] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb3f52aae763473cb09593496219dea8 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "cb3f52aae763473cb09593496219dea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:53.226424  3528 catalog_manager.cc:5697] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb3f52aae763473cb09593496219dea8 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "cb3f52aae763473cb09593496219dea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:53.227406  3528 catalog_manager.cc:5697] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb3f52aae763473cb09593496219dea8 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "cb3f52aae763473cb09593496219dea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:53.277642  3921 raft_consensus.cc:493] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:53.278121  3921 raft_consensus.cc:515] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.279851  3921 leader_election.cc:290] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:53.280638  3632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c195102cee49d39fe701c20ac7dacf" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f" is_pre_election: true
I20260709 13:12:53.280784  3781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c195102cee49d39fe701c20ac7dacf" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:53.281303  3632 raft_consensus.cc:2468] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 0.
I20260709 13:12:53.281489  3781 raft_consensus.cc:2468] T b2c195102cee49d39fe701c20ac7dacf P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 0.
I20260709 13:12:53.282290  3675 leader_election.cc:304] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, 31750ab24a654ede8e5d60d6ab45f76f; no voters: 
I20260709 13:12:53.283111  3921 raft_consensus.cc:2804] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:12:53.283799  3921 raft_consensus.cc:493] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:12:53.284121  3921 raft_consensus.cc:3060] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:53.288987  3921 raft_consensus.cc:515] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.290699  3921 leader_election.cc:290] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:53.291534  3632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c195102cee49d39fe701c20ac7dacf" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:53.291793  3781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c195102cee49d39fe701c20ac7dacf" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:53.292079  3632 raft_consensus.cc:3060] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:53.292411  3781 raft_consensus.cc:3060] T b2c195102cee49d39fe701c20ac7dacf P cb3f52aae763473cb09593496219dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:53.296977  3632 raft_consensus.cc:2468] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 1.
I20260709 13:12:53.297554  3781 raft_consensus.cc:2468] T b2c195102cee49d39fe701c20ac7dacf P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 1.
I20260709 13:12:53.298144  3675 leader_election.cc:304] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, 31750ab24a654ede8e5d60d6ab45f76f; no voters: 
I20260709 13:12:53.298996  3921 raft_consensus.cc:2804] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:12:53.299463  3921 raft_consensus.cc:697] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Running, Role: LEADER
I20260709 13:12:53.300194  3921 consensus_queue.cc:237] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.307900  3528 catalog_manager.cc:5697] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66). New cstate: current_term: 1 leader_uuid: "2462c58306174ec1a384ea49fddf64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:53.339856  3937 consensus_queue.cc:1048] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:53.347573  3931 consensus_queue.cc:1048] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:53.354202  3709 raft_consensus.cc:3060] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:53.357338  3921 raft_consensus.cc:493] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:53.361927  3921 raft_consensus.cc:515] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.365183  3783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0d21b4836594593ac3f578d56f91e4a" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:53.365973  3921 leader_election.cc:290] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:53.368191  3632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0d21b4836594593ac3f578d56f91e4a" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f" is_pre_election: true
I20260709 13:12:53.369021  3632 raft_consensus.cc:2393] T b0d21b4836594593ac3f578d56f91e4a P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2462c58306174ec1a384ea49fddf64c1 in current term 1: Already voted for candidate cb3f52aae763473cb09593496219dea8 in this term.
I20260709 13:12:53.372874  3937 consensus_queue.cc:1048] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:53.373610  3950 raft_consensus.cc:493] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:53.374619  3950 raft_consensus.cc:515] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.378043  3950 leader_election.cc:290] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:53.378551  3634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "605b25b7889545a4b965c755cbead46a" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f" is_pre_election: true
I20260709 13:12:53.379168  3781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "605b25b7889545a4b965c755cbead46a" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:53.383996  3942 consensus_queue.cc:1048] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:53.388113  3942 raft_consensus.cc:493] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:53.388064  3675 leader_election.cc:304] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1; no voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8
I20260709 13:12:53.388710  3942 raft_consensus.cc:515] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.389050  3831 raft_consensus.cc:3060] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:53.389997  3937 consensus_queue.cc:1048] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:53.391655  3942 leader_election.cc:290] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:53.392325  3781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49efa8eb906145db89256ddc020530a3" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:53.390872  3675 leader_election.cc:304] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1; no voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8
I20260709 13:12:53.397281  3831 raft_consensus.cc:2749] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:12:53.402268  3950 raft_consensus.cc:2749] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:53.403837  3950 consensus_queue.cc:1048] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:53.424109  3942 consensus_queue.cc:1048] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:53.427639  3709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49efa8eb906145db89256ddc020530a3" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2462c58306174ec1a384ea49fddf64c1" is_pre_election: true
I20260709 13:12:53.429610  3709 raft_consensus.cc:2393] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31750ab24a654ede8e5d60d6ab45f76f in current term 1: Already voted for candidate cb3f52aae763473cb09593496219dea8 in this term.
I20260709 13:12:53.428267  3950 consensus_queue.cc:1048] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:53.432238  3783 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "57720afe068341ab8da51d15b129d2d6"
dest_uuid: "cb3f52aae763473cb09593496219dea8"
mode: GRACEFUL
new_leader_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
 from {username='slave'} at 127.0.0.1:47852
I20260709 13:12:53.432322  3597 leader_election.cc:304] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f; no voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8
I20260709 13:12:53.433063  3783 raft_consensus.cc:606] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Received request to transfer leadership to TS 31750ab24a654ede8e5d60d6ab45f76f
I20260709 13:12:53.433650  3953 raft_consensus.cc:3060] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:53.443122  3953 raft_consensus.cc:2749] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:12:53.445694  3783 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "605b25b7889545a4b965c755cbead46a"
dest_uuid: "cb3f52aae763473cb09593496219dea8"
mode: GRACEFUL
new_leader_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
 from {username='slave'} at 127.0.0.1:47852
I20260709 13:12:53.446405  3783 raft_consensus.cc:606] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Received request to transfer leadership to TS 31750ab24a654ede8e5d60d6ab45f76f
I20260709 13:12:53.485284  3707 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2fe236a52db543c18530f238587d8346"
dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
mode: GRACEFUL
new_leader_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
 from {username='slave'} at 127.0.0.1:47976
I20260709 13:12:53.485767  3707 raft_consensus.cc:606] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 1 LEADER]: Received request to transfer leadership to TS 31750ab24a654ede8e5d60d6ab45f76f
I20260709 13:12:53.491010  3783 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "49efa8eb906145db89256ddc020530a3"
dest_uuid: "cb3f52aae763473cb09593496219dea8"
mode: GRACEFUL
new_leader_uuid: "2462c58306174ec1a384ea49fddf64c1"
 from {username='slave'} at 127.0.0.1:47852
I20260709 13:12:53.491466  3783 raft_consensus.cc:606] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Received request to transfer leadership to TS 2462c58306174ec1a384ea49fddf64c1
I20260709 13:12:53.496302  3937 consensus_queue.cc:1048] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:53.496433  3783 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d34d16630b304796b7cb8c4de4f0c3a8"
dest_uuid: "cb3f52aae763473cb09593496219dea8"
mode: GRACEFUL
new_leader_uuid: "2462c58306174ec1a384ea49fddf64c1"
 from {username='slave'} at 127.0.0.1:47852
I20260709 13:12:53.497363  3783 raft_consensus.cc:606] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Received request to transfer leadership to TS 2462c58306174ec1a384ea49fddf64c1
I20260709 13:12:53.503465  3631 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9fdc51f4bd024752b1b8a75cd782a03e"
dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
mode: GRACEFUL
new_leader_uuid: "2462c58306174ec1a384ea49fddf64c1"
 from {username='slave'} at 127.0.0.1:53662
I20260709 13:12:53.504041  3631 raft_consensus.cc:606] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Received request to transfer leadership to TS 2462c58306174ec1a384ea49fddf64c1
I20260709 13:12:53.505937  3931 consensus_queue.cc:1048] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:53.513892  3931 raft_consensus.cc:993] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8: : Instructing follower 31750ab24a654ede8e5d60d6ab45f76f to start an election
I20260709 13:12:53.514314  3931 raft_consensus.cc:1081] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Signalling peer 31750ab24a654ede8e5d60d6ab45f76f to start an election
I20260709 13:12:53.517325  3631 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "57720afe068341ab8da51d15b129d2d6"
dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
 from {username='slave'} at 127.0.0.1:53642
I20260709 13:12:53.517880  3631 raft_consensus.cc:493] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:53.518277  3631 raft_consensus.cc:3060] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:53.519471  3937 consensus_queue.cc:1048] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:53.525460  3631 raft_consensus.cc:515] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.530122  3783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57720afe068341ab8da51d15b129d2d6" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:53.530720  3783 raft_consensus.cc:3055] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:53.531051  3783 raft_consensus.cc:740] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:53.531354  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57720afe068341ab8da51d15b129d2d6" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:53.531919  3708 raft_consensus.cc:3060] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:53.531884  3783 consensus_queue.cc:260] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.533483  3783 raft_consensus.cc:3060] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:53.537793  3937 consensus_queue.cc:1048] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:53.538631  3708 raft_consensus.cc:2468] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 2.
I20260709 13:12:53.539868  3597 leader_election.cc:304] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, 31750ab24a654ede8e5d60d6ab45f76f; no voters: 
I20260709 13:12:53.540797  3953 raft_consensus.cc:2804] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:53.541339  3953 raft_consensus.cc:697] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 LEADER]: Becoming Leader. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:53.542152  3953 consensus_queue.cc:237] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.549283  3631 leader_election.cc:290] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 2 election: Requested vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:53.551604  3783 raft_consensus.cc:2468] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 2.
I20260709 13:12:53.555990  3975 raft_consensus.cc:993] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8: : Instructing follower 2462c58306174ec1a384ea49fddf64c1 to start an election
I20260709 13:12:53.556478  3937 raft_consensus.cc:1081] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Signalling peer 2462c58306174ec1a384ea49fddf64c1 to start an election
I20260709 13:12:53.557942  3707 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "49efa8eb906145db89256ddc020530a3"
dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
 from {username='slave'} at 127.0.0.1:47968
I20260709 13:12:53.558540  3707 raft_consensus.cc:493] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:53.558907  3707 raft_consensus.cc:3060] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:53.559528  3528 catalog_manager.cc:5697] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f reported cstate change: term changed from 1 to 2, leader changed from cb3f52aae763473cb09593496219dea8 (127.23.140.67) to 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65). New cstate: current_term: 2 leader_uuid: "31750ab24a654ede8e5d60d6ab45f76f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:53.567773  3707 raft_consensus.cc:515] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.571208  3707 leader_election.cc:290] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 2 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:53.571810  3783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49efa8eb906145db89256ddc020530a3" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:53.572485  3783 raft_consensus.cc:3055] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:53.572734  3783 raft_consensus.cc:740] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:53.573411  3783 consensus_queue.cc:260] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.574373  3783 raft_consensus.cc:3060] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:53.580891  3783 raft_consensus.cc:2468] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 2.
I20260709 13:12:53.620951  3631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49efa8eb906145db89256ddc020530a3" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:53.624807  3631 raft_consensus.cc:3060] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:53.624825  3672 leader_election.cc:304] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:53.627246  3984 raft_consensus.cc:2804] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:53.627980  3984 raft_consensus.cc:697] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Becoming Leader. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Running, Role: LEADER
I20260709 13:12:53.627699  3953 consensus_queue.cc:1048] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:12:53.630261  3984 consensus_queue.cc:237] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.638917  3631 raft_consensus.cc:2468] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 2.
I20260709 13:12:53.645771  3528 catalog_manager.cc:5697] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 reported cstate change: term changed from 1 to 2, leader changed from cb3f52aae763473cb09593496219dea8 (127.23.140.67) to 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2462c58306174ec1a384ea49fddf64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:53.652068  3707 raft_consensus.cc:3060] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:12:53.667400  3942 consensus_queue.cc:1048] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:12:53.668867  3937 consensus_queue.cc:1048] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:12:53.669471  3984 raft_consensus.cc:493] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:12:53.676668  3984 raft_consensus.cc:515] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.681072  3633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fdc51f4bd024752b1b8a75cd782a03e" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f" is_pre_election: true
I20260709 13:12:53.681991  3984 leader_election.cc:290] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:53.682811  3975 consensus_queue.cc:1048] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:53.685333  3783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fdc51f4bd024752b1b8a75cd782a03e" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3f52aae763473cb09593496219dea8" is_pre_election: true
I20260709 13:12:53.687160  3672 leader_election.cc:304] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1; no voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8
I20260709 13:12:53.705418  3984 raft_consensus.cc:2749] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:12:53.713945  3906 raft_consensus.cc:993] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f: : Instructing follower 2462c58306174ec1a384ea49fddf64c1 to start an election
I20260709 13:12:53.714512  3953 raft_consensus.cc:1081] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Signalling peer 2462c58306174ec1a384ea49fddf64c1 to start an election
I20260709 13:12:53.716224  3707 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9fdc51f4bd024752b1b8a75cd782a03e"
dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
 from {username='slave'} at 127.0.0.1:47952
I20260709 13:12:53.716877  3707 raft_consensus.cc:493] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:53.717286  3707 raft_consensus.cc:3060] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:53.723682  3707 raft_consensus.cc:515] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.725746  3707 leader_election.cc:290] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 2 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:53.726122  3633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fdc51f4bd024752b1b8a75cd782a03e" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:53.726404  3783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fdc51f4bd024752b1b8a75cd782a03e" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:53.726773  3633 raft_consensus.cc:3055] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:53.727125  3783 raft_consensus.cc:3060] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:53.727123  3633 raft_consensus.cc:740] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:53.728253  3633 consensus_queue.cc:260] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.729403  3633 raft_consensus.cc:3060] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:53.734545  3783 raft_consensus.cc:2468] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 2.
I20260709 13:12:53.735775  3672 leader_election.cc:304] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:53.739674  3984 raft_consensus.cc:2804] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:53.740271  3984 raft_consensus.cc:697] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Becoming Leader. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Running, Role: LEADER
I20260709 13:12:53.740684  3633 raft_consensus.cc:2468] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 2.
I20260709 13:12:53.742494  3984 consensus_queue.cc:237] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.751331  3530 catalog_manager.cc:5697] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 reported cstate change: term changed from 1 to 2, leader changed from 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65) to 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2462c58306174ec1a384ea49fddf64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:53.776335  3984 consensus_queue.cc:1048] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:12:53.789666  3831 consensus_queue.cc:1048] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:12:53.842696  3950 raft_consensus.cc:993] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1: : Instructing follower 31750ab24a654ede8e5d60d6ab45f76f to start an election
I20260709 13:12:53.843184  3831 raft_consensus.cc:1081] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 1 LEADER]: Signalling peer 31750ab24a654ede8e5d60d6ab45f76f to start an election
I20260709 13:12:53.844823  3633 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2fe236a52db543c18530f238587d8346"
dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
 from {username='slave'} at 127.0.0.1:53658
I20260709 13:12:53.845325  3633 raft_consensus.cc:493] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:53.845667  3633 raft_consensus.cc:3060] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:53.850952  3633 raft_consensus.cc:515] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.852577  3633 leader_election.cc:290] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 2 election: Requested vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:53.853417  3707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fe236a52db543c18530f238587d8346" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:53.853925  3781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fe236a52db543c18530f238587d8346" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:53.854118  3707 raft_consensus.cc:3055] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:53.854563  3707 raft_consensus.cc:740] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Running, Role: LEADER
I20260709 13:12:53.854616  3781 raft_consensus.cc:3060] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:53.855861  3707 consensus_queue.cc:260] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.857864  3707 raft_consensus.cc:3060] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:53.865049  3781 raft_consensus.cc:2468] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 2.
I20260709 13:12:53.865751  3707 raft_consensus.cc:2468] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 2.
I20260709 13:12:53.866478  3597 leader_election.cc:304] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:53.867254  3953 raft_consensus.cc:2804] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:53.867676  3953 raft_consensus.cc:697] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 LEADER]: Becoming Leader. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:53.868618  3953 consensus_queue.cc:237] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.875993  3528 catalog_manager.cc:5697] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f reported cstate change: term changed from 1 to 2, leader changed from 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66) to 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65). New cstate: current_term: 2 leader_uuid: "31750ab24a654ede8e5d60d6ab45f76f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:53.877419  3931 raft_consensus.cc:993] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8: : Instructing follower 31750ab24a654ede8e5d60d6ab45f76f to start an election
I20260709 13:12:53.878032  3975 raft_consensus.cc:1081] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Signalling peer 31750ab24a654ede8e5d60d6ab45f76f to start an election
I20260709 13:12:53.879756  3633 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "605b25b7889545a4b965c755cbead46a"
dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
 from {username='slave'} at 127.0.0.1:53642
I20260709 13:12:53.880343  3633 raft_consensus.cc:493] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:53.880770  3633 raft_consensus.cc:3060] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:53.889441  3633 raft_consensus.cc:515] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.891158  3633 leader_election.cc:290] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 2 election: Requested vote from peers 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:53.891907  3707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "605b25b7889545a4b965c755cbead46a" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:53.892359  3781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "605b25b7889545a4b965c755cbead46a" candidate_uuid: "31750ab24a654ede8e5d60d6ab45f76f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:53.892709  3707 raft_consensus.cc:3060] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:53.893160  3781 raft_consensus.cc:3055] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:53.893602  3781 raft_consensus.cc:740] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:53.894841  3781 consensus_queue.cc:260] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.896494  3781 raft_consensus.cc:3060] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:53.901063  3707 raft_consensus.cc:2468] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 2.
I20260709 13:12:53.902439  3597 leader_election.cc:304] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, 31750ab24a654ede8e5d60d6ab45f76f; no voters: 
I20260709 13:12:53.903246  3953 raft_consensus.cc:2804] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:53.903744  3953 raft_consensus.cc:697] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [term 2 LEADER]: Becoming Leader. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:53.904691  3953 consensus_queue.cc:237] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.905668  3781 raft_consensus.cc:2468] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31750ab24a654ede8e5d60d6ab45f76f in term 2.
I20260709 13:12:53.911269  3528 catalog_manager.cc:5697] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f reported cstate change: term changed from 1 to 2, leader changed from cb3f52aae763473cb09593496219dea8 (127.23.140.67) to 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65). New cstate: current_term: 2 leader_uuid: "31750ab24a654ede8e5d60d6ab45f76f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:53.955235  3931 raft_consensus.cc:993] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8: : Instructing follower 2462c58306174ec1a384ea49fddf64c1 to start an election
I20260709 13:12:53.955689  3975 raft_consensus.cc:1081] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Signalling peer 2462c58306174ec1a384ea49fddf64c1 to start an election
I20260709 13:12:53.957093  3707 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d34d16630b304796b7cb8c4de4f0c3a8"
dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
 from {username='slave'} at 127.0.0.1:47968
I20260709 13:12:53.957614  3707 raft_consensus.cc:493] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:53.957894  3707 raft_consensus.cc:3060] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:53.963392  3707 raft_consensus.cc:515] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.965281  3707 leader_election.cc:290] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 2 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), cb3f52aae763473cb09593496219dea8 (127.23.140.67:35263)
I20260709 13:12:53.966097  3633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d34d16630b304796b7cb8c4de4f0c3a8" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:53.966240  3781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d34d16630b304796b7cb8c4de4f0c3a8" candidate_uuid: "2462c58306174ec1a384ea49fddf64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cb3f52aae763473cb09593496219dea8"
I20260709 13:12:53.966878  3633 raft_consensus.cc:3060] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:53.966934  3781 raft_consensus.cc:3055] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:53.967375  3781 raft_consensus.cc:740] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:53.968338  3781 consensus_queue.cc:260] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.969684  3781 raft_consensus.cc:3060] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:53.973021  3783 raft_consensus.cc:1275] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Refusing update from remote peer 31750ab24a654ede8e5d60d6ab45f76f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:53.974265  3953 consensus_queue.cc:1048] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:53.974411  3633 raft_consensus.cc:2468] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 2.
I20260709 13:12:53.975898  3675 leader_election.cc:304] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, 31750ab24a654ede8e5d60d6ab45f76f; no voters: 
I20260709 13:12:53.976753  3831 raft_consensus.cc:2804] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:53.977262  3831 raft_consensus.cc:697] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Becoming Leader. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Running, Role: LEADER
I20260709 13:12:53.977456  3781 raft_consensus.cc:2468] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2462c58306174ec1a384ea49fddf64c1 in term 2.
I20260709 13:12:53.978364  3831 consensus_queue.cc:237] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:53.984236  3707 raft_consensus.cc:1275] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Refusing update from remote peer 31750ab24a654ede8e5d60d6ab45f76f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:53.987071  3906 consensus_queue.cc:1048] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:53.989955  3528 catalog_manager.cc:5697] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 reported cstate change: term changed from 1 to 2, leader changed from cb3f52aae763473cb09593496219dea8 (127.23.140.67) to 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66). New cstate: current_term: 2 leader_uuid: "2462c58306174ec1a384ea49fddf64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: UNKNOWN } } }
I20260709 13:12:54.062887  3783 raft_consensus.cc:1275] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Refusing update from remote peer 2462c58306174ec1a384ea49fddf64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:54.064417  3831 consensus_queue.cc:1048] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:54.073890  3633 raft_consensus.cc:1275] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Refusing update from remote peer 2462c58306174ec1a384ea49fddf64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:54.075274  3831 consensus_queue.cc:1048] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:54.214655  3633 raft_consensus.cc:1275] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Refusing update from remote peer 2462c58306174ec1a384ea49fddf64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:54.216002  3831 consensus_queue.cc:1048] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:54.224501  3783 raft_consensus.cc:1275] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Refusing update from remote peer 2462c58306174ec1a384ea49fddf64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:54.225832  3950 consensus_queue.cc:1048] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:54.322024  3783 raft_consensus.cc:1275] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Refusing update from remote peer 31750ab24a654ede8e5d60d6ab45f76f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:54.323262  3953 consensus_queue.cc:1048] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:54.331875  3708 raft_consensus.cc:1275] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Refusing update from remote peer 31750ab24a654ede8e5d60d6ab45f76f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:54.333103  3953 consensus_queue.cc:1048] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:54.386986  3708 raft_consensus.cc:1275] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Refusing update from remote peer 31750ab24a654ede8e5d60d6ab45f76f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:54.387965  3953 consensus_queue.cc:1048] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:54.397466  3783 raft_consensus.cc:1275] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Refusing update from remote peer 31750ab24a654ede8e5d60d6ab45f76f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:54.399609  3942 consensus_queue.cc:1048] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:54.453328  3783 raft_consensus.cc:1275] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Refusing update from remote peer 2462c58306174ec1a384ea49fddf64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:54.454624  3950 consensus_queue.cc:1048] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:54.463505  3633 raft_consensus.cc:1275] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Refusing update from remote peer 2462c58306174ec1a384ea49fddf64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:54.464820  3831 consensus_queue.cc:1048] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:56.519347  3529 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49398:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 13:12:56.539417  3529 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49398:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 13:12:56.579399 24113 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
W20260709 13:12:56.745308  4020 log.cc:927] Time spent T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
W20260709 13:12:56.788982  3672 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:35263, user_credentials={real_user=slave}} blocked reactor thread for 55593.6us
W20260709 13:12:56.809823  4026 log.cc:927] Time spent T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
W20260709 13:12:56.809930  4022 log.cc:927] Time spent T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
W20260709 13:12:56.864281  3672 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:35263, user_credentials={real_user=slave}} blocked reactor thread for 44983.8us
W20260709 13:12:56.872268  4032 log.cc:927] Time spent T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
W20260709 13:12:56.957993  4043 log.cc:927] Time spent T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1: Append to log took a long time: real 0.095s	user 0.000s	sys 0.002s
I20260709 13:12:56.957741  4059 tablet.cc:1724] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:56.931754  4052 tablet.cc:1724] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:56.996227  4062 tablet.cc:1724] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.032398  4069 tablet.cc:1724] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.051585  4064 tablet.cc:1724] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.045291  4071 tablet.cc:1724] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:56.972440  4065 tablet.cc:1724] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 13:12:57.087370  3735 tablet_mm_ops.cc:139] Rowset compaction is disabled (check --enable_rowset_compaction and disable_compaction in extra_config for tablet:49efa8eb906145db89256ddc020530a3)
I20260709 13:12:57.095981  4059 tablet.cc:1724] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 13:12:57.107491  3660 tablet_mm_ops.cc:304] Major delta compaction is disabled (check --enable_major_delta_compaction and disable_compaction in extra_config for tablet:2fe236a52db543c18530f238587d8346)
W20260709 13:12:57.108472  3660 tablet_mm_ops.cc:220] Minor delta compaction is disabled (check --enable_minor_delta_compaction and disable_compaction in extra_config for tablet:2fe236a52db543c18530f238587d8346)
I20260709 13:12:57.028687  4070 tablet.cc:1724] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.110729  4072 tablet.cc:1724] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.117652  3633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "57720afe068341ab8da51d15b129d2d6"
dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
mode: GRACEFUL
new_leader_uuid: "cb3f52aae763473cb09593496219dea8"
 from {username='slave'} at 127.0.0.1:37156
I20260709 13:12:57.118402  3633 raft_consensus.cc:606] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 LEADER]: Received request to transfer leadership to TS cb3f52aae763473cb09593496219dea8
I20260709 13:12:57.120685  4066 tablet.cc:1724] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.123939  4063 tablet.cc:1724] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.130534  4062 tablet.cc:1724] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.134935  4056 raft_consensus.cc:993] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f: : Instructing follower cb3f52aae763473cb09593496219dea8 to start an election
I20260709 13:12:57.148386  4056 raft_consensus.cc:1081] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 LEADER]: Signalling peer cb3f52aae763473cb09593496219dea8 to start an election
I20260709 13:12:57.129426  4068 tablet.cc:1724] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.150357  4052 tablet.cc:1724] T b0d21b4836594593ac3f578d56f91e4a P 31750ab24a654ede8e5d60d6ab45f76f: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.154827  3781 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "57720afe068341ab8da51d15b129d2d6"
dest_uuid: "cb3f52aae763473cb09593496219dea8"
 from {username='slave'} at 127.0.0.1:47836
I20260709 13:12:57.155562  3781 raft_consensus.cc:493] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:57.155984  3781 raft_consensus.cc:3060] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:57.172051  4064 tablet.cc:1724] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.181531  4062 tablet.cc:1724] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.190310  4065 tablet.cc:1724] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.188678  3781 raft_consensus.cc:515] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:57.195394  4069 tablet.cc:1724] T dbcad4026f8440949bb13e46b88fc340 P cb3f52aae763473cb09593496219dea8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.198487  3633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57720afe068341ab8da51d15b129d2d6" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:57.214238  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57720afe068341ab8da51d15b129d2d6" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:57.214874  3708 raft_consensus.cc:3060] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:57.199812  4071 tablet.cc:1724] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.214267  3633 raft_consensus.cc:3055] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:57.218523  3633 raft_consensus.cc:740] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:57.219369  3633 consensus_queue.cc:260] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:57.220468  3633 raft_consensus.cc:3060] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:57.225580  4064 tablet.cc:1724] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.224335  4068 tablet.cc:1724] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.232709  4071 tablet.cc:1724] T b2c195102cee49d39fe701c20ac7dacf P cb3f52aae763473cb09593496219dea8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.234711  3706 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9fdc51f4bd024752b1b8a75cd782a03e"
dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
mode: GRACEFUL
new_leader_uuid: "cb3f52aae763473cb09593496219dea8"
 from {username='slave'} at 127.0.0.1:38246
I20260709 13:12:57.235285  3706 raft_consensus.cc:606] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Received request to transfer leadership to TS cb3f52aae763473cb09593496219dea8
I20260709 13:12:57.201538  4070 tablet.cc:1724] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.201939  4059 tablet.cc:1724] T dbcad4026f8440949bb13e46b88fc340 P 2462c58306174ec1a384ea49fddf64c1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.236722 24113 auto_leader_rebalancer.cc:452] table: soft_deleted_table, leader rebalance finish, leader transfer count: 2
I20260709 13:12:57.237602 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:12:57.210764  4066 tablet.cc:1724] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.236732  3708 raft_consensus.cc:2468] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 3.
I20260709 13:12:57.244517  3746 leader_election.cc:304] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:57.244714  3633 raft_consensus.cc:2468] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 3.
I20260709 13:12:57.245651  4045 raft_consensus.cc:2804] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:57.246152  4045 raft_consensus.cc:697] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 3 LEADER]: Becoming Leader. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:57.247078  4045 consensus_queue.cc:237] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:57.257706  3528 catalog_manager.cc:5697] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 reported cstate change: term changed from 2 to 3, leader changed from 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65) to cb3f52aae763473cb09593496219dea8 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "cb3f52aae763473cb09593496219dea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:57.211443  3781 leader_election.cc:290] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 3 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:57.213001  4052 tablet.cc:1724] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:12:57.272390  3708 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "20eb2e33130740e38d072182154c715f"
dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
mode: GRACEFUL
new_leader_uuid: "cb3f52aae763473cb09593496219dea8"
 from {username='slave'} at 127.0.0.1:38246
I20260709 13:12:57.272945  3708 raft_consensus.cc:606] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Received request to transfer leadership to TS cb3f52aae763473cb09593496219dea8
I20260709 13:12:57.276970  3633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a5aed345cd424405a20d159a9f9762a0"
dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
mode: GRACEFUL
new_leader_uuid: "cb3f52aae763473cb09593496219dea8"
 from {username='slave'} at 127.0.0.1:37156
I20260709 13:12:57.277578  3633 raft_consensus.cc:606] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Received request to transfer leadership to TS cb3f52aae763473cb09593496219dea8
I20260709 13:12:57.278805 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:12:57.279259 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:12:57.284585  3530 catalog_manager.cc:6242] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610: soft_deleted_table [id=df231ab3964b41fd9f2a51becddbbb5b] alter complete (version 1)
I20260709 13:12:57.450327  4049 raft_consensus.cc:993] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1: : Instructing follower cb3f52aae763473cb09593496219dea8 to start an election
I20260709 13:12:57.450855  4036 raft_consensus.cc:1081] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Signalling peer cb3f52aae763473cb09593496219dea8 to start an election
I20260709 13:12:57.452246  3781 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "20eb2e33130740e38d072182154c715f"
dest_uuid: "cb3f52aae763473cb09593496219dea8"
 from {username='slave'} at 127.0.0.1:47850
I20260709 13:12:57.452762  3781 raft_consensus.cc:493] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:57.453042  3781 raft_consensus.cc:3060] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:57.457722  3781 raft_consensus.cc:515] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:57.459107  3781 leader_election.cc:290] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 3 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:57.460016  3633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20eb2e33130740e38d072182154c715f" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:57.460129  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20eb2e33130740e38d072182154c715f" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:57.460780  3633 raft_consensus.cc:3060] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:57.460825  3708 raft_consensus.cc:3055] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:57.461233  3708 raft_consensus.cc:740] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Running, Role: LEADER
I20260709 13:12:57.461900  3708 consensus_queue.cc:260] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:57.462900  3708 raft_consensus.cc:3060] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:57.465538  3633 raft_consensus.cc:2468] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 3.
I20260709 13:12:57.466758  3749 leader_election.cc:304] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:57.467640  4045 raft_consensus.cc:2804] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:57.467847  3708 raft_consensus.cc:2468] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 3.
I20260709 13:12:57.468149  4045 raft_consensus.cc:697] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [term 3 LEADER]: Becoming Leader. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:57.469112  4045 consensus_queue.cc:237] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:57.477655  3528 catalog_manager.cc:5697] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 reported cstate change: term changed from 2 to 3, leader changed from 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66) to cb3f52aae763473cb09593496219dea8 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "cb3f52aae763473cb09593496219dea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:57.497186  4040 raft_consensus.cc:993] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f: : Instructing follower cb3f52aae763473cb09593496219dea8 to start an election
I20260709 13:12:57.497839  4067 raft_consensus.cc:1081] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Signalling peer cb3f52aae763473cb09593496219dea8 to start an election
I20260709 13:12:57.499176  3781 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a5aed345cd424405a20d159a9f9762a0"
dest_uuid: "cb3f52aae763473cb09593496219dea8"
 from {username='slave'} at 127.0.0.1:47836
I20260709 13:12:57.499603  3781 raft_consensus.cc:493] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:57.499867  3781 raft_consensus.cc:3060] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:57.504472  3781 raft_consensus.cc:515] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:57.505940  3781 leader_election.cc:290] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 2 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:57.506724  3633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5aed345cd424405a20d159a9f9762a0" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:57.506983  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5aed345cd424405a20d159a9f9762a0" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:57.507314  3633 raft_consensus.cc:3055] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:12:57.507543  3708 raft_consensus.cc:3060] T a5aed345cd424405a20d159a9f9762a0 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:57.507665  3633 raft_consensus.cc:740] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31750ab24a654ede8e5d60d6ab45f76f, State: Running, Role: LEADER
I20260709 13:12:57.508425  3633 consensus_queue.cc:260] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:57.509383  3633 raft_consensus.cc:3060] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:12:57.512418  3708 raft_consensus.cc:2468] T a5aed345cd424405a20d159a9f9762a0 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 2.
I20260709 13:12:57.513466  3746 leader_election.cc:304] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2462c58306174ec1a384ea49fddf64c1, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:57.513852  3633 raft_consensus.cc:2468] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 2.
I20260709 13:12:57.514326  4045 raft_consensus.cc:2804] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:12:57.514875  4045 raft_consensus.cc:697] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [term 2 LEADER]: Becoming Leader. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:57.515699  4045 consensus_queue.cc:237] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:57.523861  3528 catalog_manager.cc:5697] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 reported cstate change: term changed from 1 to 2, leader changed from 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65) to cb3f52aae763473cb09593496219dea8 (127.23.140.67). New cstate: current_term: 2 leader_uuid: "cb3f52aae763473cb09593496219dea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:57.659196  3634 raft_consensus.cc:1275] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 3 FOLLOWER]: Refusing update from remote peer cb3f52aae763473cb09593496219dea8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:12:57.660490  4045 consensus_queue.cc:1048] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:12:57.675527  3708 raft_consensus.cc:1275] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1 [term 3 FOLLOWER]: Refusing update from remote peer cb3f52aae763473cb09593496219dea8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:12:57.677718  4084 consensus_queue.cc:1048] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:12:57.695143  4036 raft_consensus.cc:993] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1: : Instructing follower cb3f52aae763473cb09593496219dea8 to start an election
I20260709 13:12:57.695621  4049 raft_consensus.cc:1081] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Signalling peer cb3f52aae763473cb09593496219dea8 to start an election
I20260709 13:12:57.696946  3781 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9fdc51f4bd024752b1b8a75cd782a03e"
dest_uuid: "cb3f52aae763473cb09593496219dea8"
 from {username='slave'} at 127.0.0.1:47850
I20260709 13:12:57.697402  3781 raft_consensus.cc:493] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:12:57.697687  3781 raft_consensus.cc:3060] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:57.702116  3781 raft_consensus.cc:515] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:57.703577  3781 leader_election.cc:290] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 3 election: Requested vote from peers 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049), 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66:35473)
I20260709 13:12:57.704694  3708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fdc51f4bd024752b1b8a75cd782a03e" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "2462c58306174ec1a384ea49fddf64c1"
I20260709 13:12:57.705324  3708 raft_consensus.cc:3055] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:12:57.705287  3634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fdc51f4bd024752b1b8a75cd782a03e" candidate_uuid: "cb3f52aae763473cb09593496219dea8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "31750ab24a654ede8e5d60d6ab45f76f"
I20260709 13:12:57.705801  3708 raft_consensus.cc:740] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2462c58306174ec1a384ea49fddf64c1, State: Running, Role: LEADER
I20260709 13:12:57.706030  3634 raft_consensus.cc:3060] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:57.706611  3708 consensus_queue.cc:260] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:57.707649  3708 raft_consensus.cc:3060] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:12:57.711370  3634 raft_consensus.cc:2468] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 3.
I20260709 13:12:57.712407  3749 leader_election.cc:304] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31750ab24a654ede8e5d60d6ab45f76f, cb3f52aae763473cb09593496219dea8; no voters: 
I20260709 13:12:57.712473  3708 raft_consensus.cc:2468] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3f52aae763473cb09593496219dea8 in term 3.
I20260709 13:12:57.713167  4084 raft_consensus.cc:2804] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:12:57.713681  4084 raft_consensus.cc:697] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [term 3 LEADER]: Becoming Leader. State: Replica: cb3f52aae763473cb09593496219dea8, State: Running, Role: LEADER
I20260709 13:12:57.714552  4084 consensus_queue.cc:237] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } }
I20260709 13:12:57.720575  3530 catalog_manager.cc:5697] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 reported cstate change: term changed from 2 to 3, leader changed from 2462c58306174ec1a384ea49fddf64c1 (127.23.140.66) to cb3f52aae763473cb09593496219dea8 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "cb3f52aae763473cb09593496219dea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3f52aae763473cb09593496219dea8" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 35263 } health_report { overall_health: HEALTHY } } }
I20260709 13:12:57.866199  3708 raft_consensus.cc:1275] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 3 FOLLOWER]: Refusing update from remote peer cb3f52aae763473cb09593496219dea8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:57.867427  4084 consensus_queue.cc:1048] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:57.878839  3634 raft_consensus.cc:1275] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 3 FOLLOWER]: Refusing update from remote peer cb3f52aae763473cb09593496219dea8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:12:57.880373  4045 consensus_queue.cc:1048] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:12:58.019548  3708 raft_consensus.cc:1275] T a5aed345cd424405a20d159a9f9762a0 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Refusing update from remote peer cb3f52aae763473cb09593496219dea8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:58.020697  4045 consensus_queue.cc:1048] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:58.029444  3633 raft_consensus.cc:1275] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Refusing update from remote peer cb3f52aae763473cb09593496219dea8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:12:58.031015  4084 consensus_queue.cc:1048] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:12:58.170416  3706 raft_consensus.cc:1275] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 3 FOLLOWER]: Refusing update from remote peer cb3f52aae763473cb09593496219dea8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:12:58.171680  4084 consensus_queue.cc:1048] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2462c58306174ec1a384ea49fddf64c1" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:12:58.189869  3634 raft_consensus.cc:1275] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 3 FOLLOWER]: Refusing update from remote peer cb3f52aae763473cb09593496219dea8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:12:58.191430  4084 consensus_queue.cc:1048] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31750ab24a654ede8e5d60d6ab45f76f" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:12:58.335273 24113 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 13:12:58.342732 24113 auto_leader_rebalancer.cc:452] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 13:12:58.343245 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:12:58.348356 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 13:12:58.348743 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:12:59.376577 24113 tablet_server.cc:179] TabletServer@127.23.140.65:0 shutting down...
I20260709 13:12:59.408147 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:12:59.408900 24113 tablet_replica.cc:333] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f: stopping tablet replica
I20260709 13:12:59.409775 24113 raft_consensus.cc:2243] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.410305 24113 raft_consensus.cc:2272] T 49efa8eb906145db89256ddc020530a3 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.412581 24113 tablet_replica.cc:333] T b0d21b4836594593ac3f578d56f91e4a P 31750ab24a654ede8e5d60d6ab45f76f: stopping tablet replica
I20260709 13:12:59.413375 24113 raft_consensus.cc:2243] T b0d21b4836594593ac3f578d56f91e4a P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.413834 24113 raft_consensus.cc:2272] T b0d21b4836594593ac3f578d56f91e4a P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.415728 24113 tablet_replica.cc:333] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f: stopping tablet replica
I20260709 13:12:59.416297 24113 raft_consensus.cc:2243] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.416697 24113 raft_consensus.cc:2272] T 9fdc51f4bd024752b1b8a75cd782a03e P 31750ab24a654ede8e5d60d6ab45f76f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.418663 24113 tablet_replica.cc:333] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f: stopping tablet replica
I20260709 13:12:59.419188 24113 raft_consensus.cc:2243] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.419965 24113 raft_consensus.cc:2272] T 2fe236a52db543c18530f238587d8346 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.422518 24113 tablet_replica.cc:333] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f: stopping tablet replica
I20260709 13:12:59.423369 24113 raft_consensus.cc:2243] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.423859 24113 raft_consensus.cc:2272] T b2c195102cee49d39fe701c20ac7dacf P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.425916 24113 tablet_replica.cc:333] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f: stopping tablet replica
I20260709 13:12:59.426419 24113 raft_consensus.cc:2243] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.426836 24113 raft_consensus.cc:2272] T d34d16630b304796b7cb8c4de4f0c3a8 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.428596 24113 tablet_replica.cc:333] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f: stopping tablet replica
I20260709 13:12:59.429081 24113 raft_consensus.cc:2243] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.429960 24113 raft_consensus.cc:2272] T 605b25b7889545a4b965c755cbead46a P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.431816 24113 tablet_replica.cc:333] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f: stopping tablet replica
I20260709 13:12:59.432304 24113 raft_consensus.cc:2243] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.432686 24113 raft_consensus.cc:2272] T 57720afe068341ab8da51d15b129d2d6 P 31750ab24a654ede8e5d60d6ab45f76f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.434970 24113 tablet_replica.cc:333] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f: stopping tablet replica
I20260709 13:12:59.435601 24113 raft_consensus.cc:2243] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.436095 24113 raft_consensus.cc:2272] T 17679967d4734e678c707f380844b93c P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.438724 24113 tablet_replica.cc:333] T 1ccc13de4df944b1a6695b0f05c83ada P 31750ab24a654ede8e5d60d6ab45f76f: stopping tablet replica
I20260709 13:12:59.439299 24113 raft_consensus.cc:2243] T 1ccc13de4df944b1a6695b0f05c83ada P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.439785 24113 raft_consensus.cc:2272] T 1ccc13de4df944b1a6695b0f05c83ada P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.441883 24113 tablet_replica.cc:333] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f: stopping tablet replica
I20260709 13:12:59.442418 24113 raft_consensus.cc:2243] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.442898 24113 raft_consensus.cc:2272] T 20eb2e33130740e38d072182154c715f P 31750ab24a654ede8e5d60d6ab45f76f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.445230 24113 tablet_replica.cc:333] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f: stopping tablet replica
I20260709 13:12:59.445778 24113 raft_consensus.cc:2243] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.446622 24113 raft_consensus.cc:2272] T dbcad4026f8440949bb13e46b88fc340 P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.448765 24113 tablet_replica.cc:333] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f: stopping tablet replica
I20260709 13:12:59.449512 24113 raft_consensus.cc:2243] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.450002 24113 raft_consensus.cc:2272] T 536c6c3f241342ec804d9400cbb23da9 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.452157 24113 tablet_replica.cc:333] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f: stopping tablet replica
I20260709 13:12:59.452796 24113 raft_consensus.cc:2243] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.453697 24113 raft_consensus.cc:2272] T ab14a07bccb94e4c82b34714c94ccb48 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.455968 24113 tablet_replica.cc:333] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f: stopping tablet replica
I20260709 13:12:59.456780 24113 raft_consensus.cc:2243] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.457262 24113 raft_consensus.cc:2272] T a5aed345cd424405a20d159a9f9762a0 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:59.457980  3675 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 8 similar messages]
I20260709 13:12:59.459843 24113 tablet_replica.cc:333] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f: stopping tablet replica
I20260709 13:12:59.460505 24113 raft_consensus.cc:2243] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.460997 24113 raft_consensus.cc:2272] T e37f28b3b48449d3adecff96febd5598 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:59.462162  3675 consensus_peers.cc:597] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 -> Peer 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049): Couldn't send request to peer 31750ab24a654ede8e5d60d6ab45f76f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:59.463407 24113 tablet_replica.cc:333] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f: stopping tablet replica
I20260709 13:12:59.464020 24113 raft_consensus.cc:2243] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.464807 24113 raft_consensus.cc:2272] T c96b46ac691e478f946baba4fc981c1b P 31750ab24a654ede8e5d60d6ab45f76f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.466943 24113 tablet_replica.cc:333] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f: stopping tablet replica
I20260709 13:12:59.467437 24113 raft_consensus.cc:2243] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.468214 24113 raft_consensus.cc:2272] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 31750ab24a654ede8e5d60d6ab45f76f [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:59.488571  3749 consensus_peers.cc:597] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 -> Peer 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049): Couldn't send request to peer 31750ab24a654ede8e5d60d6ab45f76f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:59.510514 24113 tablet_server.cc:196] TabletServer@127.23.140.65:0 shutdown complete.
I20260709 13:12:59.547343 24113 tablet_server.cc:179] TabletServer@127.23.140.66:0 shutting down...
W20260709 13:12:59.574565  3749 consensus_peers.cc:597] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 -> Peer 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049): Couldn't send request to peer 31750ab24a654ede8e5d60d6ab45f76f. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:40049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:12:59.575392  3749 consensus_peers.cc:597] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 -> Peer 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049): Couldn't send request to peer 31750ab24a654ede8e5d60d6ab45f76f. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:40049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:59.586197 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260709 13:12:59.586794  3675 consensus_peers.cc:597] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 -> Peer 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049): Couldn't send request to peer 31750ab24a654ede8e5d60d6ab45f76f. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:40049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:59.587069 24113 tablet_replica.cc:333] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1: stopping tablet replica
W20260709 13:12:59.587381  3675 consensus_peers.cc:597] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 -> Peer 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049): Couldn't send request to peer 31750ab24a654ede8e5d60d6ab45f76f. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:40049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:59.588049 24113 raft_consensus.cc:2243] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.588562 24113 raft_consensus.cc:2272] T 9fdc51f4bd024752b1b8a75cd782a03e P 2462c58306174ec1a384ea49fddf64c1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.590543 24113 tablet_replica.cc:333] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1: stopping tablet replica
I20260709 13:12:59.591183 24113 raft_consensus.cc:2243] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.591940 24113 raft_consensus.cc:2272] T b2c195102cee49d39fe701c20ac7dacf P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.594447 24113 tablet_replica.cc:333] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1: stopping tablet replica
I20260709 13:12:59.595140 24113 raft_consensus.cc:2243] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.595568 24113 raft_consensus.cc:2272] T b0d21b4836594593ac3f578d56f91e4a P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:59.597239  3749 consensus_peers.cc:597] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 -> Peer 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049): Couldn't send request to peer 31750ab24a654ede8e5d60d6ab45f76f. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:40049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:59.598006 24113 tablet_replica.cc:333] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1: stopping tablet replica
I20260709 13:12:59.598682 24113 raft_consensus.cc:2243] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.599124 24113 raft_consensus.cc:2272] T 57720afe068341ab8da51d15b129d2d6 P 2462c58306174ec1a384ea49fddf64c1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.601014 24113 tablet_replica.cc:333] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1: stopping tablet replica
I20260709 13:12:59.601593 24113 raft_consensus.cc:2243] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.602030 24113 raft_consensus.cc:2272] T 605b25b7889545a4b965c755cbead46a P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.603930 24113 tablet_replica.cc:333] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1: stopping tablet replica
I20260709 13:12:59.604565 24113 raft_consensus.cc:2243] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.605381 24113 raft_consensus.cc:2272] T 49efa8eb906145db89256ddc020530a3 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.607314 24113 tablet_replica.cc:333] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1: stopping tablet replica
I20260709 13:12:59.607874 24113 raft_consensus.cc:2243] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.608620 24113 raft_consensus.cc:2272] T d34d16630b304796b7cb8c4de4f0c3a8 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.611413 24113 tablet_replica.cc:333] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1: stopping tablet replica
I20260709 13:12:59.612272 24113 raft_consensus.cc:2243] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.612819 24113 raft_consensus.cc:2272] T 2fe236a52db543c18530f238587d8346 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:59.612860  3675 consensus_peers.cc:597] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 -> Peer 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049): Couldn't send request to peer 31750ab24a654ede8e5d60d6ab45f76f. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:40049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:59.615363 24113 tablet_replica.cc:333] T 17679967d4734e678c707f380844b93c P 2462c58306174ec1a384ea49fddf64c1: stopping tablet replica
I20260709 13:12:59.615963 24113 raft_consensus.cc:2243] T 17679967d4734e678c707f380844b93c P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.616439 24113 raft_consensus.cc:2272] T 17679967d4734e678c707f380844b93c P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.618614 24113 tablet_replica.cc:333] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1: stopping tablet replica
I20260709 13:12:59.619167 24113 raft_consensus.cc:2243] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.619968 24113 raft_consensus.cc:2272] T 1ccc13de4df944b1a6695b0f05c83ada P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.622064 24113 tablet_replica.cc:333] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1: stopping tablet replica
I20260709 13:12:59.622608 24113 raft_consensus.cc:2243] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.623015 24113 raft_consensus.cc:2272] T 20eb2e33130740e38d072182154c715f P 2462c58306174ec1a384ea49fddf64c1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.625101 24113 tablet_replica.cc:333] T c96b46ac691e478f946baba4fc981c1b P 2462c58306174ec1a384ea49fddf64c1: stopping tablet replica
I20260709 13:12:59.625684 24113 raft_consensus.cc:2243] T c96b46ac691e478f946baba4fc981c1b P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.626089 24113 raft_consensus.cc:2272] T c96b46ac691e478f946baba4fc981c1b P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.628486 24113 tablet_replica.cc:333] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1: stopping tablet replica
I20260709 13:12:59.629112 24113 raft_consensus.cc:2243] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.629647 24113 raft_consensus.cc:2272] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:12:59.631206  3675 consensus_peers.cc:597] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 -> Peer 31750ab24a654ede8e5d60d6ab45f76f (127.23.140.65:40049): Couldn't send request to peer 31750ab24a654ede8e5d60d6ab45f76f. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:40049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:12:59.632238 24113 tablet_replica.cc:333] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1: stopping tablet replica
I20260709 13:12:59.632794 24113 raft_consensus.cc:2243] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.633587 24113 raft_consensus.cc:2272] T e37f28b3b48449d3adecff96febd5598 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.635542 24113 tablet_replica.cc:333] T ab14a07bccb94e4c82b34714c94ccb48 P 2462c58306174ec1a384ea49fddf64c1: stopping tablet replica
I20260709 13:12:59.636014 24113 raft_consensus.cc:2243] T ab14a07bccb94e4c82b34714c94ccb48 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.636390 24113 raft_consensus.cc:2272] T ab14a07bccb94e4c82b34714c94ccb48 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.638477 24113 tablet_replica.cc:333] T a5aed345cd424405a20d159a9f9762a0 P 2462c58306174ec1a384ea49fddf64c1: stopping tablet replica
I20260709 13:12:59.638919 24113 raft_consensus.cc:2243] T a5aed345cd424405a20d159a9f9762a0 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.639277 24113 raft_consensus.cc:2272] T a5aed345cd424405a20d159a9f9762a0 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.641291 24113 tablet_replica.cc:333] T dbcad4026f8440949bb13e46b88fc340 P 2462c58306174ec1a384ea49fddf64c1: stopping tablet replica
I20260709 13:12:59.641710 24113 raft_consensus.cc:2243] T dbcad4026f8440949bb13e46b88fc340 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.642069 24113 raft_consensus.cc:2272] T dbcad4026f8440949bb13e46b88fc340 P 2462c58306174ec1a384ea49fddf64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.643994 24113 tablet_replica.cc:333] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1: stopping tablet replica
I20260709 13:12:59.644506 24113 raft_consensus.cc:2243] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.645346 24113 raft_consensus.cc:2272] T 536c6c3f241342ec804d9400cbb23da9 P 2462c58306174ec1a384ea49fddf64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.684669 24113 tablet_server.cc:196] TabletServer@127.23.140.66:0 shutdown complete.
I20260709 13:12:59.718613 24113 tablet_server.cc:179] TabletServer@127.23.140.67:0 shutting down...
I20260709 13:12:59.743310 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:12:59.743975 24113 tablet_replica.cc:333] T b2c195102cee49d39fe701c20ac7dacf P cb3f52aae763473cb09593496219dea8: stopping tablet replica
I20260709 13:12:59.744835 24113 raft_consensus.cc:2243] T b2c195102cee49d39fe701c20ac7dacf P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.745358 24113 raft_consensus.cc:2272] T b2c195102cee49d39fe701c20ac7dacf P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.747406 24113 tablet_replica.cc:333] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8: stopping tablet replica
I20260709 13:12:59.748044 24113 raft_consensus.cc:2243] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.748836 24113 raft_consensus.cc:2272] T b0d21b4836594593ac3f578d56f91e4a P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.750870 24113 tablet_replica.cc:333] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8: stopping tablet replica
I20260709 13:12:59.751443 24113 raft_consensus.cc:2243] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.752173 24113 raft_consensus.cc:2272] T 57720afe068341ab8da51d15b129d2d6 P cb3f52aae763473cb09593496219dea8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.754348 24113 tablet_replica.cc:333] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8: stopping tablet replica
I20260709 13:12:59.754891 24113 raft_consensus.cc:2243] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.755631 24113 raft_consensus.cc:2272] T 9fdc51f4bd024752b1b8a75cd782a03e P cb3f52aae763473cb09593496219dea8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.757537 24113 tablet_replica.cc:333] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8: stopping tablet replica
I20260709 13:12:59.758077 24113 raft_consensus.cc:2243] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.758525 24113 raft_consensus.cc:2272] T d34d16630b304796b7cb8c4de4f0c3a8 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.760378 24113 tablet_replica.cc:333] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8: stopping tablet replica
I20260709 13:12:59.760946 24113 raft_consensus.cc:2243] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.761428 24113 raft_consensus.cc:2272] T 2fe236a52db543c18530f238587d8346 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.763223 24113 tablet_replica.cc:333] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8: stopping tablet replica
I20260709 13:12:59.763765 24113 raft_consensus.cc:2243] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.764180 24113 raft_consensus.cc:2272] T 605b25b7889545a4b965c755cbead46a P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.766266 24113 tablet_replica.cc:333] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8: stopping tablet replica
I20260709 13:12:59.766805 24113 raft_consensus.cc:2243] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.767241 24113 raft_consensus.cc:2272] T 1ccc13de4df944b1a6695b0f05c83ada P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.769150 24113 tablet_replica.cc:333] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8: stopping tablet replica
I20260709 13:12:59.769654 24113 raft_consensus.cc:2243] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.770413 24113 raft_consensus.cc:2272] T 20eb2e33130740e38d072182154c715f P cb3f52aae763473cb09593496219dea8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.772401 24113 tablet_replica.cc:333] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8: stopping tablet replica
I20260709 13:12:59.772948 24113 raft_consensus.cc:2243] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.773729 24113 raft_consensus.cc:2272] T 17679967d4734e678c707f380844b93c P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.775709 24113 tablet_replica.cc:333] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8: stopping tablet replica
I20260709 13:12:59.776221 24113 raft_consensus.cc:2243] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.776945 24113 raft_consensus.cc:2272] T a5aed345cd424405a20d159a9f9762a0 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.778795 24113 tablet_replica.cc:333] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8: stopping tablet replica
I20260709 13:12:59.779323 24113 raft_consensus.cc:2243] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.779729 24113 raft_consensus.cc:2272] T e37f28b3b48449d3adecff96febd5598 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.781693 24113 tablet_replica.cc:333] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8: stopping tablet replica
I20260709 13:12:59.782222 24113 raft_consensus.cc:2243] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.782683 24113 raft_consensus.cc:2272] T ab14a07bccb94e4c82b34714c94ccb48 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.784647 24113 tablet_replica.cc:333] T c96b46ac691e478f946baba4fc981c1b P cb3f52aae763473cb09593496219dea8: stopping tablet replica
I20260709 13:12:59.785151 24113 raft_consensus.cc:2243] T c96b46ac691e478f946baba4fc981c1b P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.785554 24113 raft_consensus.cc:2272] T c96b46ac691e478f946baba4fc981c1b P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.787467 24113 tablet_replica.cc:333] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8: stopping tablet replica
I20260709 13:12:59.787969 24113 raft_consensus.cc:2243] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.788432 24113 raft_consensus.cc:2272] T 98f4adc13b1f41e0a6df7506b3f2b9a5 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.790331 24113 tablet_replica.cc:333] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8: stopping tablet replica
I20260709 13:12:59.790832 24113 raft_consensus.cc:2243] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.791254 24113 raft_consensus.cc:2272] T 49efa8eb906145db89256ddc020530a3 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.795759 24113 tablet_replica.cc:333] T dbcad4026f8440949bb13e46b88fc340 P cb3f52aae763473cb09593496219dea8: stopping tablet replica
I20260709 13:12:59.796224 24113 raft_consensus.cc:2243] T dbcad4026f8440949bb13e46b88fc340 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.796646 24113 raft_consensus.cc:2272] T dbcad4026f8440949bb13e46b88fc340 P cb3f52aae763473cb09593496219dea8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.798408 24113 tablet_replica.cc:333] T 536c6c3f241342ec804d9400cbb23da9 P cb3f52aae763473cb09593496219dea8: stopping tablet replica
I20260709 13:12:59.798877 24113 raft_consensus.cc:2243] T 536c6c3f241342ec804d9400cbb23da9 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:12:59.799311 24113 raft_consensus.cc:2272] T 536c6c3f241342ec804d9400cbb23da9 P cb3f52aae763473cb09593496219dea8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.840654 24113 tablet_server.cc:196] TabletServer@127.23.140.67:0 shutdown complete.
I20260709 13:12:59.875547 24113 master.cc:562] Master@127.23.140.126:45465 shutting down...
I20260709 13:12:59.896624 24113 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:12:59.897174 24113 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:12:59.897506 24113 tablet_replica.cc:333] T 00000000000000000000000000000000 P 38fc5367513e426fa22096b7157fa610: stopping tablet replica
I20260709 13:12:59.917744 24113 master.cc:584] Master@127.23.140.126:45465 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0 (12443 ms)
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1
I20260709 13:12:59.970743 24113 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.140.126:33139
I20260709 13:12:59.971843 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:12:59.977820  4103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:59.978231  4105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:12:59.978248  4102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:12:59.978842 24113 server_base.cc:1061] running on GCE node
I20260709 13:12:59.979756 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:12:59.979964 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:12:59.980124 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602779980109 us; error 0 us; skew 500 ppm
I20260709 13:12:59.980633 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:12:59.987829 24113 webserver.cc:533] Webserver started at http://127.23.140.126:38947/ using document root <none> and password file <none>
I20260709 13:12:59.988343 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:12:59.988541 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:12:59.988816 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:12:59.989912 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/master-0-root/instance:
uuid: "2a1a5c74cdb24785b9c419554d8589b4"
format_stamp: "Formatted at 2026-07-09 13:12:59 on dist-test-slave-k73n"
I20260709 13:12:59.994122 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 13:12:59.997221  4110 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:12:59.997946 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 13:12:59.998230 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/master-0-root
uuid: "2a1a5c74cdb24785b9c419554d8589b4"
format_stamp: "Formatted at 2026-07-09 13:12:59 on dist-test-slave-k73n"
I20260709 13:12:59.998512 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:13:00.019639 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:13:00.020829 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:13:00.059103 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.126:33139
I20260709 13:13:00.059238  4161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.126:33139 every 8 connection(s)
I20260709 13:13:00.063050  4162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.074538  4162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4: Bootstrap starting.
I20260709 13:13:00.079646  4162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:00.084121  4162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4: No bootstrap required, opened a new log
I20260709 13:13:00.086448  4162 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1a5c74cdb24785b9c419554d8589b4" member_type: VOTER }
I20260709 13:13:00.086828  4162 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:00.087103  4162 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1a5c74cdb24785b9c419554d8589b4, State: Initialized, Role: FOLLOWER
I20260709 13:13:00.087818  4162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1a5c74cdb24785b9c419554d8589b4" member_type: VOTER }
I20260709 13:13:00.088301  4162 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:13:00.088553  4162 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:13:00.088855  4162 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:00.093902  4162 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1a5c74cdb24785b9c419554d8589b4" member_type: VOTER }
I20260709 13:13:00.094460  4162 leader_election.cc:304] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a1a5c74cdb24785b9c419554d8589b4; no voters: 
I20260709 13:13:00.095700  4162 leader_election.cc:290] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:13:00.096006  4165 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:00.097486  4165 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [term 1 LEADER]: Becoming Leader. State: Replica: 2a1a5c74cdb24785b9c419554d8589b4, State: Running, Role: LEADER
I20260709 13:13:00.098203  4165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1a5c74cdb24785b9c419554d8589b4" member_type: VOTER }
I20260709 13:13:00.098968  4162 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:13:00.102677  4167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a1a5c74cdb24785b9c419554d8589b4. Latest consensus state: current_term: 1 leader_uuid: "2a1a5c74cdb24785b9c419554d8589b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1a5c74cdb24785b9c419554d8589b4" member_type: VOTER } }
I20260709 13:13:00.103327  4167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [sys.catalog]: This master's current role is: LEADER
I20260709 13:13:00.103621  4166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a1a5c74cdb24785b9c419554d8589b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1a5c74cdb24785b9c419554d8589b4" member_type: VOTER } }
I20260709 13:13:00.104444  4166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [sys.catalog]: This master's current role is: LEADER
I20260709 13:13:00.108211  4172 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:13:00.114176  4172 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:13:00.115185 24113 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:13:00.123000  4172 catalog_manager.cc:1383] Generated new cluster ID: 47ab33625a554788be7b51639fbf483d
I20260709 13:13:00.123270  4172 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:13:00.160029  4172 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:13:00.162451  4172 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:13:00.189284  4172 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4: Generated new TSK 0
I20260709 13:13:00.190093  4172 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:13:00.246670 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:13:00.252741  4183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:13:00.253968  4184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:13:00.255959 24113 server_base.cc:1061] running on GCE node
W20260709 13:13:00.256417  4186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:13:00.257390 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:13:00.257622 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:13:00.257789 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602780257767 us; error 0 us; skew 500 ppm
I20260709 13:13:00.258278 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:13:00.260478 24113 webserver.cc:533] Webserver started at http://127.23.140.65:42561/ using document root <none> and password file <none>
I20260709 13:13:00.260946 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:13:00.261197 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:13:00.261461 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:13:00.262491 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/ts-0-root/instance:
uuid: "ba722a02e5c34752b5a164e6234797ee"
format_stamp: "Formatted at 2026-07-09 13:13:00 on dist-test-slave-k73n"
I20260709 13:13:00.266801 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.001s
I20260709 13:13:00.269964  4191 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:13:00.270870 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 13:13:00.271118 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/ts-0-root
uuid: "ba722a02e5c34752b5a164e6234797ee"
format_stamp: "Formatted at 2026-07-09 13:13:00 on dist-test-slave-k73n"
I20260709 13:13:00.271347 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:13:00.289243 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:13:00.290362 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:13:00.291731 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:13:00.293906 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:13:00.294098 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:13:00.294348 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:13:00.294519 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:13:00.333312 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.65:40371
I20260709 13:13:00.333376  4253 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.65:40371 every 8 connection(s)
I20260709 13:13:00.338452 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:13:00.345199  4258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:13:00.346273  4259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:13:00.353196  4254 heartbeater.cc:344] Connected to a master server at 127.23.140.126:33139
I20260709 13:13:00.353930  4254 heartbeater.cc:461] Registering TS with master...
W20260709 13:13:00.355525  4262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:13:00.355763  4254 heartbeater.cc:507] Master 127.23.140.126:33139 requested a full tablet report, sending...
I20260709 13:13:00.356238 24113 server_base.cc:1061] running on GCE node
I20260709 13:13:00.359398  4127 ts_manager.cc:194] Registered new tserver with Master: ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371)
I20260709 13:13:00.361037 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:13:00.361363 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:13:00.361563 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602780361547 us; error 0 us; skew 500 ppm
I20260709 13:13:00.362169  4127 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38556
I20260709 13:13:00.362308 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:13:00.365017 24113 webserver.cc:533] Webserver started at http://127.23.140.66:41741/ using document root <none> and password file <none>
I20260709 13:13:00.365525 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:13:00.365677 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:13:00.365877 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:13:00.366942 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/ts-1-root/instance:
uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
format_stamp: "Formatted at 2026-07-09 13:13:00 on dist-test-slave-k73n"
I20260709 13:13:00.371183 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 13:13:00.374325  4266 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:13:00.375057 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 13:13:00.375340 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/ts-1-root
uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
format_stamp: "Formatted at 2026-07-09 13:13:00 on dist-test-slave-k73n"
I20260709 13:13:00.375602 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:13:00.394917 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:13:00.396083 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:13:00.397558 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:13:00.399675 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:13:00.399874 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:13:00.400199 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:13:00.400357 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:13:00.439829 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.66:42079
I20260709 13:13:00.439939  4328 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.66:42079 every 8 connection(s)
I20260709 13:13:00.444710 24113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:13:00.452853  4332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:13:00.453938  4333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:13:00.456951  4329 heartbeater.cc:344] Connected to a master server at 127.23.140.126:33139
I20260709 13:13:00.457181 24113 server_base.cc:1061] running on GCE node
I20260709 13:13:00.457584  4329 heartbeater.cc:461] Registering TS with master...
W20260709 13:13:00.457572  4335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:13:00.458760  4329 heartbeater.cc:507] Master 127.23.140.126:33139 requested a full tablet report, sending...
I20260709 13:13:00.458786 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:13:00.459178 24113 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:13:00.459360 24113 hybrid_clock.cc:648] HybridClock initialized: now 1783602780459339 us; error 0 us; skew 500 ppm
I20260709 13:13:00.460000 24113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:13:00.461015  4127 ts_manager.cc:194] Registered new tserver with Master: 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079)
I20260709 13:13:00.462734 24113 webserver.cc:533] Webserver started at http://127.23.140.67:43985/ using document root <none> and password file <none>
I20260709 13:13:00.462803  4127 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38560
I20260709 13:13:00.463342 24113 fs_manager.cc:362] Metadata directory not provided
I20260709 13:13:00.463565 24113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:13:00.463896 24113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:13:00.465539 24113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/ts-2-root/instance:
uuid: "2a925808f54d42ed9cc086f846280594"
format_stamp: "Formatted at 2026-07-09 13:13:00 on dist-test-slave-k73n"
I20260709 13:13:00.469985 24113 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 13:13:00.473322  4340 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:13:00.474094 24113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260709 13:13:00.474352 24113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/ts-2-root
uuid: "2a925808f54d42ed9cc086f846280594"
format_stamp: "Formatted at 2026-07-09 13:13:00 on dist-test-slave-k73n"
I20260709 13:13:00.474584 24113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQz_59X/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602412999163-24113-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:13:00.486320 24113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:13:00.487401 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:13:00.488731 24113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:13:00.490944 24113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:13:00.491132 24113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:13:00.491420 24113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:13:00.491576 24113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:13:00.531569 24113 rpc_server.cc:307] RPC server started. Bound to: 127.23.140.67:37585
I20260709 13:13:00.531641  4402 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.140.67:37585 every 8 connection(s)
I20260709 13:13:00.544620  4403 heartbeater.cc:344] Connected to a master server at 127.23.140.126:33139
I20260709 13:13:00.545074  4403 heartbeater.cc:461] Registering TS with master...
I20260709 13:13:00.545816  4403 heartbeater.cc:507] Master 127.23.140.126:33139 requested a full tablet report, sending...
I20260709 13:13:00.547668  4127 ts_manager.cc:194] Registered new tserver with Master: 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:00.548085 24113 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013368802s
I20260709 13:13:00.549016  4127 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38566
I20260709 13:13:00.573897  4127 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38578:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:13:00.576768  4127 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:13:00.645190  4219 tablet_service.cc:1511] Processing CreateTablet for tablet ba55bc7ffc73446da8111405c7a7b7f0 (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:13:00.646611  4219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba55bc7ffc73446da8111405c7a7b7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.650256  4218 tablet_service.cc:1511] Processing CreateTablet for tablet 2363b6da78da46a2a8435e57a9732b3d (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:13:00.652789  4218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2363b6da78da46a2a8435e57a9732b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.656765  4217 tablet_service.cc:1511] Processing CreateTablet for tablet ee8997813dd7430ca5f9d7269b2d0d5e (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:13:00.658254  4217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee8997813dd7430ca5f9d7269b2d0d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.662284  4216 tablet_service.cc:1511] Processing CreateTablet for tablet c0e4091b5254449b9ece5ccdccee16ee (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:13:00.663638  4216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0e4091b5254449b9ece5ccdccee16ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.666289  4215 tablet_service.cc:1511] Processing CreateTablet for tablet 835876e7bd174f4184fd6294093fa911 (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:13:00.667663  4215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 835876e7bd174f4184fd6294093fa911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.670311  4214 tablet_service.cc:1511] Processing CreateTablet for tablet 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:13:00.671736  4214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a8b2f9fd9bf4cd2a4ec339ca2e08d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.674302  4213 tablet_service.cc:1511] Processing CreateTablet for tablet 02b49437a48440b7990bb6f87b03b6af (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:13:00.675736  4213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02b49437a48440b7990bb6f87b03b6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.677361  4418 tablet_bootstrap.cc:492] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee: Bootstrap starting.
I20260709 13:13:00.677637  4219 tablet_service.cc:1511] Processing CreateTablet for tablet fb2b800af71b4a95b06e180939b9cae9 (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:13:00.679076  4219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb2b800af71b4a95b06e180939b9cae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.682796  4218 tablet_service.cc:1511] Processing CreateTablet for tablet 8f1d29a41a99463fa23d6308fc2afaaa (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:13:00.684131  4218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1d29a41a99463fa23d6308fc2afaaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.691151  4418 tablet_bootstrap.cc:654] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:00.702239  4294 tablet_service.cc:1511] Processing CreateTablet for tablet ba55bc7ffc73446da8111405c7a7b7f0 (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:13:00.703680  4294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba55bc7ffc73446da8111405c7a7b7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.706374  4293 tablet_service.cc:1511] Processing CreateTablet for tablet 2363b6da78da46a2a8435e57a9732b3d (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:13:00.707780  4293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2363b6da78da46a2a8435e57a9732b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.710304  4292 tablet_service.cc:1511] Processing CreateTablet for tablet ee8997813dd7430ca5f9d7269b2d0d5e (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:13:00.711720  4292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee8997813dd7430ca5f9d7269b2d0d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.714344  4291 tablet_service.cc:1511] Processing CreateTablet for tablet c0e4091b5254449b9ece5ccdccee16ee (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:13:00.715737  4291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0e4091b5254449b9ece5ccdccee16ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.718267  4290 tablet_service.cc:1511] Processing CreateTablet for tablet 835876e7bd174f4184fd6294093fa911 (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:13:00.719635  4290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 835876e7bd174f4184fd6294093fa911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.722609  4288 tablet_service.cc:1511] Processing CreateTablet for tablet 02b49437a48440b7990bb6f87b03b6af (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:13:00.724375  4288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02b49437a48440b7990bb6f87b03b6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.726295  4289 tablet_service.cc:1511] Processing CreateTablet for tablet 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:13:00.727586  4289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a8b2f9fd9bf4cd2a4ec339ca2e08d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.720353  4287 tablet_service.cc:1511] Processing CreateTablet for tablet fb2b800af71b4a95b06e180939b9cae9 (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:13:00.730088  4287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb2b800af71b4a95b06e180939b9cae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.734495  4286 tablet_service.cc:1511] Processing CreateTablet for tablet 8f1d29a41a99463fa23d6308fc2afaaa (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:13:00.735770  4286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1d29a41a99463fa23d6308fc2afaaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.754024  4420 tablet_bootstrap.cc:492] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c: Bootstrap starting.
I20260709 13:13:00.758900  4418 tablet_bootstrap.cc:492] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee: No bootstrap required, opened a new log
I20260709 13:13:00.759500  4418 ts_tablet_manager.cc:1403] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee: Time spent bootstrapping tablet: real 0.083s	user 0.021s	sys 0.040s
I20260709 13:13:00.760771  4420 tablet_bootstrap.cc:654] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:00.771422  4368 tablet_service.cc:1511] Processing CreateTablet for tablet ba55bc7ffc73446da8111405c7a7b7f0 (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:13:00.772924  4368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba55bc7ffc73446da8111405c7a7b7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.773166  4367 tablet_service.cc:1511] Processing CreateTablet for tablet 2363b6da78da46a2a8435e57a9732b3d (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:13:00.774562  4367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2363b6da78da46a2a8435e57a9732b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.778285  4366 tablet_service.cc:1511] Processing CreateTablet for tablet ee8997813dd7430ca5f9d7269b2d0d5e (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:13:00.779675  4366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee8997813dd7430ca5f9d7269b2d0d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.782362  4365 tablet_service.cc:1511] Processing CreateTablet for tablet c0e4091b5254449b9ece5ccdccee16ee (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:13:00.784016  4365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0e4091b5254449b9ece5ccdccee16ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.786304  4364 tablet_service.cc:1511] Processing CreateTablet for tablet 835876e7bd174f4184fd6294093fa911 (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:13:00.788028  4364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 835876e7bd174f4184fd6294093fa911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.790781  4363 tablet_service.cc:1511] Processing CreateTablet for tablet 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:13:00.792493  4363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a8b2f9fd9bf4cd2a4ec339ca2e08d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.810737  4418 raft_consensus.cc:359] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:00.811887  4418 raft_consensus.cc:385] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:00.812261  4418 raft_consensus.cc:740] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Initialized, Role: FOLLOWER
I20260709 13:13:00.813218  4418 consensus_queue.cc:260] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:00.813607  4362 tablet_service.cc:1511] Processing CreateTablet for tablet 02b49437a48440b7990bb6f87b03b6af (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:13:00.816334  4362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02b49437a48440b7990bb6f87b03b6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.822438  4361 tablet_service.cc:1511] Processing CreateTablet for tablet fb2b800af71b4a95b06e180939b9cae9 (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:13:00.823911  4361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb2b800af71b4a95b06e180939b9cae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.826861  4360 tablet_service.cc:1511] Processing CreateTablet for tablet 8f1d29a41a99463fa23d6308fc2afaaa (DEFAULT_TABLE table=test-workload [id=be7c0a68e7e24eab83e94017c05d0a38]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:13:00.828233  4360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1d29a41a99463fa23d6308fc2afaaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:00.862368  4422 raft_consensus.cc:493] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:00.862394  4254 heartbeater.cc:499] Master 127.23.140.126:33139 was elected leader, sending a full tablet report...
I20260709 13:13:00.863443  4422 raft_consensus.cc:515] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:00.867003  4422 leader_election.cc:290] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:00.884832  4418 ts_tablet_manager.cc:1434] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee: Time spent starting tablet: real 0.125s	user 0.020s	sys 0.014s
I20260709 13:13:00.886083  4418 tablet_bootstrap.cc:492] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee: Bootstrap starting.
I20260709 13:13:00.903090  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba55bc7ffc73446da8111405c7a7b7f0" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
W20260709 13:13:00.905143  4192 leader_election.cc:343] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:00.905719  4420 tablet_bootstrap.cc:492] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c: No bootstrap required, opened a new log
I20260709 13:13:00.906308  4420 ts_tablet_manager.cc:1403] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent bootstrapping tablet: real 0.153s	user 0.054s	sys 0.035s
I20260709 13:13:00.913055  4425 tablet_bootstrap.cc:492] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594: Bootstrap starting.
I20260709 13:13:00.922773  4418 tablet_bootstrap.cc:654] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:00.925705  4420 raft_consensus.cc:359] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:00.926827  4420 raft_consensus.cc:385] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:00.927217  4420 raft_consensus.cc:740] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Initialized, Role: FOLLOWER
I20260709 13:13:00.928284  4420 consensus_queue.cc:260] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:00.931345  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba55bc7ffc73446da8111405c7a7b7f0" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
W20260709 13:13:00.933053  4194 leader_election.cc:343] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:00.933326  4425 tablet_bootstrap.cc:654] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:00.933636  4194 leader_election.cc:304] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba722a02e5c34752b5a164e6234797ee; no voters: 2a925808f54d42ed9cc086f846280594, 6ff6192f285c425fb9d28bac1f0cea7c
I20260709 13:13:00.934610  4423 raft_consensus.cc:2749] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:13:00.936970  4329 heartbeater.cc:499] Master 127.23.140.126:33139 was elected leader, sending a full tablet report...
I20260709 13:13:00.937526  4420 ts_tablet_manager.cc:1434] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent starting tablet: real 0.031s	user 0.010s	sys 0.001s
I20260709 13:13:00.938925  4420 tablet_bootstrap.cc:492] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c: Bootstrap starting.
I20260709 13:13:00.943539  4418 tablet_bootstrap.cc:492] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee: No bootstrap required, opened a new log
I20260709 13:13:00.944796  4425 tablet_bootstrap.cc:492] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594: No bootstrap required, opened a new log
I20260709 13:13:00.944898  4418 ts_tablet_manager.cc:1403] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee: Time spent bootstrapping tablet: real 0.059s	user 0.006s	sys 0.009s
I20260709 13:13:00.945413  4425 ts_tablet_manager.cc:1403] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594: Time spent bootstrapping tablet: real 0.033s	user 0.011s	sys 0.006s
I20260709 13:13:00.945999  4420 tablet_bootstrap.cc:654] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:00.948345  4418 raft_consensus.cc:359] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:00.948637  4425 raft_consensus.cc:359] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:00.949316  4418 raft_consensus.cc:385] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:00.949518  4425 raft_consensus.cc:385] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:00.949848  4418 raft_consensus.cc:740] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Initialized, Role: FOLLOWER
I20260709 13:13:00.949950  4425 raft_consensus.cc:740] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Initialized, Role: FOLLOWER
I20260709 13:13:00.951375  4425 consensus_queue.cc:260] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:00.951215  4418 consensus_queue.cc:260] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:00.954500  4418 ts_tablet_manager.cc:1434] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee: Time spent starting tablet: real 0.009s	user 0.001s	sys 0.006s
I20260709 13:13:00.956291  4418 tablet_bootstrap.cc:492] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee: Bootstrap starting.
I20260709 13:13:00.959107  4420 tablet_bootstrap.cc:492] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c: No bootstrap required, opened a new log
I20260709 13:13:00.959791  4420 ts_tablet_manager.cc:1403] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent bootstrapping tablet: real 0.021s	user 0.010s	sys 0.008s
I20260709 13:13:00.962968  4418 tablet_bootstrap.cc:654] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:00.963215  4420 raft_consensus.cc:359] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:00.964077  4420 raft_consensus.cc:385] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:00.964447  4420 raft_consensus.cc:740] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Initialized, Role: FOLLOWER
I20260709 13:13:00.965471  4420 consensus_queue.cc:260] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:00.968140  4420 ts_tablet_manager.cc:1434] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.001s
I20260709 13:13:00.968432  4403 heartbeater.cc:499] Master 127.23.140.126:33139 was elected leader, sending a full tablet report...
I20260709 13:13:00.969213  4420 tablet_bootstrap.cc:492] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c: Bootstrap starting.
I20260709 13:13:00.970026  4425 ts_tablet_manager.cc:1434] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.016s
I20260709 13:13:00.971189  4425 tablet_bootstrap.cc:492] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594: Bootstrap starting.
I20260709 13:13:00.977772  4425 tablet_bootstrap.cc:654] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:00.978296  4420 tablet_bootstrap.cc:654] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:00.981485  4418 tablet_bootstrap.cc:492] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee: No bootstrap required, opened a new log
I20260709 13:13:00.982141  4418 ts_tablet_manager.cc:1403] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee: Time spent bootstrapping tablet: real 0.026s	user 0.012s	sys 0.005s
I20260709 13:13:00.986063  4418 raft_consensus.cc:359] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:00.987033  4418 raft_consensus.cc:385] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:00.987401  4418 raft_consensus.cc:740] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Initialized, Role: FOLLOWER
I20260709 13:13:00.988225  4418 consensus_queue.cc:260] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:00.990792  4420 tablet_bootstrap.cc:492] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c: No bootstrap required, opened a new log
I20260709 13:13:00.991415  4420 ts_tablet_manager.cc:1403] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent bootstrapping tablet: real 0.022s	user 0.012s	sys 0.008s
I20260709 13:13:00.993413  4418 ts_tablet_manager.cc:1434] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.001s
I20260709 13:13:00.994590  4418 tablet_bootstrap.cc:492] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee: Bootstrap starting.
I20260709 13:13:00.994513  4420 raft_consensus.cc:359] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:00.995546  4420 raft_consensus.cc:385] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:00.995901  4420 raft_consensus.cc:740] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Initialized, Role: FOLLOWER
I20260709 13:13:00.996856  4420 consensus_queue.cc:260] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.001792  4420 ts_tablet_manager.cc:1434] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.001s
I20260709 13:13:01.002068  4418 tablet_bootstrap.cc:654] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:01.002903  4420 tablet_bootstrap.cc:492] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c: Bootstrap starting.
I20260709 13:13:01.011693  4420 tablet_bootstrap.cc:654] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:01.014789  4418 tablet_bootstrap.cc:492] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee: No bootstrap required, opened a new log
I20260709 13:13:01.015429  4418 ts_tablet_manager.cc:1403] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee: Time spent bootstrapping tablet: real 0.021s	user 0.013s	sys 0.005s
I20260709 13:13:01.018476  4418 raft_consensus.cc:359] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.019488  4418 raft_consensus.cc:385] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.019971  4418 raft_consensus.cc:740] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.020434  4425 tablet_bootstrap.cc:492] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594: No bootstrap required, opened a new log
I20260709 13:13:01.021071  4425 ts_tablet_manager.cc:1403] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594: Time spent bootstrapping tablet: real 0.050s	user 0.027s	sys 0.019s
I20260709 13:13:01.021178  4418 consensus_queue.cc:260] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.029106  4418 ts_tablet_manager.cc:1434] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 13:13:01.030746  4418 tablet_bootstrap.cc:492] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee: Bootstrap starting.
I20260709 13:13:01.030985  4425 raft_consensus.cc:359] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.031876  4427 raft_consensus.cc:493] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:01.031917  4425 raft_consensus.cc:385] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.032507  4425 raft_consensus.cc:740] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.032522  4427 raft_consensus.cc:515] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.033389  4425 consensus_queue.cc:260] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.038293  4427 leader_election.cc:290] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.038569  4418 tablet_bootstrap.cc:654] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:01.039294  4423 raft_consensus.cc:493] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:01.039958  4423 raft_consensus.cc:515] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.040942  4427 raft_consensus.cc:493] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:01.041467  4427 raft_consensus.cc:515] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.042395  4423 leader_election.cc:290] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.042409  4425 ts_tablet_manager.cc:1434] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 13:13:01.044057  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1d29a41a99463fa23d6308fc2afaaa" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:01.044646  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1d29a41a99463fa23d6308fc2afaaa" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
W20260709 13:13:01.045639  4194 leader_election.cc:343] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:13:01.046161  4192 leader_election.cc:343] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:01.046712  4192 leader_election.cc:304] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba722a02e5c34752b5a164e6234797ee; no voters: 2a925808f54d42ed9cc086f846280594, 6ff6192f285c425fb9d28bac1f0cea7c
I20260709 13:13:01.047174  4427 leader_election.cc:290] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.048537  4423 raft_consensus.cc:2749] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:13:01.052142  4425 tablet_bootstrap.cc:492] T 2363b6da78da46a2a8435e57a9732b3d P 2a925808f54d42ed9cc086f846280594: Bootstrap starting.
I20260709 13:13:01.052481  4420 tablet_bootstrap.cc:492] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c: No bootstrap required, opened a new log
I20260709 13:13:01.054677  4420 ts_tablet_manager.cc:1403] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent bootstrapping tablet: real 0.052s	user 0.007s	sys 0.019s
I20260709 13:13:01.059597  4425 tablet_bootstrap.cc:654] T 2363b6da78da46a2a8435e57a9732b3d P 2a925808f54d42ed9cc086f846280594: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:01.060902  4418 tablet_bootstrap.cc:492] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee: No bootstrap required, opened a new log
I20260709 13:13:01.061578  4418 ts_tablet_manager.cc:1403] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee: Time spent bootstrapping tablet: real 0.031s	user 0.008s	sys 0.007s
I20260709 13:13:01.063992  4420 raft_consensus.cc:359] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.064607  4418 raft_consensus.cc:359] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.065778  4418 raft_consensus.cc:385] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.066439  4418 raft_consensus.cc:740] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.065836  4420 raft_consensus.cc:385] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.068313  4420 raft_consensus.cc:740] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.068178  4418 consensus_queue.cc:260] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.068771  4228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0e4091b5254449b9ece5ccdccee16ee" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:01.068077  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a8b2f9fd9bf4cd2a4ec339ca2e08d03" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:01.070375  4425 tablet_bootstrap.cc:492] T 2363b6da78da46a2a8435e57a9732b3d P 2a925808f54d42ed9cc086f846280594: No bootstrap required, opened a new log
I20260709 13:13:01.070910  4425 ts_tablet_manager.cc:1403] T 2363b6da78da46a2a8435e57a9732b3d P 2a925808f54d42ed9cc086f846280594: Time spent bootstrapping tablet: real 0.019s	user 0.014s	sys 0.002s
W20260709 13:13:01.071374  4270 leader_election.cc:343] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:01.070822  4418 ts_tablet_manager.cc:1434] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.002s
W20260709 13:13:01.073158  4270 leader_election.cc:343] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:01.073766  4418 tablet_bootstrap.cc:492] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee: Bootstrap starting.
I20260709 13:13:01.075141  4425 raft_consensus.cc:359] T 2363b6da78da46a2a8435e57a9732b3d P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.075860  4425 raft_consensus.cc:385] T 2363b6da78da46a2a8435e57a9732b3d P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.076311  4425 raft_consensus.cc:740] T 2363b6da78da46a2a8435e57a9732b3d P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.077142  4425 consensus_queue.cc:260] T 2363b6da78da46a2a8435e57a9732b3d P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.079494  4420 consensus_queue.cc:260] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.080529  4418 tablet_bootstrap.cc:654] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:01.082050  4431 raft_consensus.cc:493] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:01.082556  4431 raft_consensus.cc:515] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.083865  4377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0e4091b5254449b9ece5ccdccee16ee" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:01.085361  4420 ts_tablet_manager.cc:1434] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent starting tablet: real 0.030s	user 0.007s	sys 0.001s
I20260709 13:13:01.082055  4423 raft_consensus.cc:493] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:01.087399  4423 raft_consensus.cc:515] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
W20260709 13:13:01.088256  4267 leader_election.cc:343] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:01.088745  4267 leader_election.cc:304] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c; no voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee
I20260709 13:13:01.089740  4423 leader_election.cc:290] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.090307  4427 raft_consensus.cc:2749] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:13:01.083880  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a8b2f9fd9bf4cd2a4ec339ca2e08d03" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:01.090977  4431 leader_election.cc:290] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079)
I20260709 13:13:01.091849  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb2b800af71b4a95b06e180939b9cae9" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
W20260709 13:13:01.093590  4194 leader_election.cc:343] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:01.095631  4420 tablet_bootstrap.cc:492] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c: Bootstrap starting.
W20260709 13:13:01.096815  4267 leader_election.cc:343] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:01.097299  4267 leader_election.cc:304] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c; no voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee
I20260709 13:13:01.086932  4418 tablet_bootstrap.cc:492] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee: No bootstrap required, opened a new log
I20260709 13:13:01.098062  4418 ts_tablet_manager.cc:1403] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee: Time spent bootstrapping tablet: real 0.025s	user 0.011s	sys 0.001s
I20260709 13:13:01.098479  4427 raft_consensus.cc:2749] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:13:01.097460  4425 ts_tablet_manager.cc:1434] T 2363b6da78da46a2a8435e57a9732b3d P 2a925808f54d42ed9cc086f846280594: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.001s
I20260709 13:13:01.102658  4420 tablet_bootstrap.cc:654] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:01.102660  4425 tablet_bootstrap.cc:492] T 835876e7bd174f4184fd6294093fa911 P 2a925808f54d42ed9cc086f846280594: Bootstrap starting.
W20260709 13:13:01.109192  4175 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:13:01.109205  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb2b800af71b4a95b06e180939b9cae9" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:01.110121  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee8997813dd7430ca5f9d7269b2d0d5e" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:01.112022  4425 tablet_bootstrap.cc:654] T 835876e7bd174f4184fd6294093fa911 P 2a925808f54d42ed9cc086f846280594: Neither blocks nor log segments found. Creating new log.
W20260709 13:13:01.113889  4192 leader_election.cc:343] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:01.114400  4192 leader_election.cc:304] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba722a02e5c34752b5a164e6234797ee; no voters: 2a925808f54d42ed9cc086f846280594, 6ff6192f285c425fb9d28bac1f0cea7c
W20260709 13:13:01.115485  4343 leader_election.cc:343] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:01.115890  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee8997813dd7430ca5f9d7269b2d0d5e" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:01.116634  4423 raft_consensus.cc:2749] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:13:01.117553  4344 leader_election.cc:343] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:01.118006  4344 leader_election.cc:304] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a925808f54d42ed9cc086f846280594; no voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee
I20260709 13:13:01.118510  4418 raft_consensus.cc:359] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.119087  4431 raft_consensus.cc:2749] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:13:01.119249  4418 raft_consensus.cc:385] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.119580  4418 raft_consensus.cc:740] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.120321  4418 consensus_queue.cc:260] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.122339  4418 ts_tablet_manager.cc:1434] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee: Time spent starting tablet: real 0.024s	user 0.005s	sys 0.001s
I20260709 13:13:01.123330  4418 tablet_bootstrap.cc:492] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee: Bootstrap starting.
I20260709 13:13:01.127874  4425 tablet_bootstrap.cc:492] T 835876e7bd174f4184fd6294093fa911 P 2a925808f54d42ed9cc086f846280594: No bootstrap required, opened a new log
I20260709 13:13:01.128669  4425 ts_tablet_manager.cc:1403] T 835876e7bd174f4184fd6294093fa911 P 2a925808f54d42ed9cc086f846280594: Time spent bootstrapping tablet: real 0.026s	user 0.008s	sys 0.008s
I20260709 13:13:01.130908  4418 tablet_bootstrap.cc:654] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:01.131742  4420 tablet_bootstrap.cc:492] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c: No bootstrap required, opened a new log
I20260709 13:13:01.132328  4420 ts_tablet_manager.cc:1403] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent bootstrapping tablet: real 0.037s	user 0.026s	sys 0.002s
I20260709 13:13:01.131883  4425 raft_consensus.cc:359] T 835876e7bd174f4184fd6294093fa911 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.133375  4425 raft_consensus.cc:385] T 835876e7bd174f4184fd6294093fa911 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.133752  4425 raft_consensus.cc:740] T 835876e7bd174f4184fd6294093fa911 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.134687  4425 consensus_queue.cc:260] T 835876e7bd174f4184fd6294093fa911 P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.135299  4420 raft_consensus.cc:359] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.136094  4420 raft_consensus.cc:385] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.136438  4420 raft_consensus.cc:740] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.137295  4420 consensus_queue.cc:260] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.138972  4418 tablet_bootstrap.cc:492] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee: No bootstrap required, opened a new log
I20260709 13:13:01.139063  4425 ts_tablet_manager.cc:1434] T 835876e7bd174f4184fd6294093fa911 P 2a925808f54d42ed9cc086f846280594: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.002s
I20260709 13:13:01.139617  4418 ts_tablet_manager.cc:1403] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee: Time spent bootstrapping tablet: real 0.016s	user 0.010s	sys 0.005s
I20260709 13:13:01.140278  4425 tablet_bootstrap.cc:492] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594: Bootstrap starting.
I20260709 13:13:01.141932  4420 ts_tablet_manager.cc:1434] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.003s
I20260709 13:13:01.143056  4420 tablet_bootstrap.cc:492] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c: Bootstrap starting.
I20260709 13:13:01.147620  4425 tablet_bootstrap.cc:654] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:01.149376  4420 tablet_bootstrap.cc:654] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:01.149672  4418 raft_consensus.cc:359] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.150576  4418 raft_consensus.cc:385] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.150921  4418 raft_consensus.cc:740] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.151818  4418 consensus_queue.cc:260] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.154454  4418 ts_tablet_manager.cc:1434] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 13:13:01.155922  4418 tablet_bootstrap.cc:492] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P ba722a02e5c34752b5a164e6234797ee: Bootstrap starting.
I20260709 13:13:01.156641  4423 raft_consensus.cc:493] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:01.157198  4423 raft_consensus.cc:515] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.159334  4423 leader_election.cc:290] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.160125  4425 tablet_bootstrap.cc:492] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594: No bootstrap required, opened a new log
I20260709 13:13:01.160740  4425 ts_tablet_manager.cc:1403] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594: Time spent bootstrapping tablet: real 0.021s	user 0.010s	sys 0.009s
I20260709 13:13:01.161351  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee8997813dd7430ca5f9d7269b2d0d5e" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:01.161814  4420 tablet_bootstrap.cc:492] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c: No bootstrap required, opened a new log
I20260709 13:13:01.162299  4420 ts_tablet_manager.cc:1403] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent bootstrapping tablet: real 0.019s	user 0.001s	sys 0.015s
I20260709 13:13:01.162601  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee8997813dd7430ca5f9d7269b2d0d5e" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:01.163276  4378 raft_consensus.cc:2468] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 0.
I20260709 13:13:01.163730  4425 raft_consensus.cc:359] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.164533  4425 raft_consensus.cc:385] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.164844  4425 raft_consensus.cc:740] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Initialized, Role: FOLLOWER
W20260709 13:13:01.164913  4194 leader_election.cc:343] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:01.165336  4420 raft_consensus.cc:359] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.166193  4420 raft_consensus.cc:385] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.165871  4192 leader_election.cc:304] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee; no voters: 6ff6192f285c425fb9d28bac1f0cea7c
I20260709 13:13:01.165825  4425 consensus_queue.cc:260] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.166558  4420 raft_consensus.cc:740] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.167228  4423 raft_consensus.cc:2804] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:01.167817  4423 raft_consensus.cc:493] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:01.167371  4420 consensus_queue.cc:260] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.168264  4423 raft_consensus.cc:3060] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.174335  4418 tablet_bootstrap.cc:654] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P ba722a02e5c34752b5a164e6234797ee: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:01.175271  4425 ts_tablet_manager.cc:1434] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 13:13:01.177013  4420 ts_tablet_manager.cc:1434] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260709 13:13:01.177491  4423 raft_consensus.cc:515] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.179935  4425 tablet_bootstrap.cc:492] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594: Bootstrap starting.
I20260709 13:13:01.180271  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee8997813dd7430ca5f9d7269b2d0d5e" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
W20260709 13:13:01.181770  4194 leader_election.cc:343] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:01.183843  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee8997813dd7430ca5f9d7269b2d0d5e" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:01.184422  4378 raft_consensus.cc:3060] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.187705  4420 tablet_bootstrap.cc:492] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c: Bootstrap starting.
I20260709 13:13:01.190747  4425 tablet_bootstrap.cc:654] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:01.192356  4378 raft_consensus.cc:2468] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 1.
I20260709 13:13:01.193925  4192 leader_election.cc:304] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee; no voters: 6ff6192f285c425fb9d28bac1f0cea7c
I20260709 13:13:01.194686  4420 tablet_bootstrap.cc:654] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:01.192539  4423 leader_election.cc:290] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Requested vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.195868  4423 raft_consensus.cc:2804] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:01.196396  4423 raft_consensus.cc:697] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Becoming Leader. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Running, Role: LEADER
I20260709 13:13:01.197326  4423 consensus_queue.cc:237] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.205853  4418 tablet_bootstrap.cc:492] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P ba722a02e5c34752b5a164e6234797ee: No bootstrap required, opened a new log
I20260709 13:13:01.206105  4420 tablet_bootstrap.cc:492] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c: No bootstrap required, opened a new log
I20260709 13:13:01.206631  4418 ts_tablet_manager.cc:1403] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P ba722a02e5c34752b5a164e6234797ee: Time spent bootstrapping tablet: real 0.051s	user 0.014s	sys 0.024s
I20260709 13:13:01.207604  4420 ts_tablet_manager.cc:1403] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent bootstrapping tablet: real 0.020s	user 0.016s	sys 0.002s
I20260709 13:13:01.206885  4425 tablet_bootstrap.cc:492] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594: No bootstrap required, opened a new log
I20260709 13:13:01.209406  4425 ts_tablet_manager.cc:1403] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594: Time spent bootstrapping tablet: real 0.030s	user 0.018s	sys 0.004s
I20260709 13:13:01.210127  4418 raft_consensus.cc:359] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.210903  4418 raft_consensus.cc:385] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.211194  4418 raft_consensus.cc:740] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.211923  4418 consensus_queue.cc:260] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.211916  4420 raft_consensus.cc:359] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.212999  4420 raft_consensus.cc:385] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.212607  4425 raft_consensus.cc:359] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.213596  4425 raft_consensus.cc:385] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.213387  4420 raft_consensus.cc:740] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.214114  4425 raft_consensus.cc:740] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.214401  4418 ts_tablet_manager.cc:1434] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P ba722a02e5c34752b5a164e6234797ee: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:13:01.213322  4127 catalog_manager.cc:5697] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ba722a02e5c34752b5a164e6234797ee (127.23.140.65). New cstate: current_term: 1 leader_uuid: "ba722a02e5c34752b5a164e6234797ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:01.214679  4420 consensus_queue.cc:260] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.215049  4425 consensus_queue.cc:260] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.215525  4418 tablet_bootstrap.cc:492] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee: Bootstrap starting.
I20260709 13:13:01.217084  4420 ts_tablet_manager.cc:1434] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 13:13:01.217605  4423 raft_consensus.cc:493] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:01.218291  4420 tablet_bootstrap.cc:492] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c: Bootstrap starting.
I20260709 13:13:01.218099  4423 raft_consensus.cc:515] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.220233  4423 leader_election.cc:290] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.221817  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02b49437a48440b7990bb6f87b03b6af" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:01.222585  4304 raft_consensus.cc:2468] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 0.
I20260709 13:13:01.223251  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02b49437a48440b7990bb6f87b03b6af" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:01.223898  4194 leader_election.cc:304] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:01.224911  4423 raft_consensus.cc:2804] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:01.225310  4423 raft_consensus.cc:493] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:01.221796  4425 ts_tablet_manager.cc:1434] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.001s
I20260709 13:13:01.225725  4423 raft_consensus.cc:3060] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.226275  4418 tablet_bootstrap.cc:654] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:01.226759  4425 tablet_bootstrap.cc:492] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594: Bootstrap starting.
I20260709 13:13:01.227412  4422 raft_consensus.cc:493] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:01.227905  4422 raft_consensus.cc:515] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.223963  4378 raft_consensus.cc:2468] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 0.
I20260709 13:13:01.229988  4422 leader_election.cc:290] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.232599  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2363b6da78da46a2a8435e57a9732b3d" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:01.233304  4423 raft_consensus.cc:515] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
W20260709 13:13:01.234113  4194 leader_election.cc:343] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:01.235373  4423 leader_election.cc:290] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Requested vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.225369  4420 tablet_bootstrap.cc:654] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:01.237552  4418 tablet_bootstrap.cc:492] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee: No bootstrap required, opened a new log
I20260709 13:13:01.237486  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2363b6da78da46a2a8435e57a9732b3d" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:01.238135  4418 ts_tablet_manager.cc:1403] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee: Time spent bootstrapping tablet: real 0.023s	user 0.013s	sys 0.004s
I20260709 13:13:01.238232  4378 raft_consensus.cc:2468] T 2363b6da78da46a2a8435e57a9732b3d P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 0.
I20260709 13:13:01.240818  4418 raft_consensus.cc:359] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.240885  4192 leader_election.cc:304] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee; no voters: 6ff6192f285c425fb9d28bac1f0cea7c
I20260709 13:13:01.241591  4418 raft_consensus.cc:385] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.242026  4418 raft_consensus.cc:740] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.242805  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02b49437a48440b7990bb6f87b03b6af" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:01.243100  4423 raft_consensus.cc:2804] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:01.243511  4304 raft_consensus.cc:3060] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.243675  4423 raft_consensus.cc:493] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:01.244310  4423 raft_consensus.cc:3060] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.245487  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02b49437a48440b7990bb6f87b03b6af" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:01.248833  4427 raft_consensus.cc:493] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:01.249336  4427 raft_consensus.cc:515] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.251449  4304 raft_consensus.cc:2468] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 1.
I20260709 13:13:01.252672  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb2b800af71b4a95b06e180939b9cae9" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:01.253419  4229 raft_consensus.cc:2468] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 0.
I20260709 13:13:01.253891  4377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb2b800af71b4a95b06e180939b9cae9" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:01.254673  4270 leader_election.cc:304] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
W20260709 13:13:01.255385  4267 leader_election.cc:343] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:01.252673  4194 leader_election.cc:304] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:01.257193  4422 raft_consensus.cc:2804] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:01.251451  4427 leader_election.cc:290] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.259425  4427 raft_consensus.cc:2804] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:01.259965  4427 raft_consensus.cc:493] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:01.242887  4418 consensus_queue.cc:260] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.260819  4427 raft_consensus.cc:3060] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.264091  4423 raft_consensus.cc:515] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.248323  4378 raft_consensus.cc:3060] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.265626  4425 tablet_bootstrap.cc:654] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:01.269443  4458 raft_consensus.cc:493] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:01.273355  4458 raft_consensus.cc:515] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.278949  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "835876e7bd174f4184fd6294093fa911" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:01.279289  4303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2363b6da78da46a2a8435e57a9732b3d" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:01.279714  4304 raft_consensus.cc:2468] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 0.
I20260709 13:13:01.281165  4194 leader_election.cc:304] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
W20260709 13:13:01.282259  4194 leader_election.cc:343] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:01.271234  4418 ts_tablet_manager.cc:1434] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee: Time spent starting tablet: real 0.033s	user 0.006s	sys 0.003s
I20260709 13:13:01.283176  4423 leader_election.cc:290] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Requested vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.284521  4423 raft_consensus.cc:2804] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:01.284936  4423 raft_consensus.cc:493] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:01.285392  4423 raft_consensus.cc:3060] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.294046  4431 raft_consensus.cc:493] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:01.294577  4431 raft_consensus.cc:515] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.295044  4420 tablet_bootstrap.cc:492] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c: No bootstrap required, opened a new log
I20260709 13:13:01.295688  4420 ts_tablet_manager.cc:1403] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent bootstrapping tablet: real 0.078s	user 0.017s	sys 0.038s
I20260709 13:13:01.296756  4431 leader_election.cc:290] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079)
I20260709 13:13:01.298339  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a8b2f9fd9bf4cd2a4ec339ca2e08d03" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:01.298532  4378 raft_consensus.cc:2468] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 1.
I20260709 13:13:01.286509  4458 leader_election.cc:290] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.298916  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a8b2f9fd9bf4cd2a4ec339ca2e08d03" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:01.299065  4229 raft_consensus.cc:2468] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 0.
I20260709 13:13:01.298880  4420 raft_consensus.cc:359] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.299995  4420 raft_consensus.cc:385] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.300105  4304 raft_consensus.cc:2468] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 0.
I20260709 13:13:01.300313  4420 raft_consensus.cc:740] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.300892  4344 leader_election.cc:304] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:01.301308  4420 consensus_queue.cc:260] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.298933  4422 raft_consensus.cc:697] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Becoming Leader. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Running, Role: LEADER
I20260709 13:13:01.302626  4431 raft_consensus.cc:2804] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:01.306471  4431 raft_consensus.cc:493] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:01.307138  4431 raft_consensus.cc:3060] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.304641  4423 raft_consensus.cc:515] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.295706  4427 raft_consensus.cc:515] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.314942  4423 leader_election.cc:290] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Requested vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.315527  4420 ts_tablet_manager.cc:1434] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent starting tablet: real 0.020s	user 0.010s	sys 0.000s
I20260709 13:13:01.315727  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "835876e7bd174f4184fd6294093fa911" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:01.316696  4420 tablet_bootstrap.cc:492] T 2363b6da78da46a2a8435e57a9732b3d P 6ff6192f285c425fb9d28bac1f0cea7c: Bootstrap starting.
I20260709 13:13:01.317130  4304 raft_consensus.cc:3060] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.319213  4425 tablet_bootstrap.cc:492] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594: No bootstrap required, opened a new log
I20260709 13:13:01.319733  4425 ts_tablet_manager.cc:1403] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594: Time spent bootstrapping tablet: real 0.093s	user 0.012s	sys 0.012s
I20260709 13:13:01.323014  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2363b6da78da46a2a8435e57a9732b3d" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:01.323623  4378 raft_consensus.cc:3060] T 2363b6da78da46a2a8435e57a9732b3d P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.323416  4425 raft_consensus.cc:359] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.323776  4377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "835876e7bd174f4184fd6294093fa911" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:01.324213  4425 raft_consensus.cc:385] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.324474  4377 raft_consensus.cc:2468] T 835876e7bd174f4184fd6294093fa911 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 0.
I20260709 13:13:01.324579  4425 raft_consensus.cc:740] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.324738  4420 tablet_bootstrap.cc:654] T 2363b6da78da46a2a8435e57a9732b3d P 6ff6192f285c425fb9d28bac1f0cea7c: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:01.325639  4425 consensus_queue.cc:260] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
W20260709 13:13:01.306425  4270 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.140.65:40371, user_credentials={real_user=slave}} blocked reactor thread for 51826.4us
I20260709 13:13:01.326570  4377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "835876e7bd174f4184fd6294093fa911" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:01.328758  4377 raft_consensus.cc:3060] T 835876e7bd174f4184fd6294093fa911 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.331682  4431 raft_consensus.cc:515] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.332018  4304 raft_consensus.cc:2468] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 1.
I20260709 13:13:01.334357  4376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb2b800af71b4a95b06e180939b9cae9" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:01.335412  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb2b800af71b4a95b06e180939b9cae9" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:01.336246  4229 raft_consensus.cc:3060] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.336304  4194 leader_election.cc:304] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:01.339417  4377 raft_consensus.cc:2468] T 835876e7bd174f4184fd6294093fa911 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 1.
W20260709 13:13:01.340899  4267 leader_election.cc:343] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:01.341596  4431 leader_election.cc:290] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079)
I20260709 13:13:01.344211  4228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a8b2f9fd9bf4cd2a4ec339ca2e08d03" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:01.345055  4228 raft_consensus.cc:3060] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.346626  4425 ts_tablet_manager.cc:1434] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594: Time spent starting tablet: real 0.027s	user 0.010s	sys 0.011s
I20260709 13:13:01.347681  4425 tablet_bootstrap.cc:492] T c0e4091b5254449b9ece5ccdccee16ee P 2a925808f54d42ed9cc086f846280594: Bootstrap starting.
I20260709 13:13:01.348464  4229 raft_consensus.cc:2468] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 1.
I20260709 13:13:01.343608  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a8b2f9fd9bf4cd2a4ec339ca2e08d03" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:01.349881  4304 raft_consensus.cc:3060] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.353516  4459 raft_consensus.cc:2804] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:01.358057  4431 raft_consensus.cc:493] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:01.358601  4431 raft_consensus.cc:515] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.358985  4425 tablet_bootstrap.cc:654] T c0e4091b5254449b9ece5ccdccee16ee P 2a925808f54d42ed9cc086f846280594: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:01.360716  4431 leader_election.cc:290] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079)
I20260709 13:13:01.326900  4422 consensus_queue.cc:237] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.328207  4427 leader_election.cc:290] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.366230  4228 raft_consensus.cc:2468] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 1.
I20260709 13:13:01.371248  4344 leader_election.cc:304] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:01.374245  4427 raft_consensus.cc:493] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:01.364816  4303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba55bc7ffc73446da8111405c7a7b7f0" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:01.374833  4427 raft_consensus.cc:515] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.375159  4228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba55bc7ffc73446da8111405c7a7b7f0" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:01.375608  4431 raft_consensus.cc:2804] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:01.375725  4270 leader_election.cc:304] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 2a925808f54d42ed9cc086f846280594
I20260709 13:13:01.376590  4228 raft_consensus.cc:2468] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 0.
I20260709 13:13:01.376585  4431 raft_consensus.cc:697] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Becoming Leader. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Running, Role: LEADER
I20260709 13:13:01.377377  4456 raft_consensus.cc:2804] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:01.362159  4459 raft_consensus.cc:697] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Becoming Leader. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Running, Role: LEADER
I20260709 13:13:01.378844  4304 raft_consensus.cc:2468] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 1.
I20260709 13:13:01.381855  4459 consensus_queue.cc:237] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.382388  4427 leader_election.cc:290] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.383865  4303 raft_consensus.cc:2468] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 0.
I20260709 13:13:01.377871  4431 consensus_queue.cc:237] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.388398  4344 leader_election.cc:304] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:01.389621  4420 tablet_bootstrap.cc:492] T 2363b6da78da46a2a8435e57a9732b3d P 6ff6192f285c425fb9d28bac1f0cea7c: No bootstrap required, opened a new log
I20260709 13:13:01.390182  4420 ts_tablet_manager.cc:1403] T 2363b6da78da46a2a8435e57a9732b3d P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent bootstrapping tablet: real 0.074s	user 0.020s	sys 0.018s
I20260709 13:13:01.392773  4378 raft_consensus.cc:2468] T 2363b6da78da46a2a8435e57a9732b3d P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 1.
I20260709 13:13:01.394263  4420 raft_consensus.cc:359] T 2363b6da78da46a2a8435e57a9732b3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.395231  4420 raft_consensus.cc:385] T 2363b6da78da46a2a8435e57a9732b3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.395570  4420 raft_consensus.cc:740] T 2363b6da78da46a2a8435e57a9732b3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.396113  4192 leader_election.cc:304] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee; no voters: 6ff6192f285c425fb9d28bac1f0cea7c
I20260709 13:13:01.397015  4423 raft_consensus.cc:2804] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:01.397504  4423 raft_consensus.cc:697] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Becoming Leader. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Running, Role: LEADER
I20260709 13:13:01.397912  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba55bc7ffc73446da8111405c7a7b7f0" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:01.398552  4378 raft_consensus.cc:2468] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 0.
I20260709 13:13:01.398310  4423 consensus_queue.cc:237] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.409238  4425 tablet_bootstrap.cc:492] T c0e4091b5254449b9ece5ccdccee16ee P 2a925808f54d42ed9cc086f846280594: No bootstrap required, opened a new log
I20260709 13:13:01.409775  4425 ts_tablet_manager.cc:1403] T c0e4091b5254449b9ece5ccdccee16ee P 2a925808f54d42ed9cc086f846280594: Time spent bootstrapping tablet: real 0.062s	user 0.012s	sys 0.011s
I20260709 13:13:01.409399  4420 consensus_queue.cc:260] T 2363b6da78da46a2a8435e57a9732b3d P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.409471  4267 leader_election.cc:304] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, 6ff6192f285c425fb9d28bac1f0cea7c; no voters: 
I20260709 13:13:01.391939  4456 raft_consensus.cc:697] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 LEADER]: Becoming Leader. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Running, Role: LEADER
I20260709 13:13:01.412717  4456 consensus_queue.cc:237] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.415375  4425 raft_consensus.cc:359] T c0e4091b5254449b9ece5ccdccee16ee P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.416081  4425 raft_consensus.cc:385] T c0e4091b5254449b9ece5ccdccee16ee P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.416348  4425 raft_consensus.cc:740] T c0e4091b5254449b9ece5ccdccee16ee P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.411566  4228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba55bc7ffc73446da8111405c7a7b7f0" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:01.417951  4228 raft_consensus.cc:2468] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 0.
I20260709 13:13:01.417022  4425 consensus_queue.cc:260] T c0e4091b5254449b9ece5ccdccee16ee P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.424410  4420 ts_tablet_manager.cc:1434] T 2363b6da78da46a2a8435e57a9732b3d P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent starting tablet: real 0.034s	user 0.008s	sys 0.001s
I20260709 13:13:01.412789  4462 raft_consensus.cc:2804] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:01.426915  4462 raft_consensus.cc:493] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:01.427968  4462 raft_consensus.cc:3060] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.428052  4425 ts_tablet_manager.cc:1434] T c0e4091b5254449b9ece5ccdccee16ee P 2a925808f54d42ed9cc086f846280594: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.000s
I20260709 13:13:01.429332  4425 tablet_bootstrap.cc:492] T fb2b800af71b4a95b06e180939b9cae9 P 2a925808f54d42ed9cc086f846280594: Bootstrap starting.
I20260709 13:13:01.426837  4427 raft_consensus.cc:2804] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:01.431098  4427 raft_consensus.cc:493] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:01.431491  4427 raft_consensus.cc:3060] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.434479  4425 tablet_bootstrap.cc:654] T fb2b800af71b4a95b06e180939b9cae9 P 2a925808f54d42ed9cc086f846280594: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:01.436458  4462 raft_consensus.cc:515] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.439944  4228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba55bc7ffc73446da8111405c7a7b7f0" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:01.440549  4228 raft_consensus.cc:3060] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.441697  4462 leader_election.cc:290] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079)
I20260709 13:13:01.444583  4303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba55bc7ffc73446da8111405c7a7b7f0" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:01.447563  4427 raft_consensus.cc:515] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.448758  4303 raft_consensus.cc:2393] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a925808f54d42ed9cc086f846280594 in current term 1: Already voted for candidate 6ff6192f285c425fb9d28bac1f0cea7c in this term.
I20260709 13:13:01.449182  4228 raft_consensus.cc:2468] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 1.
I20260709 13:13:01.442413  4124 catalog_manager.cc:5697] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a925808f54d42ed9cc086f846280594 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "2a925808f54d42ed9cc086f846280594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: HEALTHY } } }
I20260709 13:13:01.452762  4344 leader_election.cc:304] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee; no voters: 6ff6192f285c425fb9d28bac1f0cea7c
I20260709 13:13:01.455886  4228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba55bc7ffc73446da8111405c7a7b7f0" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:01.456727  4228 raft_consensus.cc:2393] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ff6192f285c425fb9d28bac1f0cea7c in current term 1: Already voted for candidate 2a925808f54d42ed9cc086f846280594 in this term.
I20260709 13:13:01.458873  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba55bc7ffc73446da8111405c7a7b7f0" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:01.459424  4425 tablet_bootstrap.cc:492] T fb2b800af71b4a95b06e180939b9cae9 P 2a925808f54d42ed9cc086f846280594: No bootstrap required, opened a new log
I20260709 13:13:01.459710  4378 raft_consensus.cc:2393] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ff6192f285c425fb9d28bac1f0cea7c in current term 1: Already voted for candidate 2a925808f54d42ed9cc086f846280594 in this term.
I20260709 13:13:01.459930  4425 ts_tablet_manager.cc:1403] T fb2b800af71b4a95b06e180939b9cae9 P 2a925808f54d42ed9cc086f846280594: Time spent bootstrapping tablet: real 0.031s	user 0.016s	sys 0.000s
I20260709 13:13:01.457886  4125 catalog_manager.cc:5697] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66). New cstate: current_term: 1 leader_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:01.461644  4267 leader_election.cc:304] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c; no voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee
I20260709 13:13:01.462553  4427 leader_election.cc:290] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.462713  4466 raft_consensus.cc:2749] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:13:01.462910  4425 raft_consensus.cc:359] T fb2b800af71b4a95b06e180939b9cae9 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.463685  4425 raft_consensus.cc:385] T fb2b800af71b4a95b06e180939b9cae9 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:01.463968  4425 raft_consensus.cc:740] T fb2b800af71b4a95b06e180939b9cae9 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Initialized, Role: FOLLOWER
I20260709 13:13:01.464757  4425 consensus_queue.cc:260] T fb2b800af71b4a95b06e180939b9cae9 P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.466356  4425 ts_tablet_manager.cc:1434] T fb2b800af71b4a95b06e180939b9cae9 P 2a925808f54d42ed9cc086f846280594: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:13:01.465351  4462 raft_consensus.cc:2804] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:01.467180  4462 raft_consensus.cc:697] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Becoming Leader. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Running, Role: LEADER
I20260709 13:13:01.467935  4462 consensus_queue.cc:237] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.472136  4127 catalog_manager.cc:5697] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ba722a02e5c34752b5a164e6234797ee (127.23.140.65). New cstate: current_term: 1 leader_uuid: "ba722a02e5c34752b5a164e6234797ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:01.475530  4127 catalog_manager.cc:5697] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ba722a02e5c34752b5a164e6234797ee (127.23.140.65). New cstate: current_term: 1 leader_uuid: "ba722a02e5c34752b5a164e6234797ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:01.477420  4127 catalog_manager.cc:5697] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ba722a02e5c34752b5a164e6234797ee (127.23.140.65). New cstate: current_term: 1 leader_uuid: "ba722a02e5c34752b5a164e6234797ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:01.480407  4125 catalog_manager.cc:5697] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a925808f54d42ed9cc086f846280594 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "2a925808f54d42ed9cc086f846280594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: HEALTHY } } }
I20260709 13:13:01.507032  4466 raft_consensus.cc:493] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:01.507560  4466 raft_consensus.cc:515] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.509362  4466 leader_election.cc:290] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.510355  4228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1d29a41a99463fa23d6308fc2afaaa" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:01.510504  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1d29a41a99463fa23d6308fc2afaaa" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:01.511055  4228 raft_consensus.cc:2468] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 0.
I20260709 13:13:01.511155  4378 raft_consensus.cc:2468] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 0.
I20260709 13:13:01.512181  4270 leader_election.cc:304] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:01.512898  4466 raft_consensus.cc:2804] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:01.513298  4466 raft_consensus.cc:493] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:01.513592  4466 raft_consensus.cc:3060] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.517001  4427 raft_consensus.cc:493] T 2363b6da78da46a2a8435e57a9732b3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:01.517506  4427 raft_consensus.cc:515] T 2363b6da78da46a2a8435e57a9732b3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.519323  4427 leader_election.cc:290] T 2363b6da78da46a2a8435e57a9732b3d P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.518724  4466 raft_consensus.cc:515] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.520009  4228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2363b6da78da46a2a8435e57a9732b3d" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:01.520627  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2363b6da78da46a2a8435e57a9732b3d" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:01.521553  4378 raft_consensus.cc:2393] T 2363b6da78da46a2a8435e57a9732b3d P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ff6192f285c425fb9d28bac1f0cea7c in current term 1: Already voted for candidate ba722a02e5c34752b5a164e6234797ee in this term.
I20260709 13:13:01.521500  4466 leader_election.cc:290] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.522920  4228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1d29a41a99463fa23d6308fc2afaaa" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:01.523267  4267 leader_election.cc:304] T 2363b6da78da46a2a8435e57a9732b3d P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c; no voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee
I20260709 13:13:01.523528  4228 raft_consensus.cc:3060] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.523571  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1d29a41a99463fa23d6308fc2afaaa" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:01.524519  4466 raft_consensus.cc:3060] T 2363b6da78da46a2a8435e57a9732b3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.524573  4378 raft_consensus.cc:3060] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.531167  4228 raft_consensus.cc:2468] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 1.
I20260709 13:13:01.531677  4466 raft_consensus.cc:2749] T 2363b6da78da46a2a8435e57a9732b3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:13:01.532586  4378 raft_consensus.cc:2468] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 1.
I20260709 13:13:01.532517  4270 leader_election.cc:304] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:01.536230  4466 raft_consensus.cc:2804] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:01.536818  4466 raft_consensus.cc:697] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 LEADER]: Becoming Leader. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Running, Role: LEADER
I20260709 13:13:01.537686  4466 consensus_queue.cc:237] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.545899  4125 catalog_manager.cc:5697] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66). New cstate: current_term: 1 leader_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:01.554936  4423 raft_consensus.cc:493] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:01.555456  4423 raft_consensus.cc:515] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.557343  4423 leader_election.cc:290] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.557989  4303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0e4091b5254449b9ece5ccdccee16ee" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:01.558450  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0e4091b5254449b9ece5ccdccee16ee" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:01.558729  4303 raft_consensus.cc:2468] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 0.
I20260709 13:13:01.559115  4378 raft_consensus.cc:2468] T c0e4091b5254449b9ece5ccdccee16ee P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 0.
I20260709 13:13:01.560086  4194 leader_election.cc:304] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:01.560742  4423 raft_consensus.cc:2804] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:01.561070  4423 raft_consensus.cc:493] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:01.561390  4423 raft_consensus.cc:3060] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.566634  4423 raft_consensus.cc:515] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.567713  4466 raft_consensus.cc:493] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:01.568259  4466 raft_consensus.cc:515] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.569152  4423 leader_election.cc:290] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Requested vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.569371  4303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0e4091b5254449b9ece5ccdccee16ee" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:01.570004  4303 raft_consensus.cc:3060] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.570264  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0e4091b5254449b9ece5ccdccee16ee" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:01.570886  4466 leader_election.cc:290] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.571117  4228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee8997813dd7430ca5f9d7269b2d0d5e" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:01.571396  4378 raft_consensus.cc:3060] T c0e4091b5254449b9ece5ccdccee16ee P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.571974  4377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee8997813dd7430ca5f9d7269b2d0d5e" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:01.572780  4377 raft_consensus.cc:2393] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ff6192f285c425fb9d28bac1f0cea7c in current term 1: Already voted for candidate ba722a02e5c34752b5a164e6234797ee in this term.
I20260709 13:13:01.573769  4267 leader_election.cc:304] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c; no voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee
I20260709 13:13:01.574569  4466 raft_consensus.cc:3060] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.577368  4303 raft_consensus.cc:2468] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 1.
I20260709 13:13:01.578322  4378 raft_consensus.cc:2468] T c0e4091b5254449b9ece5ccdccee16ee P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 1.
I20260709 13:13:01.578756  4194 leader_election.cc:304] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:01.579674  4423 raft_consensus.cc:2804] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:01.580137  4423 raft_consensus.cc:697] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Becoming Leader. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Running, Role: LEADER
I20260709 13:13:01.580876  4466 raft_consensus.cc:2749] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:13:01.580894  4423 consensus_queue.cc:237] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.588371  4125 catalog_manager.cc:5697] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ba722a02e5c34752b5a164e6234797ee (127.23.140.65). New cstate: current_term: 1 leader_uuid: "ba722a02e5c34752b5a164e6234797ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:01.603768  4462 raft_consensus.cc:493] T fb2b800af71b4a95b06e180939b9cae9 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:01.604283  4462 raft_consensus.cc:515] T fb2b800af71b4a95b06e180939b9cae9 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.605999  4462 leader_election.cc:290] T fb2b800af71b4a95b06e180939b9cae9 P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079)
I20260709 13:13:01.606848  4228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb2b800af71b4a95b06e180939b9cae9" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:01.606976  4303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb2b800af71b4a95b06e180939b9cae9" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:01.607621  4228 raft_consensus.cc:2393] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2a925808f54d42ed9cc086f846280594 in current term 1: Already voted for candidate 6ff6192f285c425fb9d28bac1f0cea7c in this term.
I20260709 13:13:01.608695  4344 leader_election.cc:304] T fb2b800af71b4a95b06e180939b9cae9 P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a925808f54d42ed9cc086f846280594; no voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee
I20260709 13:13:01.609334  4462 raft_consensus.cc:3060] T fb2b800af71b4a95b06e180939b9cae9 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:01.615352  4462 raft_consensus.cc:2749] T fb2b800af71b4a95b06e180939b9cae9 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:13:01.701282  4423 consensus_queue.cc:1048] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:01.713213  4458 consensus_queue.cc:1048] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:01.779953  4462 consensus_queue.cc:1048] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:01.793110  4378 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ba55bc7ffc73446da8111405c7a7b7f0"
dest_uuid: "2a925808f54d42ed9cc086f846280594"
mode: GRACEFUL
new_leader_uuid: "ba722a02e5c34752b5a164e6234797ee"
 from {username='slave'} at 127.0.0.1:45142
I20260709 13:13:01.793723  4469 consensus_queue.cc:1048] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:01.793766  4378 raft_consensus.cc:606] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Received request to transfer leadership to TS ba722a02e5c34752b5a164e6234797ee
I20260709 13:13:01.819448  4228 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "835876e7bd174f4184fd6294093fa911"
dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
mode: GRACEFUL
new_leader_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
 from {username='slave'} at 127.0.0.1:60070
I20260709 13:13:01.819968  4228 raft_consensus.cc:606] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Received request to transfer leadership to TS 6ff6192f285c425fb9d28bac1f0cea7c
I20260709 13:13:01.824673  4378 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2a8b2f9fd9bf4cd2a4ec339ca2e08d03"
dest_uuid: "2a925808f54d42ed9cc086f846280594"
mode: GRACEFUL
new_leader_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
 from {username='slave'} at 127.0.0.1:45142
I20260709 13:13:01.825189  4378 raft_consensus.cc:606] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Received request to transfer leadership to TS 6ff6192f285c425fb9d28bac1f0cea7c
I20260709 13:13:01.828738  4458 consensus_queue.cc:1048] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:01.830358  4228 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "02b49437a48440b7990bb6f87b03b6af"
dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
mode: GRACEFUL
new_leader_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
 from {username='slave'} at 127.0.0.1:60070
I20260709 13:13:01.830945  4228 raft_consensus.cc:606] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Received request to transfer leadership to TS 6ff6192f285c425fb9d28bac1f0cea7c
I20260709 13:13:01.841265  4423 consensus_queue.cc:1048] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:01.862427  4303 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8f1d29a41a99463fa23d6308fc2afaaa"
dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
mode: GRACEFUL
new_leader_uuid: "2a925808f54d42ed9cc086f846280594"
 from {username='slave'} at 127.0.0.1:47852
I20260709 13:13:01.863082  4303 raft_consensus.cc:606] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 LEADER]: Received request to transfer leadership to TS 2a925808f54d42ed9cc086f846280594
I20260709 13:13:01.878259  4423 consensus_queue.cc:1048] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:01.885273  4457 raft_consensus.cc:993] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee: : Instructing follower 6ff6192f285c425fb9d28bac1f0cea7c to start an election
I20260709 13:13:01.885723  4457 raft_consensus.cc:1081] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Signalling peer 6ff6192f285c425fb9d28bac1f0cea7c to start an election
I20260709 13:13:01.887239  4458 consensus_queue.cc:1048] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:01.891680  4303 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "835876e7bd174f4184fd6294093fa911"
dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
 from {username='slave'} at 127.0.0.1:47836
I20260709 13:13:01.892402  4303 raft_consensus.cc:493] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:13:01.892980  4303 raft_consensus.cc:3060] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:01.893827  4457 consensus_queue.cc:1048] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:13:01.901611  4303 raft_consensus.cc:515] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.905067  4457 raft_consensus.cc:993] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee: : Instructing follower 6ff6192f285c425fb9d28bac1f0cea7c to start an election
I20260709 13:13:01.905789  4423 raft_consensus.cc:1081] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Signalling peer 6ff6192f285c425fb9d28bac1f0cea7c to start an election
I20260709 13:13:01.907666  4302 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "02b49437a48440b7990bb6f87b03b6af"
dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
 from {username='slave'} at 127.0.0.1:47836
I20260709 13:13:01.910782  4457 consensus_queue.cc:1048] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:01.911054  4302 raft_consensus.cc:493] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:13:01.911540  4302 raft_consensus.cc:3060] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:01.914700  4303 leader_election.cc:290] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 2 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.914844  4228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "835876e7bd174f4184fd6294093fa911" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:01.916002  4228 raft_consensus.cc:3055] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:13:01.916335  4228 raft_consensus.cc:740] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Running, Role: LEADER
I20260709 13:13:01.917198  4228 consensus_queue.cc:260] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.917680  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "835876e7bd174f4184fd6294093fa911" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:01.918556  4228 raft_consensus.cc:3060] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:01.918745  4378 raft_consensus.cc:3060] T 835876e7bd174f4184fd6294093fa911 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:01.926648  4302 raft_consensus.cc:515] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.927771  4378 raft_consensus.cc:2468] T 835876e7bd174f4184fd6294093fa911 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 2.
I20260709 13:13:01.928573  4228 raft_consensus.cc:2468] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 2.
I20260709 13:13:01.929055  4267 leader_election.cc:304] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, 6ff6192f285c425fb9d28bac1f0cea7c; no voters: 
I20260709 13:13:01.929975  4466 raft_consensus.cc:2804] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:13:01.930444  4466 raft_consensus.cc:697] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 LEADER]: Becoming Leader. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Running, Role: LEADER
I20260709 13:13:01.931281  4466 consensus_queue.cc:237] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.933146  4427 consensus_queue.cc:1048] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:01.934454  4302 leader_election.cc:290] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 2 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:01.936214  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02b49437a48440b7990bb6f87b03b6af" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:01.937071  4228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02b49437a48440b7990bb6f87b03b6af" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:01.937407  4378 raft_consensus.cc:3060] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:01.937799  4228 raft_consensus.cc:3055] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:13:01.938190  4228 raft_consensus.cc:740] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Running, Role: LEADER
I20260709 13:13:01.938997  4228 consensus_queue.cc:260] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.940303  4228 raft_consensus.cc:3060] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:01.942279  4127 catalog_manager.cc:5697] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c reported cstate change: term changed from 1 to 2, leader changed from ba722a02e5c34752b5a164e6234797ee (127.23.140.65) to 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66). New cstate: current_term: 2 leader_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:01.946226  4378 raft_consensus.cc:2468] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 2.
I20260709 13:13:01.947366  4228 raft_consensus.cc:2468] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 2.
I20260709 13:13:01.948478  4270 leader_election.cc:304] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:01.949338  4427 raft_consensus.cc:2804] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:13:01.949999  4427 raft_consensus.cc:697] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 LEADER]: Becoming Leader. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Running, Role: LEADER
I20260709 13:13:01.951000  4427 consensus_queue.cc:237] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:01.951200  4466 consensus_queue.cc:1048] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:01.971805  4127 catalog_manager.cc:5697] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c reported cstate change: term changed from 1 to 2, leader changed from ba722a02e5c34752b5a164e6234797ee (127.23.140.65) to 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66). New cstate: current_term: 2 leader_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:01.994980  4466 consensus_queue.cc:1048] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:02.001163  4497 raft_consensus.cc:993] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c: : Instructing follower 2a925808f54d42ed9cc086f846280594 to start an election
I20260709 13:13:02.001678  4427 raft_consensus.cc:1081] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 LEADER]: Signalling peer 2a925808f54d42ed9cc086f846280594 to start an election
I20260709 13:13:02.003423  4377 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8f1d29a41a99463fa23d6308fc2afaaa"
dest_uuid: "2a925808f54d42ed9cc086f846280594"
 from {username='slave'} at 127.0.0.1:45136
I20260709 13:13:02.004695  4377 raft_consensus.cc:493] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:13:02.005103  4377 raft_consensus.cc:3060] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:02.006784  4427 consensus_queue.cc:1048] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:02.010278  4469 consensus_queue.cc:1048] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:02.015110  4377 raft_consensus.cc:515] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:02.023567  4303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1d29a41a99463fa23d6308fc2afaaa" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:02.023841  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1d29a41a99463fa23d6308fc2afaaa" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:02.024202  4303 raft_consensus.cc:3055] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:13:02.024511  4229 raft_consensus.cc:3060] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:02.024626  4303 raft_consensus.cc:740] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Running, Role: LEADER
I20260709 13:13:02.025614  4303 consensus_queue.cc:260] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:02.026784  4303 raft_consensus.cc:3060] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:02.029265  4377 leader_election.cc:290] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 2 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079)
I20260709 13:13:02.030375  4469 consensus_queue.cc:1048] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:02.031247  4229 raft_consensus.cc:2468] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 2.
I20260709 13:13:02.032341  4344 leader_election.cc:304] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:02.033178  4469 raft_consensus.cc:2804] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:13:02.033649  4469 raft_consensus.cc:697] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [term 2 LEADER]: Becoming Leader. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Running, Role: LEADER
I20260709 13:13:02.034518  4469 consensus_queue.cc:237] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:02.035871  4303 raft_consensus.cc:2468] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 2.
I20260709 13:13:02.045114  4431 raft_consensus.cc:993] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594: : Instructing follower ba722a02e5c34752b5a164e6234797ee to start an election
I20260709 13:13:02.044524  4127 catalog_manager.cc:5697] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 reported cstate change: term changed from 1 to 2, leader changed from 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66) to 2a925808f54d42ed9cc086f846280594 (127.23.140.67). New cstate: current_term: 2 leader_uuid: "2a925808f54d42ed9cc086f846280594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: HEALTHY } } }
I20260709 13:13:02.045644  4431 raft_consensus.cc:1081] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Signalling peer ba722a02e5c34752b5a164e6234797ee to start an election
I20260709 13:13:02.047322  4228 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ba55bc7ffc73446da8111405c7a7b7f0"
dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
 from {username='slave'} at 127.0.0.1:60066
I20260709 13:13:02.047883  4228 raft_consensus.cc:493] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:13:02.048285  4228 raft_consensus.cc:3060] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:02.055182  4228 raft_consensus.cc:515] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:02.056125  4229 raft_consensus.cc:1240] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Rejecting Update request from peer 2a925808f54d42ed9cc086f846280594 for earlier term 1. Current term is 2. Ops: []
I20260709 13:13:02.057912  4302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba55bc7ffc73446da8111405c7a7b7f0" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:02.058511  4302 raft_consensus.cc:3060] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:02.059676  4462 consensus_queue.cc:1059] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:02.060508  4462 raft_consensus.cc:3055] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:13:02.060829  4462 raft_consensus.cc:740] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Running, Role: LEADER
I20260709 13:13:02.061630  4462 consensus_queue.cc:260] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:02.062291  4228 leader_election.cc:290] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 2 election: Requested vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:02.062529  4377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba55bc7ffc73446da8111405c7a7b7f0" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:02.062734  4462 raft_consensus.cc:3060] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:02.065579  4302 raft_consensus.cc:2468] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 2.
I20260709 13:13:02.066805  4194 leader_election.cc:304] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:02.067471  4457 raft_consensus.cc:2804] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:13:02.067894  4457 raft_consensus.cc:697] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 2 LEADER]: Becoming Leader. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Running, Role: LEADER
I20260709 13:13:02.068677  4457 consensus_queue.cc:237] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:02.074770  4377 raft_consensus.cc:2468] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 2.
I20260709 13:13:02.076591  4127 catalog_manager.cc:5697] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee reported cstate change: term changed from 1 to 2, leader changed from 2a925808f54d42ed9cc086f846280594 (127.23.140.67) to ba722a02e5c34752b5a164e6234797ee (127.23.140.65). New cstate: current_term: 2 leader_uuid: "ba722a02e5c34752b5a164e6234797ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:02.177348  4457 consensus_queue.cc:1048] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:02.182951  4457 consensus_queue.cc:1048] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:02.229194  4431 raft_consensus.cc:993] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594: : Instructing follower 6ff6192f285c425fb9d28bac1f0cea7c to start an election
I20260709 13:13:02.229703  4469 raft_consensus.cc:1081] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Signalling peer 6ff6192f285c425fb9d28bac1f0cea7c to start an election
I20260709 13:13:02.231195  4302 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2a8b2f9fd9bf4cd2a4ec339ca2e08d03"
dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
 from {username='slave'} at 127.0.0.1:47850
I20260709 13:13:02.231686  4302 raft_consensus.cc:493] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:13:02.231989  4302 raft_consensus.cc:3060] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:02.236860  4302 raft_consensus.cc:515] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:02.238477  4302 leader_election.cc:290] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 2 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:02.239352  4228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a8b2f9fd9bf4cd2a4ec339ca2e08d03" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:02.239622  4377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a8b2f9fd9bf4cd2a4ec339ca2e08d03" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:02.240039  4228 raft_consensus.cc:3060] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:02.240178  4377 raft_consensus.cc:3055] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:13:02.240608  4377 raft_consensus.cc:740] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Running, Role: LEADER
I20260709 13:13:02.241456  4377 consensus_queue.cc:260] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:02.242692  4377 raft_consensus.cc:3060] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:02.247097  4228 raft_consensus.cc:2468] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 2.
I20260709 13:13:02.248412  4270 leader_election.cc:304] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:02.249091  4427 raft_consensus.cc:2804] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:13:02.249614  4427 raft_consensus.cc:697] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 LEADER]: Becoming Leader. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Running, Role: LEADER
I20260709 13:13:02.249567  4377 raft_consensus.cc:2468] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 2.
I20260709 13:13:02.250701  4427 consensus_queue.cc:237] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:02.256834  4127 catalog_manager.cc:5697] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c reported cstate change: term changed from 1 to 2, leader changed from 2a925808f54d42ed9cc086f846280594 (127.23.140.67) to 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66). New cstate: current_term: 2 leader_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:02.360467  4377 raft_consensus.cc:1275] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Refusing update from remote peer 6ff6192f285c425fb9d28bac1f0cea7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:02.361829  4427 consensus_queue.cc:1048] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:13:02.370013  4228 raft_consensus.cc:1275] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Refusing update from remote peer 6ff6192f285c425fb9d28bac1f0cea7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:02.371908  4497 consensus_queue.cc:1048] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:02.440429  4228 raft_consensus.cc:1275] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Refusing update from remote peer 6ff6192f285c425fb9d28bac1f0cea7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:02.441588  4497 consensus_queue.cc:1048] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:02.446396  4229 raft_consensus.cc:1275] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Refusing update from remote peer 2a925808f54d42ed9cc086f846280594: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:02.448385  4469 consensus_queue.cc:1048] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:02.452756  4377 raft_consensus.cc:1275] T 835876e7bd174f4184fd6294093fa911 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Refusing update from remote peer 6ff6192f285c425fb9d28bac1f0cea7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:02.454349  4497 consensus_queue.cc:1048] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:02.458537  4302 raft_consensus.cc:1275] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Refusing update from remote peer 2a925808f54d42ed9cc086f846280594: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:02.459962  4462 consensus_queue.cc:1048] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:02.510010  4303 raft_consensus.cc:1275] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Refusing update from remote peer ba722a02e5c34752b5a164e6234797ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:02.511423  4457 consensus_queue.cc:1048] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:02.519500  4377 raft_consensus.cc:1275] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Refusing update from remote peer ba722a02e5c34752b5a164e6234797ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:02.521411  4459 consensus_queue.cc:1048] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:02.720249  4378 raft_consensus.cc:1275] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Refusing update from remote peer 6ff6192f285c425fb9d28bac1f0cea7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:02.721737  4497 consensus_queue.cc:1048] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:02.729352  4229 raft_consensus.cc:1275] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Refusing update from remote peer 6ff6192f285c425fb9d28bac1f0cea7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:02.730958  4497 consensus_queue.cc:1048] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:04.897564  4127 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38580:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:13:04.900246  4127 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:13:04.951877  4216 tablet_service.cc:1511] Processing CreateTablet for tablet b7b904c5d9654b758bb74c910a8a22c2 (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:13:04.953199  4293 tablet_service.cc:1511] Processing CreateTablet for tablet b7b904c5d9654b758bb74c910a8a22c2 (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:13:04.953725  4216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b904c5d9654b758bb74c910a8a22c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:04.954983  4293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b904c5d9654b758bb74c910a8a22c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:04.955783  4361 tablet_service.cc:1511] Processing CreateTablet for tablet b7b904c5d9654b758bb74c910a8a22c2 (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:13:04.957410  4361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b904c5d9654b758bb74c910a8a22c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:04.957293  4213 tablet_service.cc:1511] Processing CreateTablet for tablet 6a287c34d1b14502b5541f96aa8c90bc (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:13:04.958853  4213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a287c34d1b14502b5541f96aa8c90bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:04.962154  4292 tablet_service.cc:1511] Processing CreateTablet for tablet 6a287c34d1b14502b5541f96aa8c90bc (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:13:04.963678  4292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a287c34d1b14502b5541f96aa8c90bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:04.965595  4294 tablet_service.cc:1511] Processing CreateTablet for tablet 9a69dca66f3a41e7a1a2bf40d5a5d641 (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:13:04.966997  4294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a69dca66f3a41e7a1a2bf40d5a5d641. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:04.970739  4291 tablet_service.cc:1511] Processing CreateTablet for tablet 2551af500d034f6096ef929e30795d3d (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:13:04.970950  4362 tablet_service.cc:1511] Processing CreateTablet for tablet 6a287c34d1b14502b5541f96aa8c90bc (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:13:04.972100  4291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2551af500d034f6096ef929e30795d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:04.972333  4362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a287c34d1b14502b5541f96aa8c90bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:04.975788  4290 tablet_service.cc:1511] Processing CreateTablet for tablet bd3699e5ca7e4fb7afaab54895c65b1f (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:13:04.976715  4215 tablet_service.cc:1511] Processing CreateTablet for tablet 9a69dca66f3a41e7a1a2bf40d5a5d641 (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:13:04.978143  4215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a69dca66f3a41e7a1a2bf40d5a5d641. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:04.979756  4365 tablet_service.cc:1511] Processing CreateTablet for tablet 9a69dca66f3a41e7a1a2bf40d5a5d641 (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:13:04.980202  4363 tablet_service.cc:1511] Processing CreateTablet for tablet bd3699e5ca7e4fb7afaab54895c65b1f (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:13:04.980651  4287 tablet_service.cc:1511] Processing CreateTablet for tablet 6ceea03a69bd4c5399479d732de74d50 (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:13:04.981093  4365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a69dca66f3a41e7a1a2bf40d5a5d641. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:04.990463  4363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3699e5ca7e4fb7afaab54895c65b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:04.990216  4289 tablet_service.cc:1511] Processing CreateTablet for tablet cea6e20ed83a43cdaf95392d95766700 (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:13:04.982347  4360 tablet_service.cc:1511] Processing CreateTablet for tablet 2551af500d034f6096ef929e30795d3d (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:13:04.992229  4360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2551af500d034f6096ef929e30795d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:04.982726  4214 tablet_service.cc:1511] Processing CreateTablet for tablet 2551af500d034f6096ef929e30795d3d (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:13:04.983937  4217 tablet_service.cc:1511] Processing CreateTablet for tablet bd3699e5ca7e4fb7afaab54895c65b1f (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:13:04.977200  4290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3699e5ca7e4fb7afaab54895c65b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:04.986495  4219 tablet_service.cc:1511] Processing CreateTablet for tablet 047764b4386349a198cb3ae4b599730f (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:13:04.984694  4367 tablet_service.cc:1511] Processing CreateTablet for tablet 6ceea03a69bd4c5399479d732de74d50 (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:13:04.989414  4212 tablet_service.cc:1511] Processing CreateTablet for tablet cea6e20ed83a43cdaf95392d95766700 (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:13:04.997534  4367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ceea03a69bd4c5399479d732de74d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:04.985580  4286 tablet_service.cc:1511] Processing CreateTablet for tablet 047764b4386349a198cb3ae4b599730f (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:13:04.987895  4364 tablet_service.cc:1511] Processing CreateTablet for tablet 047764b4386349a198cb3ae4b599730f (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:13:04.999259  4364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 047764b4386349a198cb3ae4b599730f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:04.985302  4218 tablet_service.cc:1511] Processing CreateTablet for tablet 6ceea03a69bd4c5399479d732de74d50 (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:13:05.002719  4214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2551af500d034f6096ef929e30795d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:05.006337  4288 tablet_service.cc:1511] Processing CreateTablet for tablet e72d13ac358342be97f27b246e85b802 (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:13:05.007537  4212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cea6e20ed83a43cdaf95392d95766700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:05.009858  4219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 047764b4386349a198cb3ae4b599730f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:05.013485  4218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ceea03a69bd4c5399479d732de74d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:05.011361  4211 tablet_service.cc:1511] Processing CreateTablet for tablet e72d13ac358342be97f27b246e85b802 (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:13:05.014638  4211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e72d13ac358342be97f27b246e85b802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:05.017634  4217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3699e5ca7e4fb7afaab54895c65b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:05.021852  4368 tablet_service.cc:1511] Processing CreateTablet for tablet cea6e20ed83a43cdaf95392d95766700 (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:13:05.023229  4368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cea6e20ed83a43cdaf95392d95766700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:05.025933  4366 tablet_service.cc:1511] Processing CreateTablet for tablet e72d13ac358342be97f27b246e85b802 (DEFAULT_TABLE table=soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:13:05.027702  4366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e72d13ac358342be97f27b246e85b802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:05.033569  4287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ceea03a69bd4c5399479d732de74d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:05.037465  4286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 047764b4386349a198cb3ae4b599730f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:05.086925  4520 tablet_bootstrap.cc:492] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee: Bootstrap starting.
I20260709 13:13:05.125906  4289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cea6e20ed83a43cdaf95392d95766700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:13:05.133495  4288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e72d13ac358342be97f27b246e85b802. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:13:05.148022  4175 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:13:05.178488  4521 tablet_bootstrap.cc:492] T b7b904c5d9654b758bb74c910a8a22c2 P 2a925808f54d42ed9cc086f846280594: Bootstrap starting.
I20260709 13:13:05.178539  4522 tablet_bootstrap.cc:492] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c: Bootstrap starting.
W20260709 13:13:05.185202  4267 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:37585, user_credentials={real_user=slave}} blocked reactor thread for 100237us
I20260709 13:13:05.206116  4521 tablet_bootstrap.cc:654] T b7b904c5d9654b758bb74c910a8a22c2 P 2a925808f54d42ed9cc086f846280594: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.214844  4522 tablet_bootstrap.cc:654] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c: Neither blocks nor log segments found. Creating new log.
W20260709 13:13:05.221092  4192 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:37585, user_credentials={real_user=slave}} blocked reactor thread for 133633us
I20260709 13:13:05.239485  4520 tablet_bootstrap.cc:654] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.307703  4521 tablet_bootstrap.cc:492] T b7b904c5d9654b758bb74c910a8a22c2 P 2a925808f54d42ed9cc086f846280594: No bootstrap required, opened a new log
I20260709 13:13:05.310967  4521 ts_tablet_manager.cc:1403] T b7b904c5d9654b758bb74c910a8a22c2 P 2a925808f54d42ed9cc086f846280594: Time spent bootstrapping tablet: real 0.133s	user 0.022s	sys 0.040s
W20260709 13:13:05.308809  4111 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.23.140.67:37585, user_credentials={real_user=slave}} blocked reactor thread for 43922.5us
I20260709 13:13:05.314282  4521 raft_consensus.cc:359] T b7b904c5d9654b758bb74c910a8a22c2 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.315017  4521 raft_consensus.cc:385] T b7b904c5d9654b758bb74c910a8a22c2 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.315285  4521 raft_consensus.cc:740] T b7b904c5d9654b758bb74c910a8a22c2 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.315984  4521 consensus_queue.cc:260] T b7b904c5d9654b758bb74c910a8a22c2 P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.318459  4521 ts_tablet_manager.cc:1434] T b7b904c5d9654b758bb74c910a8a22c2 P 2a925808f54d42ed9cc086f846280594: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.006s
I20260709 13:13:05.319518  4521 tablet_bootstrap.cc:492] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594: Bootstrap starting.
I20260709 13:13:05.310334  4520 tablet_bootstrap.cc:492] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee: No bootstrap required, opened a new log
I20260709 13:13:05.321455  4520 ts_tablet_manager.cc:1403] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee: Time spent bootstrapping tablet: real 0.235s	user 0.018s	sys 0.017s
I20260709 13:13:05.324585  4520 raft_consensus.cc:359] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.325273  4520 raft_consensus.cc:385] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.325562  4520 raft_consensus.cc:740] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.326238  4520 consensus_queue.cc:260] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.327122  4522 tablet_bootstrap.cc:492] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c: No bootstrap required, opened a new log
I20260709 13:13:05.327687  4522 ts_tablet_manager.cc:1403] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent bootstrapping tablet: real 0.150s	user 0.007s	sys 0.009s
I20260709 13:13:05.331331  4520 ts_tablet_manager.cc:1434] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 13:13:05.332399  4520 tablet_bootstrap.cc:492] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee: Bootstrap starting.
I20260709 13:13:05.338917  4520 tablet_bootstrap.cc:654] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.330979  4522 raft_consensus.cc:359] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.340770  4522 raft_consensus.cc:385] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.341109  4522 raft_consensus.cc:740] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.341993  4522 consensus_queue.cc:260] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.343101  4521 tablet_bootstrap.cc:654] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.344862  4520 tablet_bootstrap.cc:492] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee: No bootstrap required, opened a new log
I20260709 13:13:05.345433  4520 ts_tablet_manager.cc:1403] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee: Time spent bootstrapping tablet: real 0.013s	user 0.012s	sys 0.000s
I20260709 13:13:05.345806  4522 ts_tablet_manager.cc:1434] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.004s
I20260709 13:13:05.346725  4522 tablet_bootstrap.cc:492] T 6a287c34d1b14502b5541f96aa8c90bc P 6ff6192f285c425fb9d28bac1f0cea7c: Bootstrap starting.
I20260709 13:13:05.349004  4520 raft_consensus.cc:359] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.350020  4520 raft_consensus.cc:385] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.350378  4520 raft_consensus.cc:740] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.352257  4520 consensus_queue.cc:260] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.353221  4522 tablet_bootstrap.cc:654] T 6a287c34d1b14502b5541f96aa8c90bc P 6ff6192f285c425fb9d28bac1f0cea7c: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.355682  4520 ts_tablet_manager.cc:1434] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.001s
I20260709 13:13:05.356658  4520 tablet_bootstrap.cc:492] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee: Bootstrap starting.
I20260709 13:13:05.363627  4521 tablet_bootstrap.cc:492] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594: No bootstrap required, opened a new log
I20260709 13:13:05.363668  4520 tablet_bootstrap.cc:654] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.364424  4521 ts_tablet_manager.cc:1403] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594: Time spent bootstrapping tablet: real 0.045s	user 0.011s	sys 0.006s
I20260709 13:13:05.367952  4521 raft_consensus.cc:359] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.368897  4521 raft_consensus.cc:385] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.369451  4521 raft_consensus.cc:740] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.370368  4521 consensus_queue.cc:260] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.372892  4521 ts_tablet_manager.cc:1434] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.004s
I20260709 13:13:05.374915  4521 tablet_bootstrap.cc:492] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594: Bootstrap starting.
I20260709 13:13:05.378923  4522 tablet_bootstrap.cc:492] T 6a287c34d1b14502b5541f96aa8c90bc P 6ff6192f285c425fb9d28bac1f0cea7c: No bootstrap required, opened a new log
I20260709 13:13:05.379539  4522 ts_tablet_manager.cc:1403] T 6a287c34d1b14502b5541f96aa8c90bc P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent bootstrapping tablet: real 0.033s	user 0.009s	sys 0.020s
I20260709 13:13:05.382685  4522 raft_consensus.cc:359] T 6a287c34d1b14502b5541f96aa8c90bc P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.383441  4522 raft_consensus.cc:385] T 6a287c34d1b14502b5541f96aa8c90bc P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.383736  4522 raft_consensus.cc:740] T 6a287c34d1b14502b5541f96aa8c90bc P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.384516  4522 consensus_queue.cc:260] T 6a287c34d1b14502b5541f96aa8c90bc P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.385565  4521 tablet_bootstrap.cc:654] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.387217  4522 ts_tablet_manager.cc:1434] T 6a287c34d1b14502b5541f96aa8c90bc P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.004s
I20260709 13:13:05.388334  4522 tablet_bootstrap.cc:492] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c: Bootstrap starting.
I20260709 13:13:05.388628  4520 tablet_bootstrap.cc:492] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee: No bootstrap required, opened a new log
I20260709 13:13:05.389276  4520 ts_tablet_manager.cc:1403] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee: Time spent bootstrapping tablet: real 0.033s	user 0.011s	sys 0.009s
I20260709 13:13:05.392699  4520 raft_consensus.cc:359] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.393767  4520 raft_consensus.cc:385] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.394152  4520 raft_consensus.cc:740] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.395354  4520 consensus_queue.cc:260] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.397218  4522 tablet_bootstrap.cc:654] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.398723  4520 ts_tablet_manager.cc:1434] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 13:13:05.405737  4520 tablet_bootstrap.cc:492] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee: Bootstrap starting.
I20260709 13:13:05.406741  4522 tablet_bootstrap.cc:492] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c: No bootstrap required, opened a new log
I20260709 13:13:05.407218  4522 ts_tablet_manager.cc:1403] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent bootstrapping tablet: real 0.019s	user 0.008s	sys 0.004s
I20260709 13:13:05.410145  4522 raft_consensus.cc:359] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.410740  4522 raft_consensus.cc:385] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.410769  4521 tablet_bootstrap.cc:492] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594: No bootstrap required, opened a new log
I20260709 13:13:05.411051  4522 raft_consensus.cc:740] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.411336  4521 ts_tablet_manager.cc:1403] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594: Time spent bootstrapping tablet: real 0.037s	user 0.014s	sys 0.002s
I20260709 13:13:05.411860  4522 consensus_queue.cc:260] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.414610  4522 ts_tablet_manager.cc:1434] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.006s
I20260709 13:13:05.414886  4521 raft_consensus.cc:359] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.415680  4520 tablet_bootstrap.cc:654] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.415654  4521 raft_consensus.cc:385] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.416062  4522 tablet_bootstrap.cc:492] T b7b904c5d9654b758bb74c910a8a22c2 P 6ff6192f285c425fb9d28bac1f0cea7c: Bootstrap starting.
I20260709 13:13:05.417445  4521 raft_consensus.cc:740] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Initialized, Role: FOLLOWER
W20260709 13:13:05.418509  4255 tablet.cc:2406] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 13:13:05.418288  4521 consensus_queue.cc:260] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.420672  4521 ts_tablet_manager.cc:1434] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.003s
I20260709 13:13:05.421785  4521 tablet_bootstrap.cc:492] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594: Bootstrap starting.
I20260709 13:13:05.424901  4522 tablet_bootstrap.cc:654] T b7b904c5d9654b758bb74c910a8a22c2 P 6ff6192f285c425fb9d28bac1f0cea7c: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.431910  4527 raft_consensus.cc:493] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:05.432677  4527 raft_consensus.cc:515] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.436280  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2551af500d034f6096ef929e30795d3d" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:05.437309  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2551af500d034f6096ef929e30795d3d" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
W20260709 13:13:05.438674  4270 leader_election.cc:343] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:13:05.439381  4267 leader_election.cc:343] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:05.439956  4267 leader_election.cc:304] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c; no voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee
I20260709 13:13:05.440904  4518 raft_consensus.cc:2749] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:13:05.441902  4521 tablet_bootstrap.cc:654] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.440904  4527 leader_election.cc:290] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:05.448887  4522 tablet_bootstrap.cc:492] T b7b904c5d9654b758bb74c910a8a22c2 P 6ff6192f285c425fb9d28bac1f0cea7c: No bootstrap required, opened a new log
I20260709 13:13:05.449702  4522 ts_tablet_manager.cc:1403] T b7b904c5d9654b758bb74c910a8a22c2 P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent bootstrapping tablet: real 0.034s	user 0.008s	sys 0.012s
I20260709 13:13:05.452930  4524 raft_consensus.cc:493] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:05.453692  4524 raft_consensus.cc:515] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.453882  4522 raft_consensus.cc:359] T b7b904c5d9654b758bb74c910a8a22c2 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.456243  4522 raft_consensus.cc:385] T b7b904c5d9654b758bb74c910a8a22c2 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.456619  4522 raft_consensus.cc:740] T b7b904c5d9654b758bb74c910a8a22c2 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.457722  4302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ceea03a69bd4c5399479d732de74d50" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
W20260709 13:13:05.466321  4194 leader_election.cc:343] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:05.466903  4524 leader_election.cc:290] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:05.469161  4520 tablet_bootstrap.cc:492] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee: No bootstrap required, opened a new log
I20260709 13:13:05.469822  4520 ts_tablet_manager.cc:1403] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee: Time spent bootstrapping tablet: real 0.064s	user 0.024s	sys 0.034s
I20260709 13:13:05.457536  4522 consensus_queue.cc:260] T b7b904c5d9654b758bb74c910a8a22c2 P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.472915  4522 ts_tablet_manager.cc:1434] T b7b904c5d9654b758bb74c910a8a22c2 P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent starting tablet: real 0.023s	user 0.004s	sys 0.003s
I20260709 13:13:05.473321  4520 raft_consensus.cc:359] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.473875  4522 tablet_bootstrap.cc:492] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c: Bootstrap starting.
I20260709 13:13:05.474115  4520 raft_consensus.cc:385] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.474478  4520 raft_consensus.cc:740] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.475214  4520 consensus_queue.cc:260] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.458354  4521 tablet_bootstrap.cc:492] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594: No bootstrap required, opened a new log
I20260709 13:13:05.476760  4521 ts_tablet_manager.cc:1403] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594: Time spent bootstrapping tablet: real 0.055s	user 0.013s	sys 0.003s
I20260709 13:13:05.477746  4520 ts_tablet_manager.cc:1434] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.003s
I20260709 13:13:05.478817  4520 tablet_bootstrap.cc:492] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee: Bootstrap starting.
I20260709 13:13:05.460479  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ceea03a69bd4c5399479d732de74d50" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:05.479904  4521 raft_consensus.cc:359] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
W20260709 13:13:05.480377  4192 leader_election.cc:343] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:05.480661  4521 raft_consensus.cc:385] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.480857  4192 leader_election.cc:304] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba722a02e5c34752b5a164e6234797ee; no voters: 2a925808f54d42ed9cc086f846280594, 6ff6192f285c425fb9d28bac1f0cea7c
I20260709 13:13:05.486163  4520 tablet_bootstrap.cc:654] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.480986  4521 raft_consensus.cc:740] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.488355  4524 raft_consensus.cc:2749] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:13:05.493417  4521 consensus_queue.cc:260] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.495741  4522 tablet_bootstrap.cc:654] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.496053  4521 ts_tablet_manager.cc:1434] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 13:13:05.496208  4520 tablet_bootstrap.cc:492] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee: No bootstrap required, opened a new log
I20260709 13:13:05.497579  4520 ts_tablet_manager.cc:1403] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee: Time spent bootstrapping tablet: real 0.019s	user 0.011s	sys 0.002s
I20260709 13:13:05.497586  4521 tablet_bootstrap.cc:492] T 6ceea03a69bd4c5399479d732de74d50 P 2a925808f54d42ed9cc086f846280594: Bootstrap starting.
I20260709 13:13:05.500859  4520 raft_consensus.cc:359] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.501597  4520 raft_consensus.cc:385] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.501876  4520 raft_consensus.cc:740] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.502622  4520 consensus_queue.cc:260] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.505152  4520 ts_tablet_manager.cc:1434] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
I20260709 13:13:05.506356  4520 tablet_bootstrap.cc:492] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee: Bootstrap starting.
I20260709 13:13:05.510854  4521 tablet_bootstrap.cc:654] T 6ceea03a69bd4c5399479d732de74d50 P 2a925808f54d42ed9cc086f846280594: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.512953  4520 tablet_bootstrap.cc:654] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.518849  4524 raft_consensus.cc:493] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:05.519541  4524 raft_consensus.cc:515] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.524567  4302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a287c34d1b14502b5541f96aa8c90bc" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:05.525702  4302 raft_consensus.cc:2468] T 6a287c34d1b14502b5541f96aa8c90bc P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 0.
I20260709 13:13:05.527470  4194 leader_election.cc:304] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:05.528788  4505 raft_consensus.cc:2804] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:05.529294  4505 raft_consensus.cc:493] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:05.529702  4505 raft_consensus.cc:3060] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.533859  4524 leader_election.cc:290] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:05.537770  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a287c34d1b14502b5541f96aa8c90bc" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:05.537987  4518 raft_consensus.cc:493] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:05.538524  4518 raft_consensus.cc:515] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
W20260709 13:13:05.539237  4192 leader_election.cc:343] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:05.540756  4518 leader_election.cc:290] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:05.541602  4505 raft_consensus.cc:515] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.541683  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3699e5ca7e4fb7afaab54895c65b1f" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:05.543018  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3699e5ca7e4fb7afaab54895c65b1f" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
W20260709 13:13:05.543684  4270 leader_election.cc:343] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:13:05.544493  4267 leader_election.cc:343] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:05.544889  4267 leader_election.cc:304] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c; no voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee
I20260709 13:13:05.544999  4302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a287c34d1b14502b5541f96aa8c90bc" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:05.545864  4302 raft_consensus.cc:3060] T 6a287c34d1b14502b5541f96aa8c90bc P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.546118  4518 raft_consensus.cc:2749] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:13:05.556214  4522 tablet_bootstrap.cc:492] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c: No bootstrap required, opened a new log
I20260709 13:13:05.557025  4522 ts_tablet_manager.cc:1403] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent bootstrapping tablet: real 0.083s	user 0.026s	sys 0.032s
I20260709 13:13:05.557893  4505 leader_election.cc:290] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Requested vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:05.558297  4302 raft_consensus.cc:2468] T 6a287c34d1b14502b5541f96aa8c90bc P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 1.
I20260709 13:13:05.559093  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a287c34d1b14502b5541f96aa8c90bc" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:05.560004  4194 leader_election.cc:304] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
W20260709 13:13:05.560593  4192 leader_election.cc:343] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:05.560942  4505 raft_consensus.cc:2804] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:05.561193  4521 tablet_bootstrap.cc:492] T 6ceea03a69bd4c5399479d732de74d50 P 2a925808f54d42ed9cc086f846280594: No bootstrap required, opened a new log
I20260709 13:13:05.560969  4522 raft_consensus.cc:359] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.561851  4505 raft_consensus.cc:697] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Becoming Leader. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Running, Role: LEADER
I20260709 13:13:05.561981  4521 ts_tablet_manager.cc:1403] T 6ceea03a69bd4c5399479d732de74d50 P 2a925808f54d42ed9cc086f846280594: Time spent bootstrapping tablet: real 0.065s	user 0.015s	sys 0.003s
I20260709 13:13:05.562173  4522 raft_consensus.cc:385] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.562780  4522 raft_consensus.cc:740] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.562937  4505 consensus_queue.cc:237] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.563966  4520 tablet_bootstrap.cc:492] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee: No bootstrap required, opened a new log
I20260709 13:13:05.563606  4522 consensus_queue.cc:260] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.564531  4520 ts_tablet_manager.cc:1403] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee: Time spent bootstrapping tablet: real 0.058s	user 0.020s	sys 0.025s
I20260709 13:13:05.565409  4521 raft_consensus.cc:359] T 6ceea03a69bd4c5399479d732de74d50 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.566145  4521 raft_consensus.cc:385] T 6ceea03a69bd4c5399479d732de74d50 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.566429  4521 raft_consensus.cc:740] T 6ceea03a69bd4c5399479d732de74d50 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.566715  4522 ts_tablet_manager.cc:1434] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.004s
I20260709 13:13:05.567451  4521 consensus_queue.cc:260] T 6ceea03a69bd4c5399479d732de74d50 P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.569164  4524 raft_consensus.cc:493] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:05.569628  4524 raft_consensus.cc:515] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.570030  4520 raft_consensus.cc:359] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.570803  4520 raft_consensus.cc:385] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.571107  4520 raft_consensus.cc:740] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.571650  4524 leader_election.cc:290] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:05.572710  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b904c5d9654b758bb74c910a8a22c2" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:05.573040  4522 tablet_bootstrap.cc:492] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c: Bootstrap starting.
I20260709 13:13:05.573179  4302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b904c5d9654b758bb74c910a8a22c2" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:05.573964  4302 raft_consensus.cc:2468] T b7b904c5d9654b758bb74c910a8a22c2 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 0.
I20260709 13:13:05.575682  4521 ts_tablet_manager.cc:1434] T 6ceea03a69bd4c5399479d732de74d50 P 2a925808f54d42ed9cc086f846280594: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 13:13:05.573382  4378 raft_consensus.cc:2468] T b7b904c5d9654b758bb74c910a8a22c2 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 0.
I20260709 13:13:05.576103  4194 leader_election.cc:304] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:05.576874  4521 tablet_bootstrap.cc:492] T 2551af500d034f6096ef929e30795d3d P 2a925808f54d42ed9cc086f846280594: Bootstrap starting.
I20260709 13:13:05.577167  4524 raft_consensus.cc:2804] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:05.577541  4524 raft_consensus.cc:493] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:05.577939  4524 raft_consensus.cc:3060] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.578033  4520 consensus_queue.cc:260] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.580580  4520 ts_tablet_manager.cc:1434] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.001s
I20260709 13:13:05.581995  4520 tablet_bootstrap.cc:492] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee: Bootstrap starting.
I20260709 13:13:05.584086  4521 tablet_bootstrap.cc:654] T 2551af500d034f6096ef929e30795d3d P 2a925808f54d42ed9cc086f846280594: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.585716  4522 tablet_bootstrap.cc:654] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.589056  4520 tablet_bootstrap.cc:654] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.599818  4522 tablet_bootstrap.cc:492] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c: No bootstrap required, opened a new log
I20260709 13:13:05.600726  4522 ts_tablet_manager.cc:1403] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent bootstrapping tablet: real 0.028s	user 0.008s	sys 0.009s
I20260709 13:13:05.604044  4524 raft_consensus.cc:515] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.603600  4127 catalog_manager.cc:5697] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ba722a02e5c34752b5a164e6234797ee (127.23.140.65). New cstate: current_term: 1 leader_uuid: "ba722a02e5c34752b5a164e6234797ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:05.606711  4302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b904c5d9654b758bb74c910a8a22c2" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:05.608172  4302 raft_consensus.cc:3060] T b7b904c5d9654b758bb74c910a8a22c2 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.613497  4521 tablet_bootstrap.cc:492] T 2551af500d034f6096ef929e30795d3d P 2a925808f54d42ed9cc086f846280594: No bootstrap required, opened a new log
I20260709 13:13:05.614133  4521 ts_tablet_manager.cc:1403] T 2551af500d034f6096ef929e30795d3d P 2a925808f54d42ed9cc086f846280594: Time spent bootstrapping tablet: real 0.037s	user 0.017s	sys 0.003s
I20260709 13:13:05.615088  4522 raft_consensus.cc:359] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.615962  4522 raft_consensus.cc:385] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.616272  4522 raft_consensus.cc:740] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.617377  4524 leader_election.cc:290] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Requested vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:05.617163  4522 consensus_queue.cc:260] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.619427  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b904c5d9654b758bb74c910a8a22c2" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:05.620013  4302 raft_consensus.cc:2468] T b7b904c5d9654b758bb74c910a8a22c2 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 1.
I20260709 13:13:05.620080  4378 raft_consensus.cc:3060] T b7b904c5d9654b758bb74c910a8a22c2 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.621287  4518 raft_consensus.cc:493] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:05.621891  4518 raft_consensus.cc:515] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.622474  4194 leader_election.cc:304] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:05.623764  4523 raft_consensus.cc:2804] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:05.624311  4523 raft_consensus.cc:697] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Becoming Leader. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Running, Role: LEADER
I20260709 13:13:05.624383  4518 leader_election.cc:290] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:05.613497  4520 tablet_bootstrap.cc:492] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee: No bootstrap required, opened a new log
I20260709 13:13:05.625895  4520 ts_tablet_manager.cc:1403] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee: Time spent bootstrapping tablet: real 0.044s	user 0.016s	sys 0.013s
I20260709 13:13:05.625368  4523 consensus_queue.cc:237] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.626765  4522 ts_tablet_manager.cc:1434] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.000s
I20260709 13:13:05.627990  4522 tablet_bootstrap.cc:492] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c: Bootstrap starting.
I20260709 13:13:05.629901  4520 raft_consensus.cc:359] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.630857  4520 raft_consensus.cc:385] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.631230  4520 raft_consensus.cc:740] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.632169  4520 consensus_queue.cc:260] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.632774  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ceea03a69bd4c5399479d732de74d50" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:05.633567  4229 raft_consensus.cc:2468] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 0.
I20260709 13:13:05.635243  4520 ts_tablet_manager.cc:1434] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.001s
I20260709 13:13:05.636583  4520 tablet_bootstrap.cc:492] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee: Bootstrap starting.
I20260709 13:13:05.637863  4270 leader_election.cc:304] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:05.638619  4376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ceea03a69bd4c5399479d732de74d50" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:05.639451  4376 raft_consensus.cc:2468] T 6ceea03a69bd4c5399479d732de74d50 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 0.
I20260709 13:13:05.640380  4518 raft_consensus.cc:2804] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:05.640847  4518 raft_consensus.cc:493] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:05.645200  4520 tablet_bootstrap.cc:654] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.645455  4518 raft_consensus.cc:3060] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.617937  4521 raft_consensus.cc:359] T 2551af500d034f6096ef929e30795d3d P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.648203  4521 raft_consensus.cc:385] T 2551af500d034f6096ef929e30795d3d P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.652637  4521 raft_consensus.cc:740] T 2551af500d034f6096ef929e30795d3d P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.656209  4521 consensus_queue.cc:260] T 2551af500d034f6096ef929e30795d3d P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.659122  4521 ts_tablet_manager.cc:1434] T 2551af500d034f6096ef929e30795d3d P 2a925808f54d42ed9cc086f846280594: Time spent starting tablet: real 0.045s	user 0.006s	sys 0.000s
I20260709 13:13:05.662441  4521 tablet_bootstrap.cc:492] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594: Bootstrap starting.
I20260709 13:13:05.652086  4378 raft_consensus.cc:2468] T b7b904c5d9654b758bb74c910a8a22c2 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 1.
I20260709 13:13:05.675495  4522 tablet_bootstrap.cc:654] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.660583  4518 raft_consensus.cc:515] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.679263  4518 leader_election.cc:290] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:05.683569  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ceea03a69bd4c5399479d732de74d50" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:05.684391  4229 raft_consensus.cc:3060] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.686169  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ceea03a69bd4c5399479d732de74d50" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:05.686570  4539 raft_consensus.cc:493] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:05.687021  4378 raft_consensus.cc:3060] T 6ceea03a69bd4c5399479d732de74d50 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.687184  4539 raft_consensus.cc:515] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.691406  4303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "047764b4386349a198cb3ae4b599730f" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:05.692122  4303 raft_consensus.cc:2468] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 0.
I20260709 13:13:05.693907  4343 leader_election.cc:304] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, 6ff6192f285c425fb9d28bac1f0cea7c; no voters: 
I20260709 13:13:05.694967  4431 raft_consensus.cc:2804] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:05.695379  4431 raft_consensus.cc:493] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:05.695746  4431 raft_consensus.cc:3060] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.698575  4228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "047764b4386349a198cb3ae4b599730f" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:05.698699  4378 raft_consensus.cc:2468] T 6ceea03a69bd4c5399479d732de74d50 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 1.
I20260709 13:13:05.699491  4228 raft_consensus.cc:2468] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 0.
I20260709 13:13:05.701962  4267 leader_election.cc:304] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, 6ff6192f285c425fb9d28bac1f0cea7c; no voters: 
I20260709 13:13:05.705219  4520 tablet_bootstrap.cc:492] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee: No bootstrap required, opened a new log
I20260709 13:13:05.705353  4518 raft_consensus.cc:2804] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:05.705829  4229 raft_consensus.cc:2468] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 1.
I20260709 13:13:05.706010  4520 ts_tablet_manager.cc:1403] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee: Time spent bootstrapping tablet: real 0.070s	user 0.020s	sys 0.044s
I20260709 13:13:05.710135  4520 raft_consensus.cc:359] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.711035  4520 raft_consensus.cc:385] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.711172  4521 tablet_bootstrap.cc:654] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.711416  4520 raft_consensus.cc:740] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.712373  4520 consensus_queue.cc:260] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.714488  4520 ts_tablet_manager.cc:1434] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:13:05.720160  4520 tablet_bootstrap.cc:492] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee: Bootstrap starting.
I20260709 13:13:05.706138  4518 raft_consensus.cc:697] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 LEADER]: Becoming Leader. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Running, Role: LEADER
I20260709 13:13:05.701665  4539 leader_election.cc:290] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079)
I20260709 13:13:05.724246  4518 consensus_queue.cc:237] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.728259  4553 raft_consensus.cc:493] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:05.728814  4520 tablet_bootstrap.cc:654] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.731653  4431 raft_consensus.cc:515] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.738196  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "047764b4386349a198cb3ae4b599730f" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:05.739089  4229 raft_consensus.cc:3060] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.739784  4522 tablet_bootstrap.cc:492] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c: No bootstrap required, opened a new log
I20260709 13:13:05.740489  4522 ts_tablet_manager.cc:1403] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent bootstrapping tablet: real 0.113s	user 0.026s	sys 0.024s
I20260709 13:13:05.742637  4521 tablet_bootstrap.cc:492] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594: No bootstrap required, opened a new log
I20260709 13:13:05.742748  4303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "047764b4386349a198cb3ae4b599730f" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:05.743341  4521 ts_tablet_manager.cc:1403] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594: Time spent bootstrapping tablet: real 0.081s	user 0.008s	sys 0.009s
I20260709 13:13:05.743518  4303 raft_consensus.cc:3060] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.754717  4229 raft_consensus.cc:2468] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 1.
I20260709 13:13:05.757496  4431 leader_election.cc:290] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079)
I20260709 13:13:05.758380  4522 raft_consensus.cc:359] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.761024  4522 raft_consensus.cc:385] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.761337  4522 raft_consensus.cc:740] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.762269  4520 tablet_bootstrap.cc:492] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee: No bootstrap required, opened a new log
I20260709 13:13:05.763319  4520 ts_tablet_manager.cc:1403] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee: Time spent bootstrapping tablet: real 0.048s	user 0.022s	sys 0.000s
I20260709 13:13:05.729466  4553 raft_consensus.cc:515] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.766727  4344 leader_election.cc:304] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:05.768405  4522 consensus_queue.cc:260] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.770637  4124 catalog_manager.cc:5697] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66). New cstate: current_term: 1 leader_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:05.771654  4522 ts_tablet_manager.cc:1434] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent starting tablet: real 0.031s	user 0.008s	sys 0.004s
I20260709 13:13:05.771473  4520 raft_consensus.cc:359] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.772226  4520 raft_consensus.cc:385] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.768534  4523 raft_consensus.cc:493] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:05.772819  4522 tablet_bootstrap.cc:492] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c: Bootstrap starting.
I20260709 13:13:05.773083  4523 raft_consensus.cc:515] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.775141  4523 leader_election.cc:290] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:05.778328  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a69dca66f3a41e7a1a2bf40d5a5d641" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:05.779021  4304 raft_consensus.cc:2468] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 0.
I20260709 13:13:05.779349  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a69dca66f3a41e7a1a2bf40d5a5d641" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:05.780393  4522 tablet_bootstrap.cc:654] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.780292  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a69dca66f3a41e7a1a2bf40d5a5d641" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:05.780995  4378 raft_consensus.cc:2468] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 0.
I20260709 13:13:05.781291  4431 raft_consensus.cc:2804] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:05.781824  4431 raft_consensus.cc:697] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Becoming Leader. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Running, Role: LEADER
I20260709 13:13:05.782469  4192 leader_election.cc:304] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:05.782670  4431 consensus_queue.cc:237] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.780959  4127 catalog_manager.cc:5697] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ba722a02e5c34752b5a164e6234797ee (127.23.140.65). New cstate: current_term: 1 leader_uuid: "ba722a02e5c34752b5a164e6234797ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:05.772505  4520 raft_consensus.cc:740] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.786145  4520 consensus_queue.cc:260] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.788134  4520 ts_tablet_manager.cc:1434] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee: Time spent starting tablet: real 0.024s	user 0.004s	sys 0.001s
I20260709 13:13:05.788630  4524 raft_consensus.cc:493] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:05.789206  4524 raft_consensus.cc:515] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.767144  4521 raft_consensus.cc:359] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.791357  4524 leader_election.cc:290] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:05.792737  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e72d13ac358342be97f27b246e85b802" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
W20260709 13:13:05.796065  4194 leader_election.cc:343] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:05.794111  4303 raft_consensus.cc:2468] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 1.
I20260709 13:13:05.794427  4377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e72d13ac358342be97f27b246e85b802" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:05.796231  4521 raft_consensus.cc:385] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.798563  4521 raft_consensus.cc:740] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Initialized, Role: FOLLOWER
W20260709 13:13:05.799916  4192 leader_election.cc:343] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:05.800547  4192 leader_election.cc:304] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba722a02e5c34752b5a164e6234797ee; no voters: 2a925808f54d42ed9cc086f846280594, 6ff6192f285c425fb9d28bac1f0cea7c
I20260709 13:13:05.799574  4521 consensus_queue.cc:260] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.801954  4523 raft_consensus.cc:2749] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:13:05.809674  4521 ts_tablet_manager.cc:1434] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594: Time spent starting tablet: real 0.066s	user 0.011s	sys 0.000s
I20260709 13:13:05.810834  4521 tablet_bootstrap.cc:492] T bd3699e5ca7e4fb7afaab54895c65b1f P 2a925808f54d42ed9cc086f846280594: Bootstrap starting.
I20260709 13:13:05.815163  4229 raft_consensus.cc:2468] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 0.
I20260709 13:13:05.816170  4519 raft_consensus.cc:2804] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:05.817605  4553 leader_election.cc:290] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079)
I20260709 13:13:05.818022  4519 raft_consensus.cc:493] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:05.818457  4519 raft_consensus.cc:3060] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.818845  4521 tablet_bootstrap.cc:654] T bd3699e5ca7e4fb7afaab54895c65b1f P 2a925808f54d42ed9cc086f846280594: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.822203  4344 leader_election.cc:304] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:05.822939  4553 raft_consensus.cc:2804] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:05.823352  4553 raft_consensus.cc:493] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:05.823675  4553 raft_consensus.cc:3060] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.824213  4303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a69dca66f3a41e7a1a2bf40d5a5d641" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:05.824761  4303 raft_consensus.cc:2468] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 0.
I20260709 13:13:05.827407  4125 catalog_manager.cc:5697] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a925808f54d42ed9cc086f846280594 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "2a925808f54d42ed9cc086f846280594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: HEALTHY } } }
I20260709 13:13:05.828576  4523 raft_consensus.cc:493] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:05.829087  4523 raft_consensus.cc:515] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.831766  4553 raft_consensus.cc:515] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.834421  4303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2551af500d034f6096ef929e30795d3d" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:05.835072  4303 raft_consensus.cc:2468] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 0.
I20260709 13:13:05.836503  4194 leader_election.cc:304] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:05.836848  4522 tablet_bootstrap.cc:492] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c: No bootstrap required, opened a new log
I20260709 13:13:05.837363  4522 ts_tablet_manager.cc:1403] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent bootstrapping tablet: real 0.065s	user 0.025s	sys 0.008s
I20260709 13:13:05.837370  4524 raft_consensus.cc:2804] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:05.837857  4524 raft_consensus.cc:493] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:05.838198  4524 raft_consensus.cc:3060] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.841010  4553 leader_election.cc:290] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079)
I20260709 13:13:05.841687  4522 raft_consensus.cc:359] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.842337  4522 raft_consensus.cc:385] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.842602  4522 raft_consensus.cc:740] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.842882  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a69dca66f3a41e7a1a2bf40d5a5d641" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:05.843218  4522 consensus_queue.cc:260] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.845847  4522 ts_tablet_manager.cc:1434] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.001s
I20260709 13:13:05.845970  4303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a69dca66f3a41e7a1a2bf40d5a5d641" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:05.846591  4303 raft_consensus.cc:3060] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.846786  4522 tablet_bootstrap.cc:492] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c: Bootstrap starting.
I20260709 13:13:05.850857  4505 raft_consensus.cc:493] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:05.851358  4505 raft_consensus.cc:515] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.853348  4505 leader_election.cc:290] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:05.856261  4522 tablet_bootstrap.cc:654] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.859508  4524 raft_consensus.cc:515] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.862608  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cea6e20ed83a43cdaf95392d95766700" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:05.863016  4523 leader_election.cc:290] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:05.861634  4377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cea6e20ed83a43cdaf95392d95766700" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:05.864094  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2551af500d034f6096ef929e30795d3d" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:05.865533  4377 raft_consensus.cc:2468] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 0.
I20260709 13:13:05.865546  4378 raft_consensus.cc:2468] T 2551af500d034f6096ef929e30795d3d P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 0.
W20260709 13:13:05.864741  4194 leader_election.cc:343] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:05.867766  4192 leader_election.cc:304] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee; no voters: 6ff6192f285c425fb9d28bac1f0cea7c
I20260709 13:13:05.868214  4519 raft_consensus.cc:515] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.868798  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2551af500d034f6096ef929e30795d3d" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:05.868986  4523 raft_consensus.cc:2804] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:05.870751  4523 raft_consensus.cc:493] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:05.870819  4304 raft_consensus.cc:3060] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.871134  4523 raft_consensus.cc:3060] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.871440  4377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2551af500d034f6096ef929e30795d3d" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:05.872138  4377 raft_consensus.cc:3060] T 2551af500d034f6096ef929e30795d3d P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.873345  4524 leader_election.cc:290] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Requested vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:05.881152  4377 raft_consensus.cc:2468] T 2551af500d034f6096ef929e30795d3d P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 1.
I20260709 13:13:05.882959  4192 leader_election.cc:304] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:05.883050  4523 raft_consensus.cc:515] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.883915  4524 raft_consensus.cc:2804] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:05.870015  4303 raft_consensus.cc:2468] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 1.
I20260709 13:13:05.884469  4524 raft_consensus.cc:697] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Becoming Leader. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Running, Role: LEADER
I20260709 13:13:05.886364  4343 leader_election.cc:304] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, 6ff6192f285c425fb9d28bac1f0cea7c; no voters: 
I20260709 13:13:05.887917  4523 leader_election.cc:290] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Requested vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:05.889192  4519 leader_election.cc:290] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Requested vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:05.889530  4302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a69dca66f3a41e7a1a2bf40d5a5d641" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:05.891495  4302 raft_consensus.cc:2393] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ba722a02e5c34752b5a164e6234797ee in current term 1: Already voted for candidate 2a925808f54d42ed9cc086f846280594 in this term.
I20260709 13:13:05.886494  4229 raft_consensus.cc:2393] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a925808f54d42ed9cc086f846280594 in current term 1: Already voted for candidate ba722a02e5c34752b5a164e6234797ee in this term.
I20260709 13:13:05.892490  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cea6e20ed83a43cdaf95392d95766700" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:05.893323  4378 raft_consensus.cc:3060] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.894583  4523 raft_consensus.cc:493] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:05.895040  4523 raft_consensus.cc:515] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.897020  4523 leader_election.cc:290] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:05.898427  4376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a69dca66f3a41e7a1a2bf40d5a5d641" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:05.899284  4302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3699e5ca7e4fb7afaab54895c65b1f" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:05.900472  4302 raft_consensus.cc:2468] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 0.
I20260709 13:13:05.901439  4375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3699e5ca7e4fb7afaab54895c65b1f" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:05.902354  4194 leader_election.cc:304] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:05.890331  4304 raft_consensus.cc:2468] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 1.
W20260709 13:13:05.903322  4192 leader_election.cc:343] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:05.887790  4553 raft_consensus.cc:2804] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:05.903395  4523 raft_consensus.cc:2804] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:05.911128  4378 raft_consensus.cc:2468] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 1.
I20260709 13:13:05.909710  4553 raft_consensus.cc:697] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Becoming Leader. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Running, Role: LEADER
I20260709 13:13:05.906040  4523 raft_consensus.cc:493] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:05.913635  4523 raft_consensus.cc:3060] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.913049  4553 consensus_queue.cc:237] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.913811  4192 leader_election.cc:304] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:05.914918  4524 consensus_queue.cc:237] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.917609  4303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cea6e20ed83a43cdaf95392d95766700" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
W20260709 13:13:05.918859  4194 leader_election.cc:343] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:13:05.916615  4519 raft_consensus.cc:2804] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:05.921674  4519 raft_consensus.cc:697] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Becoming Leader. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Running, Role: LEADER
I20260709 13:13:05.922889  4519 consensus_queue.cc:237] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.933396  4523 raft_consensus.cc:515] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.934810  4522 tablet_bootstrap.cc:492] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c: No bootstrap required, opened a new log
I20260709 13:13:05.935108  4521 tablet_bootstrap.cc:492] T bd3699e5ca7e4fb7afaab54895c65b1f P 2a925808f54d42ed9cc086f846280594: No bootstrap required, opened a new log
I20260709 13:13:05.935453  4522 ts_tablet_manager.cc:1403] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent bootstrapping tablet: real 0.089s	user 0.049s	sys 0.029s
I20260709 13:13:05.935606  4521 ts_tablet_manager.cc:1403] T bd3699e5ca7e4fb7afaab54895c65b1f P 2a925808f54d42ed9cc086f846280594: Time spent bootstrapping tablet: real 0.125s	user 0.023s	sys 0.048s
I20260709 13:13:05.936306  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3699e5ca7e4fb7afaab54895c65b1f" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:05.936935  4304 raft_consensus.cc:3060] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.938741  4521 raft_consensus.cc:359] T bd3699e5ca7e4fb7afaab54895c65b1f P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.939471  4521 raft_consensus.cc:385] T bd3699e5ca7e4fb7afaab54895c65b1f P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.939731  4521 raft_consensus.cc:740] T bd3699e5ca7e4fb7afaab54895c65b1f P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.940510  4521 consensus_queue.cc:260] T bd3699e5ca7e4fb7afaab54895c65b1f P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.941854  4522 raft_consensus.cc:359] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.942685  4522 raft_consensus.cc:385] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.943008  4522 raft_consensus.cc:740] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.943063  4521 ts_tablet_manager.cc:1434] T bd3699e5ca7e4fb7afaab54895c65b1f P 2a925808f54d42ed9cc086f846280594: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:13:05.944414  4521 tablet_bootstrap.cc:492] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594: Bootstrap starting.
I20260709 13:13:05.943992  4522 consensus_queue.cc:260] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.947113  4522 ts_tablet_manager.cc:1434] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20260709 13:13:05.952139  4523 leader_election.cc:290] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Requested vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:05.954218  4304 raft_consensus.cc:2468] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 1.
I20260709 13:13:05.954566  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3699e5ca7e4fb7afaab54895c65b1f" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:05.955175  4378 raft_consensus.cc:3060] T bd3699e5ca7e4fb7afaab54895c65b1f P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.955438  4194 leader_election.cc:304] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:05.956276  4523 raft_consensus.cc:2804] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:05.956580  4192 leader_election.cc:304] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba722a02e5c34752b5a164e6234797ee; no voters: 2a925808f54d42ed9cc086f846280594, 6ff6192f285c425fb9d28bac1f0cea7c
I20260709 13:13:05.957897  4519 raft_consensus.cc:2749] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:13:05.956699  4523 raft_consensus.cc:697] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Becoming Leader. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Running, Role: LEADER
I20260709 13:13:05.959103  4521 tablet_bootstrap.cc:654] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594: Neither blocks nor log segments found. Creating new log.
I20260709 13:13:05.953840  4124 catalog_manager.cc:5697] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a925808f54d42ed9cc086f846280594 (127.23.140.67). New cstate: current_term: 1 leader_uuid: "2a925808f54d42ed9cc086f846280594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: HEALTHY } } }
I20260709 13:13:05.959054  4523 consensus_queue.cc:237] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.968591  4378 raft_consensus.cc:2468] T bd3699e5ca7e4fb7afaab54895c65b1f P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 1.
I20260709 13:13:05.973192  4125 catalog_manager.cc:5697] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ba722a02e5c34752b5a164e6234797ee (127.23.140.65). New cstate: current_term: 1 leader_uuid: "ba722a02e5c34752b5a164e6234797ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:05.973994  4521 tablet_bootstrap.cc:492] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594: No bootstrap required, opened a new log
I20260709 13:13:05.974588  4521 ts_tablet_manager.cc:1403] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594: Time spent bootstrapping tablet: real 0.030s	user 0.015s	sys 0.005s
I20260709 13:13:05.977350  4521 raft_consensus.cc:359] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.977962  4521 raft_consensus.cc:385] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:13:05.978214  4521 raft_consensus.cc:740] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Initialized, Role: FOLLOWER
I20260709 13:13:05.978848  4521 consensus_queue.cc:260] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.980667  4521 ts_tablet_manager.cc:1434] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.001s
I20260709 13:13:05.983879  4431 raft_consensus.cc:493] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:05.984409  4431 raft_consensus.cc:515] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:05.986416  4431 leader_election.cc:290] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079)
I20260709 13:13:05.987102  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a287c34d1b14502b5541f96aa8c90bc" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:05.989602  4304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a287c34d1b14502b5541f96aa8c90bc" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" is_pre_election: true
I20260709 13:13:05.990227  4304 raft_consensus.cc:2393] T 6a287c34d1b14502b5541f96aa8c90bc P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2a925808f54d42ed9cc086f846280594 in current term 1: Already voted for candidate ba722a02e5c34752b5a164e6234797ee in this term.
I20260709 13:13:05.991096  4343 leader_election.cc:304] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a925808f54d42ed9cc086f846280594; no voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee
I20260709 13:13:05.991752  4431 raft_consensus.cc:3060] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:05.991606  4125 catalog_manager.cc:5697] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ba722a02e5c34752b5a164e6234797ee (127.23.140.65). New cstate: current_term: 1 leader_uuid: "ba722a02e5c34752b5a164e6234797ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:05.993314  4125 catalog_manager.cc:5697] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ba722a02e5c34752b5a164e6234797ee (127.23.140.65). New cstate: current_term: 1 leader_uuid: "ba722a02e5c34752b5a164e6234797ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:05.997989  4431 raft_consensus.cc:2749] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:13:06.047001  4518 raft_consensus.cc:493] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:06.047405  4518 raft_consensus.cc:515] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.048976  4518 leader_election.cc:290] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:06.049885  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e72d13ac358342be97f27b246e85b802" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:06.050071  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e72d13ac358342be97f27b246e85b802" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:06.050678  4229 raft_consensus.cc:2468] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 0.
I20260709 13:13:06.050741  4378 raft_consensus.cc:2468] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 0.
I20260709 13:13:06.051681  4270 leader_election.cc:304] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:06.052322  4518 raft_consensus.cc:2804] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:13:06.052699  4518 raft_consensus.cc:493] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:13:06.052969  4518 raft_consensus.cc:3060] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:06.057845  4518 raft_consensus.cc:515] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.059497  4518 leader_election.cc:290] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:06.060232  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e72d13ac358342be97f27b246e85b802" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:06.060580  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e72d13ac358342be97f27b246e85b802" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:06.060842  4229 raft_consensus.cc:3060] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:06.061138  4378 raft_consensus.cc:3060] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:06.067811  4229 raft_consensus.cc:2468] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 1.
I20260709 13:13:06.068739  4378 raft_consensus.cc:2468] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 1.
I20260709 13:13:06.068811  4270 leader_election.cc:304] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:06.070065  4518 raft_consensus.cc:2804] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:13:06.070566  4518 raft_consensus.cc:697] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 LEADER]: Becoming Leader. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Running, Role: LEADER
I20260709 13:13:06.071269  4518 consensus_queue.cc:237] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.077865  4523 consensus_queue.cc:1048] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:06.081494  4125 catalog_manager.cc:5697] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66). New cstate: current_term: 1 leader_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:06.093278  4505 consensus_queue.cc:1048] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:06.105010  4518 raft_consensus.cc:493] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:13:06.105532  4518 raft_consensus.cc:515] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.108400  4518 leader_election.cc:290] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:06.108557  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cea6e20ed83a43cdaf95392d95766700" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a925808f54d42ed9cc086f846280594" is_pre_election: true
I20260709 13:13:06.108994  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cea6e20ed83a43cdaf95392d95766700" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba722a02e5c34752b5a164e6234797ee" is_pre_election: true
I20260709 13:13:06.109320  4378 raft_consensus.cc:2393] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ff6192f285c425fb9d28bac1f0cea7c in current term 1: Already voted for candidate ba722a02e5c34752b5a164e6234797ee in this term.
I20260709 13:13:06.110496  4267 leader_election.cc:304] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c; no voters: 2a925808f54d42ed9cc086f846280594, ba722a02e5c34752b5a164e6234797ee
I20260709 13:13:06.111176  4518 raft_consensus.cc:3060] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:13:06.116483  4518 raft_consensus.cc:2749] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:13:06.128149  4523 consensus_queue.cc:1048] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:06.137444  4505 consensus_queue.cc:1048] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:06.218278  4518 consensus_queue.cc:1048] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:06.230181  4527 consensus_queue.cc:1048] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:06.294131  4378 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9a69dca66f3a41e7a1a2bf40d5a5d641"
dest_uuid: "2a925808f54d42ed9cc086f846280594"
mode: GRACEFUL
new_leader_uuid: "ba722a02e5c34752b5a164e6234797ee"
 from {username='slave'} at 127.0.0.1:45142
I20260709 13:13:06.294802  4378 raft_consensus.cc:606] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Received request to transfer leadership to TS ba722a02e5c34752b5a164e6234797ee
I20260709 13:13:06.301745  4229 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bd3699e5ca7e4fb7afaab54895c65b1f"
dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
mode: GRACEFUL
new_leader_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
 from {username='slave'} at 127.0.0.1:60070
I20260709 13:13:06.302232  4229 raft_consensus.cc:606] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Received request to transfer leadership to TS 6ff6192f285c425fb9d28bac1f0cea7c
I20260709 13:13:06.309020  4378 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "047764b4386349a198cb3ae4b599730f"
dest_uuid: "2a925808f54d42ed9cc086f846280594"
mode: GRACEFUL
new_leader_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
 from {username='slave'} at 127.0.0.1:45142
I20260709 13:13:06.309533  4378 raft_consensus.cc:606] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Received request to transfer leadership to TS 6ff6192f285c425fb9d28bac1f0cea7c
I20260709 13:13:06.313747  4229 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cea6e20ed83a43cdaf95392d95766700"
dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
mode: GRACEFUL
new_leader_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
 from {username='slave'} at 127.0.0.1:60070
I20260709 13:13:06.314222  4229 raft_consensus.cc:606] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Received request to transfer leadership to TS 6ff6192f285c425fb9d28bac1f0cea7c
I20260709 13:13:06.318372  4431 consensus_queue.cc:1048] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:06.318989  4303 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e72d13ac358342be97f27b246e85b802"
dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
mode: GRACEFUL
new_leader_uuid: "2a925808f54d42ed9cc086f846280594"
 from {username='slave'} at 127.0.0.1:47852
I20260709 13:13:06.319576  4303 raft_consensus.cc:606] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 LEADER]: Received request to transfer leadership to TS 2a925808f54d42ed9cc086f846280594
I20260709 13:13:06.331419  4580 consensus_queue.cc:1048] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:06.335938  4524 consensus_queue.cc:1048] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:06.345908  4580 raft_consensus.cc:993] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594: : Instructing follower 6ff6192f285c425fb9d28bac1f0cea7c to start an election
I20260709 13:13:06.346230  4505 raft_consensus.cc:993] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee: : Instructing follower 6ff6192f285c425fb9d28bac1f0cea7c to start an election
I20260709 13:13:06.346376  4580 raft_consensus.cc:1081] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Signalling peer 6ff6192f285c425fb9d28bac1f0cea7c to start an election
I20260709 13:13:06.346755  4505 raft_consensus.cc:1081] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Signalling peer 6ff6192f285c425fb9d28bac1f0cea7c to start an election
I20260709 13:13:06.349076  4303 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "047764b4386349a198cb3ae4b599730f"
dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
 from {username='slave'} at 127.0.0.1:47850
I20260709 13:13:06.349157  4302 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cea6e20ed83a43cdaf95392d95766700"
dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
 from {username='slave'} at 127.0.0.1:47836
I20260709 13:13:06.349764  4303 raft_consensus.cc:493] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:13:06.349927  4302 raft_consensus.cc:493] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:13:06.350154  4303 raft_consensus.cc:3060] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:06.350430  4302 raft_consensus.cc:3060] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:06.352147  4505 consensus_queue.cc:1048] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:06.356240  4303 raft_consensus.cc:515] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.358033  4302 raft_consensus.cc:515] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.360147  4376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "047764b4386349a198cb3ae4b599730f" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:06.360839  4376 raft_consensus.cc:3055] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:13:06.361227  4376 raft_consensus.cc:740] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Running, Role: LEADER
I20260709 13:13:06.362040  4376 consensus_queue.cc:260] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.362993  4376 raft_consensus.cc:3060] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:06.367955  4553 consensus_queue.cc:1048] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:06.358769  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "047764b4386349a198cb3ae4b599730f" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:06.369506  4229 raft_consensus.cc:3060] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:06.360271  4303 leader_election.cc:290] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 2 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:06.374042  4304 raft_consensus.cc:1240] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Rejecting Update request from peer ba722a02e5c34752b5a164e6234797ee for earlier term 1. Current term is 2. Ops: []
I20260709 13:13:06.375646  4228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cea6e20ed83a43cdaf95392d95766700" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:06.376273  4228 raft_consensus.cc:3055] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:13:06.376585  4228 raft_consensus.cc:740] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Running, Role: LEADER
I20260709 13:13:06.376811  4524 consensus_queue.cc:1059] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:06.377923  4228 consensus_queue.cc:260] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.378854  4228 raft_consensus.cc:3060] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:06.379535  4376 raft_consensus.cc:2468] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 2.
I20260709 13:13:06.381070  4267 leader_election.cc:304] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, 6ff6192f285c425fb9d28bac1f0cea7c; no voters: 
I20260709 13:13:06.382225  4518 raft_consensus.cc:2804] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:13:06.382688  4518 raft_consensus.cc:697] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 LEADER]: Becoming Leader. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Running, Role: LEADER
I20260709 13:13:06.383458  4518 consensus_queue.cc:237] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.385812  4228 raft_consensus.cc:2468] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 2.
I20260709 13:13:06.386979  4270 leader_election.cc:304] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:06.387679  4527 raft_consensus.cc:2804] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:13:06.387900  4302 leader_election.cc:290] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 2 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:06.388173  4527 raft_consensus.cc:697] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 LEADER]: Becoming Leader. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Running, Role: LEADER
I20260709 13:13:06.389008  4527 consensus_queue.cc:237] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.390489  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cea6e20ed83a43cdaf95392d95766700" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:06.391089  4378 raft_consensus.cc:3060] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:06.397764  4431 consensus_queue.cc:1048] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:06.398142  4378 raft_consensus.cc:2468] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 2.
I20260709 13:13:06.402251  4229 raft_consensus.cc:2468] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 2.
I20260709 13:13:06.401860  4124 catalog_manager.cc:5697] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c reported cstate change: term changed from 1 to 2, leader changed from 2a925808f54d42ed9cc086f846280594 (127.23.140.67) to 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66). New cstate: current_term: 2 leader_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:06.403512  4124 catalog_manager.cc:5697] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c reported cstate change: term changed from 1 to 2, leader changed from ba722a02e5c34752b5a164e6234797ee (127.23.140.65) to 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66). New cstate: current_term: 2 leader_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:06.409062  4580 raft_consensus.cc:993] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594: : Instructing follower ba722a02e5c34752b5a164e6234797ee to start an election
I20260709 13:13:06.409541  4580 raft_consensus.cc:1081] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Signalling peer ba722a02e5c34752b5a164e6234797ee to start an election
I20260709 13:13:06.411096  4228 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9a69dca66f3a41e7a1a2bf40d5a5d641"
dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
 from {username='slave'} at 127.0.0.1:60066
I20260709 13:13:06.411670  4228 raft_consensus.cc:493] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:13:06.412015  4228 raft_consensus.cc:3060] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:06.419497  4524 consensus_queue.cc:1048] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:13:06.420604  4228 raft_consensus.cc:515] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.422685  4228 leader_election.cc:290] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 2 election: Requested vote from peers 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:06.423408  4301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a69dca66f3a41e7a1a2bf40d5a5d641" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:06.424007  4301 raft_consensus.cc:3060] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:06.426362  4378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a69dca66f3a41e7a1a2bf40d5a5d641" candidate_uuid: "ba722a02e5c34752b5a164e6234797ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:06.426949  4378 raft_consensus.cc:3055] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:13:06.427248  4378 raft_consensus.cc:740] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Running, Role: LEADER
I20260709 13:13:06.428100  4378 consensus_queue.cc:260] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.429343  4378 raft_consensus.cc:3060] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:06.431118  4301 raft_consensus.cc:2468] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 2.
I20260709 13:13:06.432261  4194 leader_election.cc:304] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:06.433105  4505 raft_consensus.cc:2804] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:13:06.433586  4505 raft_consensus.cc:697] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 2 LEADER]: Becoming Leader. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Running, Role: LEADER
I20260709 13:13:06.434471  4505 consensus_queue.cc:237] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.436233  4378 raft_consensus.cc:2468] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba722a02e5c34752b5a164e6234797ee in term 2.
I20260709 13:13:06.444006  4505 consensus_queue.cc:1048] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:06.445448  4124 catalog_manager.cc:5697] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee reported cstate change: term changed from 1 to 2, leader changed from 2a925808f54d42ed9cc086f846280594 (127.23.140.67) to ba722a02e5c34752b5a164e6234797ee (127.23.140.65). New cstate: current_term: 2 leader_uuid: "ba722a02e5c34752b5a164e6234797ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:06.472399  4527 consensus_queue.cc:1048] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:06.478736  4518 raft_consensus.cc:993] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c: : Instructing follower 2a925808f54d42ed9cc086f846280594 to start an election
I20260709 13:13:06.479192  4577 raft_consensus.cc:1081] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 LEADER]: Signalling peer 2a925808f54d42ed9cc086f846280594 to start an election
I20260709 13:13:06.480988  4376 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e72d13ac358342be97f27b246e85b802"
dest_uuid: "2a925808f54d42ed9cc086f846280594"
 from {username='slave'} at 127.0.0.1:45136
I20260709 13:13:06.481585  4376 raft_consensus.cc:493] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:13:06.481950  4376 raft_consensus.cc:3060] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:06.486114  4577 consensus_queue.cc:1048] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:06.489539  4376 raft_consensus.cc:515] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.494746  4229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e72d13ac358342be97f27b246e85b802" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:06.497186  4376 leader_election.cc:290] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 2 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079)
I20260709 13:13:06.498770  4301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e72d13ac358342be97f27b246e85b802" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:06.499404  4301 raft_consensus.cc:3055] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:13:06.499725  4301 raft_consensus.cc:740] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Running, Role: LEADER
I20260709 13:13:06.500466  4301 consensus_queue.cc:260] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.501602  4301 raft_consensus.cc:3060] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:06.502193  4505 consensus_queue.cc:1048] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:06.508770  4301 raft_consensus.cc:2468] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 2.
I20260709 13:13:06.510004  4343 leader_election.cc:304] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, 6ff6192f285c425fb9d28bac1f0cea7c; no voters: ba722a02e5c34752b5a164e6234797ee
I20260709 13:13:06.512137  4580 raft_consensus.cc:2804] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:13:06.512598  4580 raft_consensus.cc:697] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [term 2 LEADER]: Becoming Leader. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Running, Role: LEADER
I20260709 13:13:06.513335  4519 consensus_queue.cc:1048] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:13:06.513387  4580 consensus_queue.cc:237] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.524473  4523 raft_consensus.cc:993] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee: : Instructing follower 6ff6192f285c425fb9d28bac1f0cea7c to start an election
I20260709 13:13:06.524917  4519 raft_consensus.cc:1081] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Signalling peer 6ff6192f285c425fb9d28bac1f0cea7c to start an election
I20260709 13:13:06.526551  4302 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bd3699e5ca7e4fb7afaab54895c65b1f"
dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
 from {username='slave'} at 127.0.0.1:47836
I20260709 13:13:06.527113  4302 raft_consensus.cc:493] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:13:06.527462  4302 raft_consensus.cc:3060] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:06.528899  4124 catalog_manager.cc:5697] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 reported cstate change: term changed from 1 to 2, leader changed from 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66) to 2a925808f54d42ed9cc086f846280594 (127.23.140.67). New cstate: current_term: 2 leader_uuid: "2a925808f54d42ed9cc086f846280594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: HEALTHY } } }
I20260709 13:13:06.533764  4302 raft_consensus.cc:515] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.535718  4302 leader_election.cc:290] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 2 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 2a925808f54d42ed9cc086f846280594 (127.23.140.67:37585)
I20260709 13:13:06.536543  4228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3699e5ca7e4fb7afaab54895c65b1f" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:06.537205  4228 raft_consensus.cc:3055] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:13:06.537549  4228 raft_consensus.cc:740] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Running, Role: LEADER
I20260709 13:13:06.538148  4376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3699e5ca7e4fb7afaab54895c65b1f" candidate_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a925808f54d42ed9cc086f846280594"
I20260709 13:13:06.538343  4228 consensus_queue.cc:260] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.538764  4376 raft_consensus.cc:3060] T bd3699e5ca7e4fb7afaab54895c65b1f P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:06.539666  4228 raft_consensus.cc:3060] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:06.546692  4228 raft_consensus.cc:2468] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 2.
I20260709 13:13:06.547114  4376 raft_consensus.cc:2468] T bd3699e5ca7e4fb7afaab54895c65b1f P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff6192f285c425fb9d28bac1f0cea7c in term 2.
I20260709 13:13:06.547770  4270 leader_election.cc:304] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff6192f285c425fb9d28bac1f0cea7c, ba722a02e5c34752b5a164e6234797ee; no voters: 
I20260709 13:13:06.548642  4577 raft_consensus.cc:2804] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:13:06.549084  4577 raft_consensus.cc:697] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 LEADER]: Becoming Leader. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Running, Role: LEADER
I20260709 13:13:06.549889  4577 consensus_queue.cc:237] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:06.556329  4124 catalog_manager.cc:5697] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c reported cstate change: term changed from 1 to 2, leader changed from ba722a02e5c34752b5a164e6234797ee (127.23.140.65) to 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66). New cstate: current_term: 2 leader_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260709 13:13:06.823886  4228 raft_consensus.cc:1275] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Refusing update from remote peer 6ff6192f285c425fb9d28bac1f0cea7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:06.825070  4527 consensus_queue.cc:1048] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:06.833847  4229 raft_consensus.cc:1275] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Refusing update from remote peer 6ff6192f285c425fb9d28bac1f0cea7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:06.834865  4376 raft_consensus.cc:1275] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Refusing update from remote peer 6ff6192f285c425fb9d28bac1f0cea7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:06.835341  4518 consensus_queue.cc:1048] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:06.836473  4527 consensus_queue.cc:1048] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:06.845278  4302 raft_consensus.cc:1275] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Refusing update from remote peer ba722a02e5c34752b5a164e6234797ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:06.847141  4519 consensus_queue.cc:1048] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:06.851898  4376 raft_consensus.cc:1275] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Refusing update from remote peer 6ff6192f285c425fb9d28bac1f0cea7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:06.855906  4592 consensus_queue.cc:1048] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:06.859589  4376 raft_consensus.cc:1275] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Refusing update from remote peer ba722a02e5c34752b5a164e6234797ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:06.861455  4505 consensus_queue.cc:1048] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:07.073204  4378 raft_consensus.cc:1275] T bd3699e5ca7e4fb7afaab54895c65b1f P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Refusing update from remote peer 6ff6192f285c425fb9d28bac1f0cea7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:07.074445  4527 consensus_queue.cc:1048] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:07.081905  4229 raft_consensus.cc:1275] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Refusing update from remote peer 6ff6192f285c425fb9d28bac1f0cea7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:07.083247  4527 consensus_queue.cc:1048] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:07.118577  4229 raft_consensus.cc:3060] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:07.123920  4229 raft_consensus.cc:1275] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Refusing update from remote peer 2a925808f54d42ed9cc086f846280594: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:07.125051  4580 consensus_queue.cc:1048] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:07.133065  4302 raft_consensus.cc:1275] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Refusing update from remote peer 2a925808f54d42ed9cc086f846280594: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:07.135003  4593 consensus_queue.cc:1048] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:09.322883  4124 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38580:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 13:13:09.340260  4124 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38580:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 13:13:09.389024 24113 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:13:09.447865  4610 log.cc:927] Time spent T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
W20260709 13:13:09.556557  4267 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:37585, user_credentials={real_user=slave}} blocked reactor thread for 44435.6us
W20260709 13:13:09.570685  4624 log.cc:927] Time spent T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20260709 13:13:09.604192  4630 log.cc:927] Time spent T b7b904c5d9654b758bb74c910a8a22c2 P 2a925808f54d42ed9cc086f846280594: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 13:13:09.637696  4299 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "02b49437a48440b7990bb6f87b03b6af"
dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
mode: GRACEFUL
new_leader_uuid: "2a925808f54d42ed9cc086f846280594"
 from {username='slave'} at 127.0.0.1:54854
I20260709 13:13:09.643522  4299 raft_consensus.cc:606] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 LEADER]: Received request to transfer leadership to TS 2a925808f54d42ed9cc086f846280594
I20260709 13:13:09.683027  4657 tablet.cc:1724] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.687024  4650 tablet.cc:1724] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.697522  4656 tablet.cc:1724] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.692682  4660 tablet.cc:1724] T 6a287c34d1b14502b5541f96aa8c90bc P 6ff6192f285c425fb9d28bac1f0cea7c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.685611  4658 tablet.cc:1724] T bd3699e5ca7e4fb7afaab54895c65b1f P 2a925808f54d42ed9cc086f846280594: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 13:13:09.715268  4642 log.cc:927] Time spent T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
W20260709 13:13:09.718818  4644 log.cc:927] Time spent T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
W20260709 13:13:09.721681  4643 log.cc:927] Time spent T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
W20260709 13:13:09.731808  4192 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.67:37585, user_credentials={real_user=slave}} blocked reactor thread for 43888.6us
I20260709 13:13:09.709000  4652 tablet.cc:1724] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.725047  4661 tablet.cc:1724] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.707966  4646 tablet.cc:1724] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.761204  4627 raft_consensus.cc:993] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c: : Instructing follower 2a925808f54d42ed9cc086f846280594 to start an election
I20260709 13:13:09.761739  4627 raft_consensus.cc:1081] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 LEADER]: Signalling peer 2a925808f54d42ed9cc086f846280594 to start an election
I20260709 13:13:09.766110  4376 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "02b49437a48440b7990bb6f87b03b6af"
dest_uuid: "2a925808f54d42ed9cc086f846280594"
 from {username='slave'} at 127.0.0.1:45136
I20260709 13:13:09.766690  4376 raft_consensus.cc:493] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:13:09.767046  4376 raft_consensus.cc:3060] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:13:09.788105  4652 tablet.cc:1724] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.801332  4661 tablet.cc:1724] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.809444  4227 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ee8997813dd7430ca5f9d7269b2d0d5e"
dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
mode: GRACEFUL
new_leader_uuid: "2a925808f54d42ed9cc086f846280594"
 from {username='slave'} at 127.0.0.1:49456
I20260709 13:13:09.810141  4227 raft_consensus.cc:606] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Received request to transfer leadership to TS 2a925808f54d42ed9cc086f846280594
I20260709 13:13:09.811609 24113 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:13:09.812247 24113 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
W20260709 13:13:09.828500  4194 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.140.66:42079, user_credentials={real_user=slave}} blocked reactor thread for 74993.6us
I20260709 13:13:09.834849  4657 tablet.cc:1724] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.837144  4656 tablet.cc:1724] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.791713  4666 tablet.cc:1724] T b7b904c5d9654b758bb74c910a8a22c2 P 2a925808f54d42ed9cc086f846280594: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.842937  4660 tablet.cc:1724] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.812778  4376 raft_consensus.cc:515] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:09.853349  4226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02b49437a48440b7990bb6f87b03b6af" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:09.853994  4226 raft_consensus.cc:3060] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:13:09.855633  4667 tablet.cc:1724] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.861693  4376 leader_election.cc:290] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 3 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079)
I20260709 13:13:09.861435  4650 tablet.cc:1724] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.863068  4302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02b49437a48440b7990bb6f87b03b6af" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:09.863745  4302 raft_consensus.cc:3055] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:13:09.864135  4302 raft_consensus.cc:740] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6ff6192f285c425fb9d28bac1f0cea7c, State: Running, Role: LEADER
I20260709 13:13:09.863571  4668 tablet.cc:1724] T 6ceea03a69bd4c5399479d732de74d50 P 2a925808f54d42ed9cc086f846280594: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.864912  4302 consensus_queue.cc:260] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:09.866307  4302 raft_consensus.cc:3060] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:13:09.870399  4658 tablet.cc:1724] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.881462  4302 raft_consensus.cc:2468] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 3.
I20260709 13:13:09.890914  4343 leader_election.cc:304] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, 6ff6192f285c425fb9d28bac1f0cea7c; no voters: 
I20260709 13:13:09.891920  4595 raft_consensus.cc:2804] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:13:09.892371  4595 raft_consensus.cc:697] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [term 3 LEADER]: Becoming Leader. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Running, Role: LEADER
I20260709 13:13:09.893244  4595 consensus_queue.cc:237] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:09.895748  4666 tablet.cc:1724] T 2551af500d034f6096ef929e30795d3d P 2a925808f54d42ed9cc086f846280594: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.880076  4670 tablet.cc:1724] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.900489  4661 tablet.cc:1724] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.897656  4646 tablet.cc:1724] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.911171  4226 raft_consensus.cc:2468] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 3.
I20260709 13:13:09.921831  4124 catalog_manager.cc:5697] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 reported cstate change: term changed from 2 to 3, leader changed from 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66) to 2a925808f54d42ed9cc086f846280594 (127.23.140.67). New cstate: current_term: 3 leader_uuid: "2a925808f54d42ed9cc086f846280594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: HEALTHY } } }
I20260709 13:13:09.940749  4656 tablet.cc:1724] T b7b904c5d9654b758bb74c910a8a22c2 P 6ff6192f285c425fb9d28bac1f0cea7c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.950536  4666 tablet.cc:1724] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.954504  4668 tablet.cc:1724] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.944886  4660 tablet.cc:1724] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.909768  4652 tablet.cc:1724] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:13:09.990305  4127 catalog_manager.cc:6242] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4: soft_deleted_table [id=ac321171fefe4f11821c5bc3a8d0399a] alter complete (version 1)
I20260709 13:13:10.143723  4674 raft_consensus.cc:993] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee: : Instructing follower 2a925808f54d42ed9cc086f846280594 to start an election
I20260709 13:13:10.144261  4672 raft_consensus.cc:1081] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Signalling peer 2a925808f54d42ed9cc086f846280594 to start an election
I20260709 13:13:10.145978  4374 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ee8997813dd7430ca5f9d7269b2d0d5e"
dest_uuid: "2a925808f54d42ed9cc086f846280594"
 from {username='slave'} at 127.0.0.1:45126
I20260709 13:13:10.146490  4374 raft_consensus.cc:493] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:13:10.146787  4374 raft_consensus.cc:3060] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:10.151995  4374 raft_consensus.cc:515] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:10.153645  4374 leader_election.cc:290] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 2 election: Requested vote from peers ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371), 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079)
I20260709 13:13:10.154564  4226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee8997813dd7430ca5f9d7269b2d0d5e" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ba722a02e5c34752b5a164e6234797ee"
I20260709 13:13:10.154807  4302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee8997813dd7430ca5f9d7269b2d0d5e" candidate_uuid: "2a925808f54d42ed9cc086f846280594" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ff6192f285c425fb9d28bac1f0cea7c"
I20260709 13:13:10.155200  4226 raft_consensus.cc:3055] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:13:10.155439  4302 raft_consensus.cc:3060] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:10.155606  4226 raft_consensus.cc:740] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ba722a02e5c34752b5a164e6234797ee, State: Running, Role: LEADER
I20260709 13:13:10.156548  4226 consensus_queue.cc:260] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:10.157647  4226 raft_consensus.cc:3060] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:13:10.160193  4302 raft_consensus.cc:2468] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 2.
I20260709 13:13:10.161293  4343 leader_election.cc:304] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a925808f54d42ed9cc086f846280594, 6ff6192f285c425fb9d28bac1f0cea7c; no voters: 
I20260709 13:13:10.162001  4595 raft_consensus.cc:2804] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:13:10.162405  4595 raft_consensus.cc:697] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [term 2 LEADER]: Becoming Leader. State: Replica: 2a925808f54d42ed9cc086f846280594, State: Running, Role: LEADER
I20260709 13:13:10.163126  4226 raft_consensus.cc:2468] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a925808f54d42ed9cc086f846280594 in term 2.
I20260709 13:13:10.163170  4595 consensus_queue.cc:237] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } }
I20260709 13:13:10.170367  4127 catalog_manager.cc:5697] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 reported cstate change: term changed from 1 to 2, leader changed from ba722a02e5c34752b5a164e6234797ee (127.23.140.65) to 2a925808f54d42ed9cc086f846280594 (127.23.140.67). New cstate: current_term: 2 leader_uuid: "2a925808f54d42ed9cc086f846280594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a925808f54d42ed9cc086f846280594" member_type: VOTER last_known_addr { host: "127.23.140.67" port: 37585 } health_report { overall_health: HEALTHY } } }
I20260709 13:13:10.435077  4227 raft_consensus.cc:1275] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 3 FOLLOWER]: Refusing update from remote peer 2a925808f54d42ed9cc086f846280594: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:13:10.436262  4595 consensus_queue.cc:1048] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:13:10.450433  4302 raft_consensus.cc:1275] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 3 FOLLOWER]: Refusing update from remote peer 2a925808f54d42ed9cc086f846280594: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:13:10.452044  4649 consensus_queue.cc:1048] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:13:10.719766  4303 raft_consensus.cc:1275] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Refusing update from remote peer 2a925808f54d42ed9cc086f846280594: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:10.720371  4227 raft_consensus.cc:1275] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Refusing update from remote peer 2a925808f54d42ed9cc086f846280594: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:13:10.721228  4649 consensus_queue.cc:1048] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6192f285c425fb9d28bac1f0cea7c" member_type: VOTER last_known_addr { host: "127.23.140.66" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:10.721969  4683 consensus_queue.cc:1048] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba722a02e5c34752b5a164e6234797ee" member_type: VOTER last_known_addr { host: "127.23.140.65" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:13:10.866712 24113 tablet_server.cc:179] TabletServer@127.23.140.65:0 shutting down...
W20260709 13:13:10.876268  4344 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 8 similar messages]
W20260709 13:13:10.880033  4344 consensus_peers.cc:597] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 -> Peer ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371): Couldn't send request to peer ba722a02e5c34752b5a164e6234797ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:13:10.906853  4270 consensus_peers.cc:597] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c -> Peer ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371): Couldn't send request to peer ba722a02e5c34752b5a164e6234797ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:13:10.911163 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:13:10.912184 24113 tablet_replica.cc:333] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee: stopping tablet replica
I20260709 13:13:10.913066 24113 raft_consensus.cc:2243] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:10.913730 24113 raft_consensus.cc:2272] T bd3699e5ca7e4fb7afaab54895c65b1f P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:10.916498 24113 tablet_replica.cc:333] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee: stopping tablet replica
I20260709 13:13:10.917321 24113 raft_consensus.cc:2243] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:10.917745 24113 raft_consensus.cc:2272] T cea6e20ed83a43cdaf95392d95766700 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:10.920161 24113 tablet_replica.cc:333] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee: stopping tablet replica
I20260709 13:13:10.920706 24113 raft_consensus.cc:2243] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:10.921095 24113 raft_consensus.cc:2272] T 047764b4386349a198cb3ae4b599730f P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:10.923374 24113 tablet_replica.cc:333] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee: stopping tablet replica
I20260709 13:13:10.924026 24113 raft_consensus.cc:2243] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:13:10.924758 24113 raft_consensus.cc:2272] T 2551af500d034f6096ef929e30795d3d P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:10.926745 24113 tablet_replica.cc:333] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee: stopping tablet replica
I20260709 13:13:10.927290 24113 raft_consensus.cc:2243] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:13:10.928009 24113 raft_consensus.cc:2272] T b7b904c5d9654b758bb74c910a8a22c2 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:10.929953 24113 tablet_replica.cc:333] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee: stopping tablet replica
I20260709 13:13:10.930480 24113 raft_consensus.cc:2243] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:10.930924 24113 raft_consensus.cc:2272] T 02b49437a48440b7990bb6f87b03b6af P ba722a02e5c34752b5a164e6234797ee [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:10.933365 24113 tablet_replica.cc:333] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee: stopping tablet replica
I20260709 13:13:10.933919 24113 raft_consensus.cc:2243] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:10.934300 24113 raft_consensus.cc:2272] T 835876e7bd174f4184fd6294093fa911 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:10.936142 24113 tablet_replica.cc:333] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee: stopping tablet replica
I20260709 13:13:10.936585 24113 raft_consensus.cc:2243] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:13:10.937310 24113 raft_consensus.cc:2272] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:10.938962 24113 tablet_replica.cc:333] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee: stopping tablet replica
I20260709 13:13:10.939570 24113 raft_consensus.cc:2243] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:10.940052 24113 raft_consensus.cc:2272] T 6ceea03a69bd4c5399479d732de74d50 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:10.942425 24113 tablet_replica.cc:333] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee: stopping tablet replica
I20260709 13:13:10.942973 24113 raft_consensus.cc:2243] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Raft consensus shutting down.
W20260709 13:13:10.942999  4344 consensus_peers.cc:597] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 -> Peer ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371): Couldn't send request to peer ba722a02e5c34752b5a164e6234797ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:13:10.943822 24113 raft_consensus.cc:2272] T c0e4091b5254449b9ece5ccdccee16ee P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:10.945968 24113 tablet_replica.cc:333] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee: stopping tablet replica
I20260709 13:13:10.946570 24113 raft_consensus.cc:2243] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:13:10.947257 24113 raft_consensus.cc:2272] T 6a287c34d1b14502b5541f96aa8c90bc P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:10.948980 24113 tablet_replica.cc:333] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee: stopping tablet replica
I20260709 13:13:10.949542 24113 raft_consensus.cc:2243] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:10.949978 24113 raft_consensus.cc:2272] T 8f1d29a41a99463fa23d6308fc2afaaa P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:10.952049 24113 tablet_replica.cc:333] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee: stopping tablet replica
I20260709 13:13:10.952605 24113 raft_consensus.cc:2243] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:10.953006 24113 raft_consensus.cc:2272] T ee8997813dd7430ca5f9d7269b2d0d5e P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:10.955567 24113 tablet_replica.cc:333] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee: stopping tablet replica
I20260709 13:13:10.956130 24113 raft_consensus.cc:2243] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:10.956571 24113 raft_consensus.cc:2272] T fb2b800af71b4a95b06e180939b9cae9 P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:10.958773 24113 tablet_replica.cc:333] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P ba722a02e5c34752b5a164e6234797ee: stopping tablet replica
I20260709 13:13:10.959280 24113 raft_consensus.cc:2243] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:10.959699 24113 raft_consensus.cc:2272] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:10.961622 24113 tablet_replica.cc:333] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee: stopping tablet replica
I20260709 13:13:10.962105 24113 raft_consensus.cc:2243] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:13:10.962837 24113 raft_consensus.cc:2272] T ba55bc7ffc73446da8111405c7a7b7f0 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:10.964747 24113 tablet_replica.cc:333] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee: stopping tablet replica
I20260709 13:13:10.965272 24113 raft_consensus.cc:2243] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:10.965680 24113 raft_consensus.cc:2272] T e72d13ac358342be97f27b246e85b802 P ba722a02e5c34752b5a164e6234797ee [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:10.967413 24113 tablet_replica.cc:333] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee: stopping tablet replica
I20260709 13:13:10.967989 24113 raft_consensus.cc:2243] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:13:10.968757 24113 raft_consensus.cc:2272] T 2363b6da78da46a2a8435e57a9732b3d P ba722a02e5c34752b5a164e6234797ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.007521 24113 tablet_server.cc:196] TabletServer@127.23.140.65:0 shutdown complete.
I20260709 13:13:11.038334 24113 tablet_server.cc:179] TabletServer@127.23.140.66:0 shutting down...
I20260709 13:13:11.066401 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:13:11.067025 24113 tablet_replica.cc:333] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c: stopping tablet replica
I20260709 13:13:11.067684 24113 raft_consensus.cc:2243] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:13:11.068552 24113 raft_consensus.cc:2272] T cea6e20ed83a43cdaf95392d95766700 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.070454 24113 tablet_replica.cc:333] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c: stopping tablet replica
I20260709 13:13:11.071043 24113 raft_consensus.cc:2243] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:13:11.071846 24113 raft_consensus.cc:2272] T 047764b4386349a198cb3ae4b599730f P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.073971 24113 tablet_replica.cc:333] T b7b904c5d9654b758bb74c910a8a22c2 P 6ff6192f285c425fb9d28bac1f0cea7c: stopping tablet replica
I20260709 13:13:11.074555 24113 raft_consensus.cc:2243] T b7b904c5d9654b758bb74c910a8a22c2 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.075042 24113 raft_consensus.cc:2272] T b7b904c5d9654b758bb74c910a8a22c2 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.076711 24113 tablet_replica.cc:333] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c: stopping tablet replica
I20260709 13:13:11.077351 24113 raft_consensus.cc:2243] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:13:11.078140 24113 raft_consensus.cc:2272] T bd3699e5ca7e4fb7afaab54895c65b1f P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.079999 24113 tablet_replica.cc:333] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c: stopping tablet replica
I20260709 13:13:11.080608 24113 raft_consensus.cc:2243] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.081048 24113 raft_consensus.cc:2272] T 8f1d29a41a99463fa23d6308fc2afaaa P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.082978 24113 tablet_replica.cc:333] T 6a287c34d1b14502b5541f96aa8c90bc P 6ff6192f285c425fb9d28bac1f0cea7c: stopping tablet replica
I20260709 13:13:11.083518 24113 raft_consensus.cc:2243] T 6a287c34d1b14502b5541f96aa8c90bc P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.083952 24113 raft_consensus.cc:2272] T 6a287c34d1b14502b5541f96aa8c90bc P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.085741 24113 tablet_replica.cc:333] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c: stopping tablet replica
I20260709 13:13:11.086274 24113 raft_consensus.cc:2243] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.086673 24113 raft_consensus.cc:2272] T 2551af500d034f6096ef929e30795d3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.088487 24113 tablet_replica.cc:333] T 2363b6da78da46a2a8435e57a9732b3d P 6ff6192f285c425fb9d28bac1f0cea7c: stopping tablet replica
I20260709 13:13:11.089027 24113 raft_consensus.cc:2243] T 2363b6da78da46a2a8435e57a9732b3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.089553 24113 raft_consensus.cc:2272] T 2363b6da78da46a2a8435e57a9732b3d P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.091513 24113 tablet_replica.cc:333] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c: stopping tablet replica
I20260709 13:13:11.092031 24113 raft_consensus.cc:2243] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.092424 24113 raft_consensus.cc:2272] T ee8997813dd7430ca5f9d7269b2d0d5e P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.095847 24113 tablet_replica.cc:333] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c: stopping tablet replica
I20260709 13:13:11.096429 24113 raft_consensus.cc:2243] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:13:11.097247 24113 raft_consensus.cc:2272] T fb2b800af71b4a95b06e180939b9cae9 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.099560 24113 tablet_replica.cc:333] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c: stopping tablet replica
W20260709 13:13:11.099762  4344 consensus_peers.cc:597] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 -> Peer ba722a02e5c34752b5a164e6234797ee (127.23.140.65:40371): Couldn't send request to peer ba722a02e5c34752b5a164e6234797ee. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.65:40371: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:13:11.100083 24113 raft_consensus.cc:2243] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.100723 24113 raft_consensus.cc:2272] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.102654 24113 tablet_replica.cc:333] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c: stopping tablet replica
I20260709 13:13:11.103158 24113 raft_consensus.cc:2243] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.103565 24113 raft_consensus.cc:2272] T 02b49437a48440b7990bb6f87b03b6af P 6ff6192f285c425fb9d28bac1f0cea7c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.106361 24113 tablet_replica.cc:333] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c: stopping tablet replica
I20260709 13:13:11.106875 24113 raft_consensus.cc:2243] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:13:11.107630 24113 raft_consensus.cc:2272] T 835876e7bd174f4184fd6294093fa911 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.109728 24113 tablet_replica.cc:333] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c: stopping tablet replica
I20260709 13:13:11.110219 24113 raft_consensus.cc:2243] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:13:11.110970 24113 raft_consensus.cc:2272] T 6ceea03a69bd4c5399479d732de74d50 P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.112732 24113 tablet_replica.cc:333] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c: stopping tablet replica
I20260709 13:13:11.113334 24113 raft_consensus.cc:2243] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.113751 24113 raft_consensus.cc:2272] T c0e4091b5254449b9ece5ccdccee16ee P 6ff6192f285c425fb9d28bac1f0cea7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.115626 24113 tablet_replica.cc:333] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c: stopping tablet replica
I20260709 13:13:11.116127 24113 raft_consensus.cc:2243] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.116544 24113 raft_consensus.cc:2272] T e72d13ac358342be97f27b246e85b802 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.118283 24113 tablet_replica.cc:333] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c: stopping tablet replica
I20260709 13:13:11.118793 24113 raft_consensus.cc:2243] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.119189 24113 raft_consensus.cc:2272] T ba55bc7ffc73446da8111405c7a7b7f0 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.121238 24113 tablet_replica.cc:333] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c: stopping tablet replica
I20260709 13:13:11.121704 24113 raft_consensus.cc:2243] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:13:11.122453 24113 raft_consensus.cc:2272] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 6ff6192f285c425fb9d28bac1f0cea7c [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:13:11.148418  4343 consensus_peers.cc:597] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 -> Peer 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079): Couldn't send request to peer 6ff6192f285c425fb9d28bac1f0cea7c. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:13:11.160629 24113 tablet_server.cc:196] TabletServer@127.23.140.66:0 shutdown complete.
I20260709 13:13:11.191460 24113 tablet_server.cc:179] TabletServer@127.23.140.67:0 shutting down...
W20260709 13:13:11.194727  4343 consensus_peers.cc:597] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 -> Peer 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079): Couldn't send request to peer 6ff6192f285c425fb9d28bac1f0cea7c. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:13:11.223075 24113 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:13:11.223868 24113 tablet_replica.cc:333] T bd3699e5ca7e4fb7afaab54895c65b1f P 2a925808f54d42ed9cc086f846280594: stopping tablet replica
I20260709 13:13:11.224465 24113 raft_consensus.cc:2243] T bd3699e5ca7e4fb7afaab54895c65b1f P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.224967 24113 raft_consensus.cc:2272] T bd3699e5ca7e4fb7afaab54895c65b1f P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.226848 24113 tablet_replica.cc:333] T 2551af500d034f6096ef929e30795d3d P 2a925808f54d42ed9cc086f846280594: stopping tablet replica
I20260709 13:13:11.227411 24113 raft_consensus.cc:2243] T 2551af500d034f6096ef929e30795d3d P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.227837 24113 raft_consensus.cc:2272] T 2551af500d034f6096ef929e30795d3d P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.229625 24113 tablet_replica.cc:333] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594: stopping tablet replica
I20260709 13:13:11.230139 24113 raft_consensus.cc:2243] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.230571 24113 raft_consensus.cc:2272] T cea6e20ed83a43cdaf95392d95766700 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.232245 24113 tablet_replica.cc:333] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594: stopping tablet replica
I20260709 13:13:11.232753 24113 raft_consensus.cc:2243] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.233212 24113 raft_consensus.cc:2272] T 047764b4386349a198cb3ae4b599730f P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.234913 24113 tablet_replica.cc:333] T b7b904c5d9654b758bb74c910a8a22c2 P 2a925808f54d42ed9cc086f846280594: stopping tablet replica
I20260709 13:13:11.235421 24113 raft_consensus.cc:2243] T b7b904c5d9654b758bb74c910a8a22c2 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.235826 24113 raft_consensus.cc:2272] T b7b904c5d9654b758bb74c910a8a22c2 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.237619 24113 tablet_replica.cc:333] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594: stopping tablet replica
I20260709 13:13:11.238159 24113 raft_consensus.cc:2243] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:13:11.238916 24113 raft_consensus.cc:2272] T ee8997813dd7430ca5f9d7269b2d0d5e P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.241253 24113 tablet_replica.cc:333] T fb2b800af71b4a95b06e180939b9cae9 P 2a925808f54d42ed9cc086f846280594: stopping tablet replica
I20260709 13:13:11.241782 24113 raft_consensus.cc:2243] T fb2b800af71b4a95b06e180939b9cae9 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.242187 24113 raft_consensus.cc:2272] T fb2b800af71b4a95b06e180939b9cae9 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.244168 24113 tablet_replica.cc:333] T 6ceea03a69bd4c5399479d732de74d50 P 2a925808f54d42ed9cc086f846280594: stopping tablet replica
I20260709 13:13:11.244697 24113 raft_consensus.cc:2243] T 6ceea03a69bd4c5399479d732de74d50 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.245203 24113 raft_consensus.cc:2272] T 6ceea03a69bd4c5399479d732de74d50 P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.247430 24113 tablet_replica.cc:333] T c0e4091b5254449b9ece5ccdccee16ee P 2a925808f54d42ed9cc086f846280594: stopping tablet replica
I20260709 13:13:11.248205 24113 raft_consensus.cc:2243] T c0e4091b5254449b9ece5ccdccee16ee P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 13:13:11.248605  4343 consensus_peers.cc:597] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 -> Peer 6ff6192f285c425fb9d28bac1f0cea7c (127.23.140.66:42079): Couldn't send request to peer 6ff6192f285c425fb9d28bac1f0cea7c. Status: Network error: Client connection negotiation failed: client connection to 127.23.140.66:42079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:13:11.248674 24113 raft_consensus.cc:2272] T c0e4091b5254449b9ece5ccdccee16ee P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.251272 24113 tablet_replica.cc:333] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594: stopping tablet replica
I20260709 13:13:11.251835 24113 raft_consensus.cc:2243] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.252244 24113 raft_consensus.cc:2272] T 6a287c34d1b14502b5541f96aa8c90bc P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.254074 24113 tablet_replica.cc:333] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594: stopping tablet replica
I20260709 13:13:11.254617 24113 raft_consensus.cc:2243] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:13:11.255398 24113 raft_consensus.cc:2272] T 8f1d29a41a99463fa23d6308fc2afaaa P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.257404 24113 tablet_replica.cc:333] T 2363b6da78da46a2a8435e57a9732b3d P 2a925808f54d42ed9cc086f846280594: stopping tablet replica
I20260709 13:13:11.257934 24113 raft_consensus.cc:2243] T 2363b6da78da46a2a8435e57a9732b3d P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.258366 24113 raft_consensus.cc:2272] T 2363b6da78da46a2a8435e57a9732b3d P 2a925808f54d42ed9cc086f846280594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.260296 24113 tablet_replica.cc:333] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594: stopping tablet replica
I20260709 13:13:11.260779 24113 raft_consensus.cc:2243] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.261242 24113 raft_consensus.cc:2272] T 9a69dca66f3a41e7a1a2bf40d5a5d641 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.262882 24113 tablet_replica.cc:333] T 835876e7bd174f4184fd6294093fa911 P 2a925808f54d42ed9cc086f846280594: stopping tablet replica
I20260709 13:13:11.263398 24113 raft_consensus.cc:2243] T 835876e7bd174f4184fd6294093fa911 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.263809 24113 raft_consensus.cc:2272] T 835876e7bd174f4184fd6294093fa911 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.265727 24113 tablet_replica.cc:333] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594: stopping tablet replica
I20260709 13:13:11.266237 24113 raft_consensus.cc:2243] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:13:11.266958 24113 raft_consensus.cc:2272] T 02b49437a48440b7990bb6f87b03b6af P 2a925808f54d42ed9cc086f846280594 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.269073 24113 tablet_replica.cc:333] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594: stopping tablet replica
I20260709 13:13:11.269621 24113 raft_consensus.cc:2243] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:13:11.270328 24113 raft_consensus.cc:2272] T e72d13ac358342be97f27b246e85b802 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.272034 24113 tablet_replica.cc:333] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594: stopping tablet replica
I20260709 13:13:11.272500 24113 raft_consensus.cc:2243] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.272948 24113 raft_consensus.cc:2272] T ba55bc7ffc73446da8111405c7a7b7f0 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.274848 24113 tablet_replica.cc:333] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594: stopping tablet replica
I20260709 13:13:11.275342 24113 raft_consensus.cc:2243] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:13:11.275771 24113 raft_consensus.cc:2272] T 2a8b2f9fd9bf4cd2a4ec339ca2e08d03 P 2a925808f54d42ed9cc086f846280594 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.313405 24113 tablet_server.cc:196] TabletServer@127.23.140.67:0 shutdown complete.
I20260709 13:13:11.341969 24113 master.cc:562] Master@127.23.140.126:33139 shutting down...
I20260709 13:13:11.365303 24113 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:13:11.365895 24113 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:13:11.366266 24113 tablet_replica.cc:333] T 00000000000000000000000000000000 P 2a1a5c74cdb24785b9c419554d8589b4: stopping tablet replica
I20260709 13:13:11.386755 24113 master.cc:584] Master@127.23.140.126:33139 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1 (11468 ms)
[----------] 2 tests from FilterSoftDeletedTableTest (23911 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 2 test suites ran. (378269 ms total)
[  PASSED  ] 7 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] LeaderRebalancerTest.AddTserver

 1 FAILED TEST
I20260709 13:13:11.443112 24113 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:220: suppressed but not reported on 393 messages since previous log ~14 seconds ago
I20260709 13:13:11.443439 24113 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:304: suppressed but not reported on 393 messages since previous log ~14 seconds ago
I20260709 13:13:11.443645 24113 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:139: suppressed but not reported on 390 messages since previous log ~14 seconds ago
I20260709 13:13:11.443909 24113 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:600: suppressed but not reported on 9 messages since previous log ~32 seconds ago
I20260709 13:13:11.444088 24113 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:548: suppressed but not reported on 9 messages since previous log ~32 seconds ago
I20260709 13:13:11.444316 24113 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 8 messages since previous log ~184 seconds ago
I20260709 13:13:11.444648 24113 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 6 messages since previous log ~0 seconds ago
I20260709 13:13:11.444998 24113 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet.cc:2406: suppressed but not reported on 26 messages since previous log ~6 seconds ago
